carray-netcdf 1.0.13 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +79 -0
- data/CONTRIBUTING.md +75 -0
- data/LICENSE +21 -0
- data/README.md +108 -2
- data/bin/ncheader +223 -0
- data/bin/ncvar +160 -0
- data/carray-netcdf.gemspec +37 -19
- data/docs/00_introduction.md +72 -0
- data/docs/01_getting_started.md +114 -0
- data/docs/02_reading.md +216 -0
- data/docs/03_writing.md +418 -0
- data/docs/04_codec.md +373 -0
- data/docs/05_groups.md +200 -0
- data/docs/06_layer1.md +160 -0
- data/docs/07_cookbook.md +345 -0
- data/docs/08_cheatsheet.md +326 -0
- data/docs/09_cli.md +79 -0
- data/lib/carray/netcdf/codec.rb +536 -0
- data/lib/carray/netcdf/dim.rb +67 -0
- data/lib/carray/netcdf/file.rb +354 -0
- data/lib/carray/netcdf/group.rb +517 -0
- data/lib/carray/netcdf/var.rb +539 -0
- data/lib/carray/netcdf/version.rb +5 -0
- data/lib/carray/netcdf.rb +287 -0
- data/lib/carray-netcdf.rb +3 -1
- metadata +80 -25
- data/API.md +0 -270
- data/Rakefile +0 -12
- data/bin/ncdef2yaml +0 -60
- data/examples/test.nc +0 -0
- data/examples/test.rb +0 -41
- data/examples/time.rb +0 -11
- data/ext/extconf.rb +0 -11
- data/ext/rb_netcdflib.c +0 -2674
- data/lib/io/netcdf.rb +0 -848
data/docs/04_codec.md
ADDED
|
@@ -0,0 +1,373 @@
|
|
|
1
|
+
# Codecs and CF conventions
|
|
2
|
+
|
|
3
|
+
A **codec** turns raw on-disk values into Ruby-side values on read and back again on write. There are three built in:
|
|
4
|
+
|
|
5
|
+
- **`Identity`** — no transform. Raw values in, raw values out.
|
|
6
|
+
- **`CF`** — the CF Conventions value transforms.
|
|
7
|
+
- **`COARDS`** — CF's predecessor; functionally identical for the value transforms this codec touches.
|
|
8
|
+
|
|
9
|
+
The codec runs transparently inside `Var#[]` / `Var#[]=` (both array and scalar paths — see the [reading chapter](02_reading.md#hyperslab-reads-ncvari-j-k)). You usually don't call codec methods directly.
|
|
10
|
+
|
|
11
|
+
## What CF actually does (and doesn't)
|
|
12
|
+
|
|
13
|
+
| CF attribute | Codec does |
|
|
14
|
+
|---|---|
|
|
15
|
+
| `_Unsigned` | Read: bit-reinterpret signed on-disk type to its unsigned counterpart (`int8 → uint8`, etc.). Write: reverse. Runs FIRST so fill/valid/scale all see the correct numeric interpretation. |
|
|
16
|
+
| `_FillValue`, `missing_value` | Read: mask (`UNDEF`). Write: `UNDEF` → fill sentinel. If `_Unsigned` is set, the attribute value is reinterpreted to unsigned space before comparison. |
|
|
17
|
+
| `scale_factor`, `add_offset` | Read: `packed × scale + offset`. Write: `round((v − offset) / scale)` into var type. |
|
|
18
|
+
| `valid_range`, `valid_min`, `valid_max` | Read (opt-in): out-of-range → `UNDEF`. Write: no effect. |
|
|
19
|
+
| `units` = `"<u> since <ref>"` (+ `calendar`) | Read: `CATime`. Write: `CATime` → count since ref (a fraction only into a float variable). |
|
|
20
|
+
| `units` = anything else (`"m s-1"`, `"K"`, `"kg m-3"`, …) | Raw attribute, **no UDUNITS conversion**. |
|
|
21
|
+
| everything else (`standard_name`, `grid_mapping`, `cell_methods`, `coordinates`, `ancillary_variables`, …) | Raw attribute, no interpretation. |
|
|
22
|
+
|
|
23
|
+
The last two rows are important: **this codec is a var-local value codec, not a CF data model builder**. Coordinate systems, cell methods, standard names, grid mappings — all preserved as attributes (`var.att("grid_mapping")` works), but the codec does not interpret them, nor does it walk to sibling variables. If you need CF data model support (a real coordinate-system layer akin to netCDF-Java's `CoordSystemBuilder`), either build it on top of this codec or reach for a heavier library.
|
|
24
|
+
|
|
25
|
+
### Contract for codec methods
|
|
26
|
+
|
|
27
|
+
Every codec method takes `(var, ca)` and only that. A codec:
|
|
28
|
+
|
|
29
|
+
- Reads attributes off `var` (own attributes only).
|
|
30
|
+
- Transforms the CArray `ca` (own values only).
|
|
31
|
+
- **Does NOT** walk to `var.file` to reach other variables or global state. Ruby cannot enforce this structurally, but doing so is a contract violation and puts the codec on the wrong side of the boundary.
|
|
32
|
+
- **Does NOT** rely on `var.dims` beyond reading their names (for documentation purposes, if at all) — again, that leaves the variable's frame of reference.
|
|
33
|
+
|
|
34
|
+
The dispatch that picks a codec at File.open time DOES look at the whole file (specifically the `Conventions` attribute) — that's a one-shot selection, not a per-transform operation, so it stays on the right side of the boundary.
|
|
35
|
+
|
|
36
|
+
## Codec selection
|
|
37
|
+
|
|
38
|
+
By default, `File.open` inspects the `Conventions` global attribute:
|
|
39
|
+
|
|
40
|
+
```ruby
|
|
41
|
+
NC::File.open("cf_file.nc") do |nc|
|
|
42
|
+
nc.codec # => #<CA::NetCDF::Codec::CF ...>
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
NC::File.open("coards_file.nc") do |nc|
|
|
46
|
+
nc.codec # => #<CA::NetCDF::Codec::COARDS ...>
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
NC::File.open("no_conventions.nc") do |nc|
|
|
50
|
+
nc.codec # => CA::NetCDF::Codec::Identity (raw values)
|
|
51
|
+
end
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Naming rules (checked in registry order):
|
|
55
|
+
|
|
56
|
+
- `Conventions: "COARDS"` (or combos like `"COARDS/CF-1.0"`) → `COARDS`.
|
|
57
|
+
- `Conventions: "CF-1.7"` / `"CF-1.8"` / `"CF 1.7"` / `"CF"` → `CF`.
|
|
58
|
+
- When the whole string matches neither, each of its tokens (split on spaces, commas and slashes) is tried, so `"ACDD-1.3, CF-1.8"` → `CF`.
|
|
59
|
+
- Anything else, including missing → `Identity`.
|
|
60
|
+
|
|
61
|
+
Override on open:
|
|
62
|
+
|
|
63
|
+
```ruby
|
|
64
|
+
NC::File.open("data.nc", codec: :identity) # raw only
|
|
65
|
+
NC::File.open("data.nc", codec: :cf) # force CF
|
|
66
|
+
NC::File.open("data.nc", codec: :coards) # force COARDS
|
|
67
|
+
NC::File.open("data.nc", codec: [:cf, mask_valid_range: true]) # CF with option
|
|
68
|
+
NC::File.open("data.nc", codec: NC::Codec::CF.new(mask_valid_range: true))
|
|
69
|
+
NC::File.open("data.nc", codec: NC::Codec::CF) # a class: instantiated with defaults
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
## `_FillValue` and `missing_value`
|
|
73
|
+
|
|
74
|
+
Round-trip:
|
|
75
|
+
|
|
76
|
+
```ruby
|
|
77
|
+
NC::File.create("fill.nc") do |nc|
|
|
78
|
+
nc.def_dim("n", 5)
|
|
79
|
+
nc.def_var("v", :short, ["n"], _FillValue: -32768)
|
|
80
|
+
nc.def_att("Conventions", "CF-1.7")
|
|
81
|
+
nc["v"] = CA_SHORT([10, -32768, 30, 40, 50]) # cell 1 written as fill
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
NC::File.open("fill.nc") do |nc|
|
|
85
|
+
v = nc["v"].to_ca
|
|
86
|
+
v.is_masked[1] # => true (decoded to mask)
|
|
87
|
+
v[1] # => UNDEF (scalar decode too)
|
|
88
|
+
v.count_masked # => 1
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
# Write UNDEF -> fill on encode
|
|
92
|
+
NC::File.open("fill.nc", :write) do |nc|
|
|
93
|
+
nc["v"][3] = UNDEF # written back as -32768
|
|
94
|
+
end
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
If a variable has **both** `_FillValue` and `missing_value`, both sentinels are masked. `missing_value` may be a scalar or an array (multiple sentinels).
|
|
98
|
+
|
|
99
|
+
Writing `UNDEF` to a variable without a `_FillValue` raises:
|
|
100
|
+
|
|
101
|
+
```ruby
|
|
102
|
+
ca = CA_SHORT([1, 2, 3])
|
|
103
|
+
ca[1] = UNDEF
|
|
104
|
+
nc["v"] = ca # ArgumentError: no _FillValue to encode masked cells
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
## `scale_factor` and `add_offset`
|
|
108
|
+
|
|
109
|
+
CF's packing formula (§8.1):
|
|
110
|
+
|
|
111
|
+
```
|
|
112
|
+
unpacked = packed × scale_factor + add_offset
|
|
113
|
+
packed = round((unpacked − add_offset) / scale_factor)
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
Round-trip:
|
|
117
|
+
|
|
118
|
+
```ruby
|
|
119
|
+
NC::File.create("packed.nc") do |nc|
|
|
120
|
+
nc.def_dim("n", 4)
|
|
121
|
+
nc.def_var("temp_kelvin", :short, ["n"],
|
|
122
|
+
scale_factor: 0.01, add_offset: 273.15, units: "K")
|
|
123
|
+
nc.def_att("Conventions", "CF-1.7")
|
|
124
|
+
|
|
125
|
+
# write physical values; codec packs them into short
|
|
126
|
+
nc["temp_kelvin"] = CA_DOUBLE([273.15, 283.15, 293.15, 303.15])
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
NC::File.open("packed.nc") do |nc|
|
|
130
|
+
nc["temp_kelvin"].to_ca.to_a # => [273.15, 283.15, 293.15, 303.15]
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
NC::File.open("packed.nc", codec: :identity) do |nc|
|
|
134
|
+
nc["temp_kelvin"].to_ca.to_a # => [0, 1000, 2000, 3000] (raw packed short)
|
|
135
|
+
end
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
**Rounding is required.** A bare cast to an integer variable would truncate (`1.0 / 0.1 == 9.999...` → `9`); the codec rounds. Symmetric on read: no rounding, just the linear transform.
|
|
139
|
+
|
|
140
|
+
Interaction with `_FillValue`: **fill is judged in packed space, before unpacking.** So `_FillValue: -32768` on a packed short is compared against the raw short values, not against the unpacked physical values.
|
|
141
|
+
|
|
142
|
+
## `valid_range` / `valid_min` / `valid_max`
|
|
143
|
+
|
|
144
|
+
CF says values outside `valid_range` are missing. The codec **does not** apply this by default — see the design note in `codec.rb` — because producers often set conservative ranges and silently masking a large fraction of the data is a debugging trap. `_FillValue` masking (which is per-cell explicit) is on; range masking (which is per-file policy) is opt-in.
|
|
145
|
+
|
|
146
|
+
```ruby
|
|
147
|
+
# default: raw attribute readable, values not masked
|
|
148
|
+
NC::File.open("ranged.nc") do |nc|
|
|
149
|
+
nc["v"].att("valid_range") # => e.g. [0, 100]
|
|
150
|
+
nc["v"].to_ca # => everything decoded, out-of-range visible
|
|
151
|
+
end
|
|
152
|
+
|
|
153
|
+
# opt-in: hard-mask
|
|
154
|
+
NC::File.open("ranged.nc", codec: [:cf, mask_valid_range: true]) do |nc|
|
|
155
|
+
nc["v"].to_ca # => values outside [0, 100] are UNDEF
|
|
156
|
+
end
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Rules (CF §2.5.1):
|
|
160
|
+
- `valid_range` takes precedence over `valid_min`/`valid_max`.
|
|
161
|
+
- All three are judged in **packed** space (before scale/offset). Encode a physical-range test yourself if you want the physical-space check.
|
|
162
|
+
- Attribute type should match the variable's on-disk type; the codec passes whatever's on disk through to the comparison.
|
|
163
|
+
|
|
164
|
+
## CF time coordinates
|
|
165
|
+
|
|
166
|
+
CF encodes time as a numeric count with a units string:
|
|
167
|
+
|
|
168
|
+
```
|
|
169
|
+
units: "hours since 2022-01-01T00:00:00Z"
|
|
170
|
+
units: "days since 2000-01-01"
|
|
171
|
+
units: "seconds since 1970-01-01 00:00:00"
|
|
172
|
+
units: "5 days since 2024-06-15" # any positive integer count
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
The codec maps this to CATime on read and back on write.
|
|
176
|
+
|
|
177
|
+
### Read: `count × base since ref` → `CATime`
|
|
178
|
+
|
|
179
|
+
```ruby
|
|
180
|
+
NC::File.open("cf_time.nc") do |nc|
|
|
181
|
+
t = nc["time"].to_ca
|
|
182
|
+
t.class # => CATime
|
|
183
|
+
t.unit # => #<CATime::Resolution 1 h>
|
|
184
|
+
t[0] # => #<CATime::Element 2022-01-01T00:00:00Z>
|
|
185
|
+
t[0].to_time # => 2022-01-01 00:00:00 UTC (Ruby Time)
|
|
186
|
+
t[0].to_date # => #<Date: 2022-01-01>
|
|
187
|
+
end
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
### Write: `CATime` → `count since ref`
|
|
191
|
+
|
|
192
|
+
```ruby
|
|
193
|
+
NC::File.create("t.nc") do |nc|
|
|
194
|
+
nc.def_dim("time", 24)
|
|
195
|
+
nc.def_var("time", :int, ["time"],
|
|
196
|
+
units: "hours since 2022-01-01T00:00:00Z", calendar: "standard")
|
|
197
|
+
nc.def_att("Conventions", "CF-1.7")
|
|
198
|
+
|
|
199
|
+
nc["time"] = CArray.time_series("2022-01-01T00:00:00Z", count: 24, unit: :h)
|
|
200
|
+
end
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
Scalar assignment works too:
|
|
204
|
+
|
|
205
|
+
```ruby
|
|
206
|
+
NC::File.open("t.nc", :write) do |nc|
|
|
207
|
+
# a CATime::Element
|
|
208
|
+
nc["time"][0] = CArray.time("2022-01-01T05:00:00Z", unit: :h)[0]
|
|
209
|
+
# or a plain integer (interpreted as "count since ref" — bypasses CATime encode)
|
|
210
|
+
nc["time"][1] = 10
|
|
211
|
+
end
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
### N-of-a-unit intervals: `"5 days since ..."`
|
|
215
|
+
|
|
216
|
+
The codec stores CATime at the base resolution (day, in this case) and folds the count `5` into the numeric values. That way any reference date representable in the base works — including the normal case where the ref doesn't land on the coarser `count × base` grid from the epoch (e.g. `"5 days since 2024-06-15"` — 2024-06-15 is not on a 5-day-from-1970 grid, but it IS on the day grid). The count information isn't lost: it stays on `atts["units"]`, we just don't bake it into CATime's storage tick.
|
|
217
|
+
|
|
218
|
+
```ruby
|
|
219
|
+
# raw N reads as N*5 days past 2024-06-15
|
|
220
|
+
dt = CA_INT64([0, 5, 10]).time(unit: :D, origin: "2024-06-15")
|
|
221
|
+
# writing dt to units="5 days since 2024-06-15":
|
|
222
|
+
# codec.timesteps(unit: :D, origin: "2024-06-15") -> [0, 5, 10]
|
|
223
|
+
# then / 5 -> [0, 1, 2] (on-disk)
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
### Float time coordinates
|
|
227
|
+
|
|
228
|
+
Many CF files store time as `double`. Whole values decode at the declared base; a value with a fraction is carried to the first finer unit that holds it whole, so nothing is rounded away:
|
|
229
|
+
|
|
230
|
+
```
|
|
231
|
+
units: "hours since ..." with float values 0.0, 1.0, 2.0 -> CATime at 1 h
|
|
232
|
+
units: "days since ..." with float values 0.0, 0.5, 1.25 -> CATime at 1 h: +0 h, +12 h, +30 h
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Only `year` and `month` fractions are rounded to the nearest whole tick, since those units have no fixed length.
|
|
236
|
+
|
|
237
|
+
On write, an instant finer than the units' interval is written as a fraction into a float variable (`01:30` into `"days since ..."` is `0.0625`). An integer variable can't hold one, so the write raises `ArgumentError` instead of truncating.
|
|
238
|
+
|
|
239
|
+
### Calendars
|
|
240
|
+
|
|
241
|
+
The codec handles these calendar attribute values:
|
|
242
|
+
|
|
243
|
+
- `standard`, `gregorian`, `proleptic_gregorian`, or missing → CATime (proleptic Gregorian, UTC).
|
|
244
|
+
- Anything else — `noleap`, `365_day`, `all_leap`, `366_day`, `360_day`, `julian`, `none` — is unsupported. The codec warns once per distinct calendar value and returns raw values unchanged. Writing a `CATime` to a variable with an unsupported calendar raises.
|
|
245
|
+
|
|
246
|
+
For post-1582 dates, `standard` and `proleptic_gregorian` agree, so CATime is exact for modern climate/ocean data. Pre-1582 or non-standard calendars belong in an external gem.
|
|
247
|
+
|
|
248
|
+
## Identity codec
|
|
249
|
+
|
|
250
|
+
Skip all transforms — raw bytes only:
|
|
251
|
+
|
|
252
|
+
```ruby
|
|
253
|
+
NC::File.open("data.nc", codec: :identity) do |nc|
|
|
254
|
+
nc["temp"].to_ca # packed shorts, or whatever's on disk
|
|
255
|
+
nc["time"].to_ca # raw int/double, no CATime wrapping
|
|
256
|
+
nc["temp"][0, 0, 0] # raw scalar, no _FillValue -> UNDEF
|
|
257
|
+
end
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
Useful for round-tripping unknown files, debugging what a producer actually wrote, or side-stepping a codec bug.
|
|
261
|
+
|
|
262
|
+
## COARDS codec
|
|
263
|
+
|
|
264
|
+
COARDS (1995) is CF's predecessor. Every adherent COARDS file is also an adherent CF file — spec-inclusion, not just similarity — so `COARDS < CF` inherits every transform; only `self.claims?` differs. Everything above (`_FillValue`, scale/offset, `valid_range`, CF time, calendar policy) applies verbatim.
|
|
265
|
+
|
|
266
|
+
```ruby
|
|
267
|
+
NC::File.open("coards_file.nc") do |nc|
|
|
268
|
+
nc.codec.is_a?(NC::Codec::COARDS) # => true
|
|
269
|
+
nc.codec.is_a?(NC::Codec::CF) # => true (COARDS < CF)
|
|
270
|
+
# all the same transforms work
|
|
271
|
+
end
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
**Check `is_a?(COARDS)` BEFORE `is_a?(CF)`** in an upper-layer dispatcher — COARDS pins the coordinate axis order to `T, Z, Y, X`, so an upper layer that treats a COARDS file as generic CF may silently accept looser axis orderings. (This value codec doesn't touch axis order, so at THIS layer the distinction doesn't matter; it matters at the layer above.)
|
|
275
|
+
|
|
276
|
+
`Conventions` strings that route to COARDS:
|
|
277
|
+
- `"COARDS"` alone.
|
|
278
|
+
- Combos starting with `COARDS` (e.g. `"COARDS/CF-1.0"`) — some tools emit these.
|
|
279
|
+
|
|
280
|
+
COARDS predates the `calendar` attribute; a COARDS time variable without an explicit calendar lands on the same proleptic-Gregorian axis as CF `"standard"` (identical for post-1582 dates).
|
|
281
|
+
|
|
282
|
+
## Writing your own codec
|
|
283
|
+
|
|
284
|
+
The codec interface is minimal:
|
|
285
|
+
|
|
286
|
+
```ruby
|
|
287
|
+
class MyCodec
|
|
288
|
+
# Optional: class method for auto-selection by the Conventions attribute.
|
|
289
|
+
# Only inspects the Conventions string — deliberately does NOT take the
|
|
290
|
+
# File, to keep dispatch out of the CDM Layer 2 territory (a "look at
|
|
291
|
+
# coordinate variables to sniff CF" hook is intentionally not provided).
|
|
292
|
+
def self.claims?(conventions_string) = ...
|
|
293
|
+
|
|
294
|
+
# Required: the two transforms. Both are var-local — see the contract
|
|
295
|
+
# note earlier in this chapter.
|
|
296
|
+
def decode(var, ca) = ... # var: Var, ca: raw CArray -> processed value
|
|
297
|
+
def encode(var, ca) = ... # var: Var, ca: value -> raw CArray for the on-disk write
|
|
298
|
+
end
|
|
299
|
+
|
|
300
|
+
CA::NetCDF::Codec.register(MyCodec)
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
Registered codecs are matched in reverse registration order (later registrations win). `resolve` accepts:
|
|
304
|
+
|
|
305
|
+
- `:auto` (default) — match by `Conventions`; fallback `Identity`
|
|
306
|
+
- `:identity` / `false` / `nil` → `Identity`
|
|
307
|
+
- `:cf`, `:coards` → the named codec with defaults
|
|
308
|
+
- `[:cf, opts]`, `[:coards, opts]` → with keyword options
|
|
309
|
+
- a `Module` (used as-is, e.g. `Identity`)
|
|
310
|
+
- any object that responds to `#decode(var, ca)` and `#encode(var, ca)`
|
|
311
|
+
|
|
312
|
+
So an ad-hoc codec doesn't even need to register — you can pass an instance directly:
|
|
313
|
+
|
|
314
|
+
```ruby
|
|
315
|
+
class OnlyMasksCodec
|
|
316
|
+
def decode(var, ca)
|
|
317
|
+
# apply only fill masking, skip everything else
|
|
318
|
+
fill = var.att("_FillValue") or return ca
|
|
319
|
+
out = ca.copy
|
|
320
|
+
out[:eq, fill] = UNDEF
|
|
321
|
+
out
|
|
322
|
+
end
|
|
323
|
+
def encode(var, ca) = ca # writes unchanged
|
|
324
|
+
end
|
|
325
|
+
|
|
326
|
+
NC::File.open("data.nc", codec: OnlyMasksCodec.new) do |nc|
|
|
327
|
+
...
|
|
328
|
+
end
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
## Full pipeline order
|
|
332
|
+
|
|
333
|
+
Decode (read):
|
|
334
|
+
|
|
335
|
+
1. `decode_unsigned` — `_Unsigned` reinterprets signed on-disk bits to their unsigned counterpart. Runs FIRST so every subsequent comparison (fill, valid_range) sees the numerically-correct value.
|
|
336
|
+
2. `decode_fill` — `_FillValue` / `missing_value` → mask (in unsigned space if `_Unsigned` was set; the attribute value is reinterpreted).
|
|
337
|
+
3. `decode_valid` — `valid_range` / `valid_min` / `valid_max` → mask (opt-in, same space as fill).
|
|
338
|
+
4. `decode_scale` — `packed × scale + offset`.
|
|
339
|
+
5. `decode_time` — `count-since-ref` → `CATime`.
|
|
340
|
+
|
|
341
|
+
Encode (write) runs in reverse:
|
|
342
|
+
|
|
343
|
+
1. `encode_time` — `CATime` → `count-since-ref`.
|
|
344
|
+
2. `encode_scale` — `round((v − offset) / scale)` for integer var types.
|
|
345
|
+
3. `encode_fill` — `UNDEF` → fill sentinel (in the user's unsigned space).
|
|
346
|
+
4. `encode_unsigned` — reinterpret unsigned user values back to signed on-disk bits.
|
|
347
|
+
|
|
348
|
+
(`valid_range` has no defined encode direction in CF, and the codec doesn't invent one.)
|
|
349
|
+
|
|
350
|
+
## Debug / inspect a variable's decode
|
|
351
|
+
|
|
352
|
+
```ruby
|
|
353
|
+
NC::File.open("data.nc") do |nc|
|
|
354
|
+
v = nc["temp"]
|
|
355
|
+
v.atts # everything the codec might read
|
|
356
|
+
v.codec # which codec will run
|
|
357
|
+
|
|
358
|
+
raw = v.raw # bypass codec → the on-disk values
|
|
359
|
+
decoded = v.to_ca # === v.codec.decode(v, v.raw) (identity)
|
|
360
|
+
|
|
361
|
+
# diff them cell by cell to see what the codec actually changed
|
|
362
|
+
end
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
The identity `v.to_ca == v.codec.decode(v, v.raw)` holds by construction, so `v.raw` is the canonical descent point when you want to work below the codec — no `instance_variable_get` needed.
|
|
366
|
+
|
|
367
|
+
You can also open the file with `codec: :identity` to skip decoding for every read in the block:
|
|
368
|
+
|
|
369
|
+
```ruby
|
|
370
|
+
NC::File.open("data.nc", codec: :identity) do |nc|
|
|
371
|
+
nc["temp"].to_ca # raw bytes, uniformly for every variable
|
|
372
|
+
end
|
|
373
|
+
```
|
data/docs/05_groups.md
ADDED
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
# Groups (NetCDF-4)
|
|
2
|
+
|
|
3
|
+
NetCDF-4 files can organize their contents in a tree of **groups** — much like directories in a filesystem. Each group has its own dimensions, variables, and global attributes; nested groups form a hierarchy.
|
|
4
|
+
|
|
5
|
+
The classic NetCDF-3 model is a single root group; every file has one, so in this library **every `File` IS a Group** (the root group). Everything you can do on `File` — `def_dim`, `def_var`, `def_att`, `nc[name]`, `nc[name] = ...` — is a `Group` method, plus subgroup navigation.
|
|
6
|
+
|
|
7
|
+
## Creating groups
|
|
8
|
+
|
|
9
|
+
Groups only work in NetCDF-4 format, so pass `netcdf4: true`:
|
|
10
|
+
|
|
11
|
+
```ruby
|
|
12
|
+
require "carray-netcdf"
|
|
13
|
+
NC = CA::NetCDF
|
|
14
|
+
|
|
15
|
+
NC::File.create("model.nc", netcdf4: true) do |nc|
|
|
16
|
+
nc.def_dim("time", 12)
|
|
17
|
+
|
|
18
|
+
ocean = nc.def_group("ocean")
|
|
19
|
+
ocean.def_dim("depth", 50)
|
|
20
|
+
ocean.def_var("temperature", :float, ["time", "depth"]) # 'time' inherited from root
|
|
21
|
+
|
|
22
|
+
atm = nc.def_group("atmosphere")
|
|
23
|
+
atm.def_dim("altitude", 30)
|
|
24
|
+
atm.def_var("pressure", :float, ["time", "altitude"])
|
|
25
|
+
|
|
26
|
+
# Nested subgroups
|
|
27
|
+
surface = ocean.def_group("surface")
|
|
28
|
+
surface.def_var("sst", :float, ["time"])
|
|
29
|
+
end
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
`def_group` returns the new `Group` and stores it under `parent.groups[name]`.
|
|
33
|
+
|
|
34
|
+
## Navigating groups
|
|
35
|
+
|
|
36
|
+
```ruby
|
|
37
|
+
NC::File.open("model.nc") do |nc|
|
|
38
|
+
nc.groups # => {"ocean" => Group, "atmosphere" => Group}
|
|
39
|
+
nc.group("ocean") # => Group
|
|
40
|
+
nc.group?("ocean") # => true
|
|
41
|
+
|
|
42
|
+
ocean = nc.group("ocean")
|
|
43
|
+
ocean.name # => "ocean"
|
|
44
|
+
ocean.path # => "/ocean"
|
|
45
|
+
ocean.parent # => the File (root Group)
|
|
46
|
+
ocean.root? # => false
|
|
47
|
+
|
|
48
|
+
surface = ocean.group("surface")
|
|
49
|
+
surface.path # => "/ocean/surface"
|
|
50
|
+
|
|
51
|
+
nc.name # => "/" (netcdf's root name)
|
|
52
|
+
nc.path # => "/"
|
|
53
|
+
nc.root? # => true
|
|
54
|
+
end
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Accessing variables in groups
|
|
58
|
+
|
|
59
|
+
Three equivalent forms:
|
|
60
|
+
|
|
61
|
+
```ruby
|
|
62
|
+
NC::File.open("model.nc") do |nc|
|
|
63
|
+
# (1) navigate then subscript
|
|
64
|
+
temp = nc.group("ocean")["temperature"]
|
|
65
|
+
|
|
66
|
+
# (2) relative path
|
|
67
|
+
temp = nc["ocean/temperature"]
|
|
68
|
+
|
|
69
|
+
# (3) absolute path
|
|
70
|
+
temp = nc["/ocean/temperature"]
|
|
71
|
+
|
|
72
|
+
# All three are the same Var object.
|
|
73
|
+
end
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Full-path lookup works on any `Group` as the receiver:
|
|
77
|
+
|
|
78
|
+
```ruby
|
|
79
|
+
ocean = nc.group("ocean")
|
|
80
|
+
ocean["surface/sst"] # relative to ocean -> Var
|
|
81
|
+
ocean["/ocean/temperature"] # absolute from root -> Var
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
## Writing to a variable in a group
|
|
85
|
+
|
|
86
|
+
Same pattern:
|
|
87
|
+
|
|
88
|
+
```ruby
|
|
89
|
+
NC::File.open("model.nc", :write) do |nc|
|
|
90
|
+
# via group
|
|
91
|
+
nc.group("ocean")["temperature"] = big_array
|
|
92
|
+
|
|
93
|
+
# via path
|
|
94
|
+
nc["/ocean/temperature", 0, nil] = one_slice
|
|
95
|
+
nc["/ocean/temperature", 0, 0] = 273.15 # single point
|
|
96
|
+
end
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## Dimension inheritance
|
|
100
|
+
|
|
101
|
+
NetCDF-4 lets a variable in a subgroup reference a dimension defined in any ancestor group. `def_var` resolves dimension names by walking up the tree:
|
|
102
|
+
|
|
103
|
+
```ruby
|
|
104
|
+
NC::File.create("nested.nc", netcdf4: true) do |nc|
|
|
105
|
+
nc.def_dim("time", 24) # defined at root
|
|
106
|
+
child = nc.def_group("child")
|
|
107
|
+
child.def_dim("station", 10) # defined in child
|
|
108
|
+
|
|
109
|
+
# temp uses one root-level dim and one child-local dim
|
|
110
|
+
child.def_var("temp", :float, ["time", "station"])
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
NC::File.open("nested.nc") do |nc|
|
|
114
|
+
t = nc["/child/temp"]
|
|
115
|
+
t.shape # => [24, 10]
|
|
116
|
+
t.dims.map(&:name) # => ["time", "station"]
|
|
117
|
+
t.dims[0].group.path # => "/" (inherited from root)
|
|
118
|
+
t.dims[1].group.path # => "/child" (defined in child)
|
|
119
|
+
end
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
**Per-group `dims`** returns only that group's local dims (not inherited):
|
|
123
|
+
|
|
124
|
+
```ruby
|
|
125
|
+
nc.dims.keys # => ["time"]
|
|
126
|
+
nc.group("child").dims.keys # => ["station"] (not "time")
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
Use `group.resolve_dim("time")` if you explicitly want the ancestor-walked lookup:
|
|
130
|
+
|
|
131
|
+
```ruby
|
|
132
|
+
nc.group("child").resolve_dim("time") # => the Dim from root
|
|
133
|
+
nc.group("child").resolve_dim("bogus") # => raises KeyError
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
## Per-group attributes
|
|
137
|
+
|
|
138
|
+
Every group has its own global-attribute set (netcdf-4 changes `NC_GLOBAL` from a single set to per-group). `def_att` / `atts` / `att` on a Group operate on that set:
|
|
139
|
+
|
|
140
|
+
```ruby
|
|
141
|
+
NC::File.create("attrs.nc", netcdf4: true) do |nc|
|
|
142
|
+
nc.def_att("title", "Root title")
|
|
143
|
+
|
|
144
|
+
ocean = nc.def_group("ocean")
|
|
145
|
+
ocean.def_att("title", "Ocean subgroup title")
|
|
146
|
+
ocean.def_att("depth_convention", "positive down")
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
NC::File.open("attrs.nc") do |nc|
|
|
150
|
+
nc.att("title") # => "Root title"
|
|
151
|
+
nc.group("ocean").att("title") # => "Ocean subgroup title"
|
|
152
|
+
nc.group("ocean").att("depth_convention") # => "positive down"
|
|
153
|
+
nc.att("depth_convention") # => nil (not on root)
|
|
154
|
+
end
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
## Codecs are inherited from the root
|
|
158
|
+
|
|
159
|
+
The codec is a whole-file policy — `Conventions` at the root determines which codec applies to every variable, including those in subgroups. `Group#codec` delegates to the File.
|
|
160
|
+
|
|
161
|
+
```ruby
|
|
162
|
+
NC::File.create("cf_groups.nc", netcdf4: true) do |nc|
|
|
163
|
+
nc.def_att("Conventions", "CF-1.7")
|
|
164
|
+
nc.def_dim("time", 24)
|
|
165
|
+
|
|
166
|
+
ocean = nc.def_group("ocean")
|
|
167
|
+
ocean.def_var("time_utc", :int, ["time"],
|
|
168
|
+
units: "hours since 2022-01-01T00:00:00Z")
|
|
169
|
+
ocean["time_utc"] = CArray.time_series("2022-01-01T00:00:00Z", count: 24, unit: :h)
|
|
170
|
+
end
|
|
171
|
+
|
|
172
|
+
NC::File.open("cf_groups.nc") do |nc|
|
|
173
|
+
nc.codec.is_a?(NC::Codec::CF) # => true
|
|
174
|
+
nc.group("ocean").codec.equal?(nc.codec) # => true (same instance)
|
|
175
|
+
|
|
176
|
+
t = nc["/ocean/time_utc"].to_ca
|
|
177
|
+
t.is_a?(CATime) # => true (decoded via CF)
|
|
178
|
+
t[3].to_s # => "2022-01-01T03:00:00Z"
|
|
179
|
+
end
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
## Introspection reference
|
|
183
|
+
|
|
184
|
+
| Method | Returns |
|
|
185
|
+
|---|---|
|
|
186
|
+
| `group.name` | `"foo"` (or `"/"` for root) |
|
|
187
|
+
| `group.path` | `"/foo/bar"` (full hierarchy path) |
|
|
188
|
+
| `group.parent` | parent Group, or `nil` for root |
|
|
189
|
+
| `group.root?` | Boolean |
|
|
190
|
+
| `group.file` | the enclosing File (root Group) |
|
|
191
|
+
| `group.ncid` | raw group ncid (int) |
|
|
192
|
+
| `group.dims` | `{String => Dim}` — this group only |
|
|
193
|
+
| `group.vars` | `{String => Var}` — this group only |
|
|
194
|
+
| `group.groups` | `{String => Group}` — direct children |
|
|
195
|
+
| `group.atts` | `{String => value}` (frozen) — this group's globals |
|
|
196
|
+
| `group.dim(name)` / `.var(name)` / `.group(name)` | one lookup |
|
|
197
|
+
| `group.dim?(name)` / `.var?(name)` / `.group?(name)` | Boolean |
|
|
198
|
+
| `group.att(name)` | one attribute value |
|
|
199
|
+
| `group.resolve_dim(name)` | walk up ancestors |
|
|
200
|
+
| `group.codec` | codec (same as `file.codec`) |
|