opencode-skills-collection 4.0.72 → 4.0.73
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled-skills/.antigravity-install-manifest.json +7 -1
- package/bundled-skills/agent-memory-discipline/SKILL.md +204 -0
- package/bundled-skills/agent-memory-discipline/references/entry-format.md +37 -0
- package/bundled-skills/antigravity-maintainer-batch-release/SKILL.md +2 -2
- package/bundled-skills/jev-social/SKILL.md +5 -5
- package/bundled-skills/noaa-radar-satellite-fetching/SKILL.md +253 -0
- package/bundled-skills/weather-data-lifecycle-management/SKILL.md +159 -0
- package/bundled-skills/weather-data-reproducibility/SKILL.md +292 -0
- package/bundled-skills/weather-model-run-discovery/SKILL.md +241 -0
- package/bundled-skills/weather-pipeline-performance-diagnosis/SKILL.md +184 -0
- package/package.json +1 -1
- package/skills_index.json +188 -1
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: weather-data-lifecycle-management
|
|
3
|
+
description: "Manage ownership, retention, and cleanup of downloaded weather data across one-shot jobs, interactive viewers, caches, failures, and cancellation."
|
|
4
|
+
category: data
|
|
5
|
+
risk: safe
|
|
6
|
+
source: self
|
|
7
|
+
source_type: self
|
|
8
|
+
date_added: "2026-09-24"
|
|
9
|
+
author: ShianMike
|
|
10
|
+
tags: [weather, data-lifecycle, cleanup, caching, desktop-apps]
|
|
11
|
+
tools: [claude, cursor, gemini, codex]
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Weather Data Lifecycle Management
|
|
15
|
+
|
|
16
|
+
## Overview
|
|
17
|
+
|
|
18
|
+
Keep downloaded and derived weather data for exactly as long as its consumer
|
|
19
|
+
needs it. Make ownership explicit so one-shot jobs clean promptly, interactive
|
|
20
|
+
views retain their inputs until close, and shared caches are never deleted as
|
|
21
|
+
request cleanup.
|
|
22
|
+
|
|
23
|
+
This skill governs lifetime and ownership. It does not choose a provider,
|
|
24
|
+
decode scientific data, or define a cache eviction policy by itself.
|
|
25
|
+
|
|
26
|
+
## When to Use This Skill
|
|
27
|
+
|
|
28
|
+
- A completed render leaves large temporary files behind.
|
|
29
|
+
- An interactive view fails because its backing data was deleted too early.
|
|
30
|
+
- Cancellation or exceptions leak partial downloads and derived files.
|
|
31
|
+
- Request-owned files and a shared user cache are mixed together.
|
|
32
|
+
- Background workers transfer ownership of data to a UI or later processing
|
|
33
|
+
stage.
|
|
34
|
+
|
|
35
|
+
## Classify Every Path
|
|
36
|
+
|
|
37
|
+
Assign each file or directory one owner and one lifetime:
|
|
38
|
+
|
|
39
|
+
| Class | Owner | Delete when |
|
|
40
|
+
| --- | --- | --- |
|
|
41
|
+
| Request workspace | One operation | Operation finishes or fails after durable outputs are saved |
|
|
42
|
+
| Interactive-session data | Viewer or session | Final consumer closes or display creation fails |
|
|
43
|
+
| Shared cache | Cache manager | Explicit eviction policy permits it |
|
|
44
|
+
| User export | User | Never as automatic request cleanup |
|
|
45
|
+
| Partial file | Active writer | Failure, cancellation, or successful atomic promotion |
|
|
46
|
+
|
|
47
|
+
If ownership cannot be stated, do not delete the path. Resolve ownership first.
|
|
48
|
+
|
|
49
|
+
## One-Shot Workflow
|
|
50
|
+
|
|
51
|
+
Use a request-specific temporary directory, keep all transient downloads and
|
|
52
|
+
intermediate products inside it, and copy or atomically move only the requested
|
|
53
|
+
artifact to its durable destination.
|
|
54
|
+
|
|
55
|
+
```python
|
|
56
|
+
from pathlib import Path
|
|
57
|
+
from tempfile import TemporaryDirectory
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
def render_once(destination):
|
|
61
|
+
destination = Path(destination)
|
|
62
|
+
destination.parent.mkdir(parents=True, exist_ok=True)
|
|
63
|
+
|
|
64
|
+
with TemporaryDirectory(prefix="weather-job-") as workspace:
|
|
65
|
+
workspace = Path(workspace)
|
|
66
|
+
source = fetch_into(workspace)
|
|
67
|
+
temporary_output = render(source, workspace / "result.png")
|
|
68
|
+
temporary_output.replace(destination)
|
|
69
|
+
|
|
70
|
+
return destination
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
The final artifact must live outside the temporary directory. If replacement
|
|
74
|
+
across filesystems is not atomic, copy to a temporary file beside the final
|
|
75
|
+
destination, verify it, then replace locally.
|
|
76
|
+
|
|
77
|
+
## Interactive Workflow
|
|
78
|
+
|
|
79
|
+
An interactive consumer outlives the worker that created its data. Transfer
|
|
80
|
+
ownership only after the viewer or session is successfully created:
|
|
81
|
+
|
|
82
|
+
1. The worker owns the request workspace during acquisition.
|
|
83
|
+
2. If acquisition or display creation fails, the worker cleans it immediately.
|
|
84
|
+
3. On successful display, attach one idempotent cleanup callback to the real
|
|
85
|
+
consumer close or destroyed event.
|
|
86
|
+
4. The consumer becomes the owner; the worker must not also delete the data in
|
|
87
|
+
its ordinary `finally` path.
|
|
88
|
+
5. If multiple consumers share the same data, release it only after the final
|
|
89
|
+
lease closes.
|
|
90
|
+
|
|
91
|
+
Do not tie cleanup to a temporary dialog, local variable, or signal that can
|
|
92
|
+
fire before the actual consumer is finished.
|
|
93
|
+
|
|
94
|
+
## Failure and Cancellation Rules
|
|
95
|
+
|
|
96
|
+
- Keep incomplete downloads under a distinct suffix or directory so they cannot
|
|
97
|
+
be mistaken for valid data.
|
|
98
|
+
- Close datasets, file handles, and memory maps before removing their paths.
|
|
99
|
+
- Put request-owned cleanup in `finally`, but exclude resources whose ownership
|
|
100
|
+
was successfully transferred.
|
|
101
|
+
- Make cleanup idempotent; a cancellation signal and a close event may race.
|
|
102
|
+
- Preserve a durable partial-result manifest when the user can retry remaining
|
|
103
|
+
work.
|
|
104
|
+
- On cleanup failure, report the exact owned path without deleting broader
|
|
105
|
+
parent directories.
|
|
106
|
+
|
|
107
|
+
## Shared Cache Boundary
|
|
108
|
+
|
|
109
|
+
- A request may read or populate a shared cache but does not own the cache root.
|
|
110
|
+
- Promote validated entries atomically; never expose a partial file as a hit.
|
|
111
|
+
- Use cache identity and eviction metadata rather than deleting files by age or
|
|
112
|
+
filename guesswork inside request code.
|
|
113
|
+
- Do not place user exports inside a directory that normal cache cleanup owns.
|
|
114
|
+
- Test that two concurrent consumers do not remove data still leased by the
|
|
115
|
+
other.
|
|
116
|
+
|
|
117
|
+
## Verification Checklist
|
|
118
|
+
|
|
119
|
+
- Every created path has a named owner and deletion event.
|
|
120
|
+
- One-shot operations retain only requested durable outputs.
|
|
121
|
+
- Interactive data remains available until the final consumer closes.
|
|
122
|
+
- Display-creation failure cleans immediately.
|
|
123
|
+
- Exceptions and cancellation remove partial request-owned files.
|
|
124
|
+
- Cleanup is idempotent and closes open handles first.
|
|
125
|
+
- Shared caches and user exports are outside request cleanup scope.
|
|
126
|
+
- Tests exercise success, failure, cancellation, and close-event paths.
|
|
127
|
+
|
|
128
|
+
## Security & Safety Notes
|
|
129
|
+
|
|
130
|
+
- Resolve and verify cleanup targets before recursive deletion.
|
|
131
|
+
- Never delete a home directory, workspace root, shared cache root, or user-
|
|
132
|
+
selected directory as a computed fallback.
|
|
133
|
+
- Avoid globs and unresolved environment variables for destructive cleanup.
|
|
134
|
+
- Do not follow untrusted symlinks or junctions outside the owned workspace.
|
|
135
|
+
- Keep sensitive temporary data in an access-controlled location and remove it
|
|
136
|
+
when its approved lifetime ends.
|
|
137
|
+
- Report material cleanup failures instead of silently leaving private data.
|
|
138
|
+
|
|
139
|
+
## Common Pitfalls
|
|
140
|
+
|
|
141
|
+
- **The image renders but source data leaks:** Cleanup is absent from the
|
|
142
|
+
one-shot `finally` path or data was written outside the request workspace.
|
|
143
|
+
- **The viewer opens and then breaks:** The worker deleted data immediately
|
|
144
|
+
after emitting the viewer. Transfer ownership to the viewer close event.
|
|
145
|
+
- **Cancellation deletes useful completed work:** Durable artifacts and
|
|
146
|
+
request-owned intermediates share a directory. Separate them.
|
|
147
|
+
- **A cache disappears:** Request cleanup treated a shared cache as owned data.
|
|
148
|
+
Delegate eviction to the cache manager.
|
|
149
|
+
- **Windows refuses deletion:** A dataset or mapped file is still open. Close
|
|
150
|
+
every consumer before cleanup.
|
|
151
|
+
|
|
152
|
+
## Limitations
|
|
153
|
+
|
|
154
|
+
- Multi-process leases require coordination beyond an in-memory reference
|
|
155
|
+
count.
|
|
156
|
+
- Some decoders create files outside their requested directory; configure or
|
|
157
|
+
inventory those outputs before promising complete cleanup.
|
|
158
|
+
- Lifecycle correctness prevents leaks and premature deletion but does not
|
|
159
|
+
validate the scientific contents of retained data.
|
|
@@ -0,0 +1,292 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: weather-data-reproducibility
|
|
3
|
+
description: "Record and verify provenance manifests for weather-data inputs and derived artifacts, including object identity, selections, software versions, transformations, and hashes."
|
|
4
|
+
category: data
|
|
5
|
+
risk: safe
|
|
6
|
+
source: self
|
|
7
|
+
source_type: self
|
|
8
|
+
date_added: "2026-09-24"
|
|
9
|
+
author: ShianMike
|
|
10
|
+
tags: [weather, reproducibility, provenance, checksums, grib2, netcdf]
|
|
11
|
+
tools: [claude, cursor, gemini, codex]
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Weather Data Reproducibility
|
|
15
|
+
|
|
16
|
+
## Overview
|
|
17
|
+
|
|
18
|
+
Create a compact provenance manifest that identifies the exact weather inputs,
|
|
19
|
+
subsets, software, transformations, and output artifacts used by a run. Verify
|
|
20
|
+
the manifest before claiming that a result can be replayed or reproduced.
|
|
21
|
+
|
|
22
|
+
This skill records and checks lineage. It does not discover or download data by
|
|
23
|
+
itself; pair it with the appropriate model, observation, radar, or satellite
|
|
24
|
+
fetching skill.
|
|
25
|
+
|
|
26
|
+
## When to Use This Skill
|
|
27
|
+
|
|
28
|
+
- A map, sounding, time series, dataset, or benchmark must be auditable.
|
|
29
|
+
- Weather inputs came from mutable URLs, mirrors, byte-range subsets, or caches.
|
|
30
|
+
- A result must distinguish model initialization, forecast lead, and valid time.
|
|
31
|
+
- Temporary GRIB2, NetCDF, radar, satellite, or observation files will be
|
|
32
|
+
deleted after processing.
|
|
33
|
+
- Two outputs differ and their inputs or processing environment must be compared.
|
|
34
|
+
- A publication or release needs machine-readable data and software provenance.
|
|
35
|
+
|
|
36
|
+
Do not claim bit-for-bit reproducibility merely because a source URL and run
|
|
37
|
+
time were logged.
|
|
38
|
+
|
|
39
|
+
## Reproducibility Contract
|
|
40
|
+
|
|
41
|
+
Decide which claim the workflow supports:
|
|
42
|
+
|
|
43
|
+
- **Traceable:** The source and processing history can be audited.
|
|
44
|
+
- **Replayable:** The same identified inputs can still be retrieved and the
|
|
45
|
+
recorded procedure can be rerun.
|
|
46
|
+
- **Bitwise reproducible:** The rerun is expected to produce identical bytes in
|
|
47
|
+
a pinned environment.
|
|
48
|
+
- **Scientifically reproducible:** Equivalent inputs and methods produce the
|
|
49
|
+
same scientific conclusion within a stated tolerance.
|
|
50
|
+
|
|
51
|
+
Record the intended claim and its tolerance. Mutable remote objects, floating
|
|
52
|
+
software versions, lossy images, parallel reductions, and platform-dependent
|
|
53
|
+
code can make the stronger claims impossible.
|
|
54
|
+
|
|
55
|
+
## Minimum Manifest
|
|
56
|
+
|
|
57
|
+
Use a versioned JSON document. Keep request intent separate from the source that
|
|
58
|
+
was actually resolved.
|
|
59
|
+
|
|
60
|
+
```json
|
|
61
|
+
{
|
|
62
|
+
"schema_version": 1,
|
|
63
|
+
"created_utc": "2026-09-18T09:00:00Z",
|
|
64
|
+
"claim": {"level": "replayable", "tolerance": null},
|
|
65
|
+
"request": {
|
|
66
|
+
"dataset": "hrrr",
|
|
67
|
+
"product": "prs",
|
|
68
|
+
"requested_valid_time": "2026-09-18T12:00:00Z",
|
|
69
|
+
"variables": ["temperature", "relative_humidity"]
|
|
70
|
+
},
|
|
71
|
+
"resolved": {
|
|
72
|
+
"initialization": "2026-09-18T06:00:00Z",
|
|
73
|
+
"forecast_hour": 6,
|
|
74
|
+
"valid_time": "2026-09-18T12:00:00Z",
|
|
75
|
+
"provider": "aws"
|
|
76
|
+
},
|
|
77
|
+
"inputs": [
|
|
78
|
+
{
|
|
79
|
+
"role": "pressure_fields",
|
|
80
|
+
"uri": "s3://bucket/exact-object-key",
|
|
81
|
+
"object_identity": {
|
|
82
|
+
"etag": "opaque-identity",
|
|
83
|
+
"last_modified": "2026-09-18T07:01:02Z",
|
|
84
|
+
"size_bytes": 987654321
|
|
85
|
+
},
|
|
86
|
+
"selection": {
|
|
87
|
+
"inventory_uri": "s3://bucket/exact-object-key.idx",
|
|
88
|
+
"inventory_sha256": "HEX_DIGEST",
|
|
89
|
+
"byte_ranges": [[1000, 1999], [5000, 6999]]
|
|
90
|
+
},
|
|
91
|
+
"materialized_sha256": "HEX_DIGEST"
|
|
92
|
+
}
|
|
93
|
+
],
|
|
94
|
+
"processing": {
|
|
95
|
+
"software": {"application": "1.2.3", "python": "3.13.7"},
|
|
96
|
+
"parameters": {"point": [35.22, -97.44], "nearest_method": "model-aware"},
|
|
97
|
+
"transformations": ["decode GRIB2", "convert K to degC"]
|
|
98
|
+
},
|
|
99
|
+
"artifacts": [
|
|
100
|
+
{
|
|
101
|
+
"path": "sounding.png",
|
|
102
|
+
"media_type": "image/png",
|
|
103
|
+
"size_bytes": 123456,
|
|
104
|
+
"sha256": "HEX_DIGEST"
|
|
105
|
+
}
|
|
106
|
+
]
|
|
107
|
+
}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Use `null` for an intentionally absent value and omit fields whose meaning is
|
|
111
|
+
unknown. Never fill a required-looking field with a guess.
|
|
112
|
+
|
|
113
|
+
## Identify Every Input
|
|
114
|
+
|
|
115
|
+
For each remote object or API response, record as available:
|
|
116
|
+
|
|
117
|
+
- provider, dataset, endpoint, bucket, and exact key or immutable URL;
|
|
118
|
+
- version ID, generation number, release, or dataset revision;
|
|
119
|
+
- ETag as opaque object identity, `Last-Modified`, and content length;
|
|
120
|
+
- provider-supplied checksum and checksum algorithm;
|
|
121
|
+
- a locally computed SHA-256 for downloaded or materialized bytes;
|
|
122
|
+
- response media type and content encoding;
|
|
123
|
+
- retrieval time and successful fallback provider;
|
|
124
|
+
- license or attribution identifier when required.
|
|
125
|
+
|
|
126
|
+
An S3 ETag is not always an MD5 digest, particularly for multipart or encrypted
|
|
127
|
+
objects. Store it for identity and compute a cryptographic content hash when
|
|
128
|
+
the actual bytes must be verified.
|
|
129
|
+
|
|
130
|
+
## Record the Selection
|
|
131
|
+
|
|
132
|
+
The source object alone is insufficient when only part of it was used. Record:
|
|
133
|
+
|
|
134
|
+
- GRIB2 inventory URI and inventory content hash;
|
|
135
|
+
- exact inventory rows, search expression, and inclusive byte ranges;
|
|
136
|
+
- Zarr group, array names, chunk keys, and coordinate slices;
|
|
137
|
+
- API query parameters and a hash or retained copy of the raw response;
|
|
138
|
+
- station identifier scheme and observation time interval;
|
|
139
|
+
- radar site/product and scan interval;
|
|
140
|
+
- satellite platform, product, sector, channel, and scan interval;
|
|
141
|
+
- requested coordinates plus the actual selected grid point and distance.
|
|
142
|
+
|
|
143
|
+
Preserve the order in which selected binary ranges were assembled. Hash the
|
|
144
|
+
materialized subset separately from the full remote object's identity.
|
|
145
|
+
|
|
146
|
+
## Record Weather Time Semantics
|
|
147
|
+
|
|
148
|
+
Use explicit UTC timestamps and name their meaning:
|
|
149
|
+
|
|
150
|
+
- models: initialization, forecast lead, and valid time;
|
|
151
|
+
- station observations: observation, correction, and ingestion time;
|
|
152
|
+
- radar: volume or product scan start and end;
|
|
153
|
+
- satellite: scan start, scan end, and file creation time;
|
|
154
|
+
- retrieval: when the client fetched the metadata and bytes.
|
|
155
|
+
|
|
156
|
+
Do not replace these with one ambiguous `timestamp` field. Record the calendar
|
|
157
|
+
and leap-second handling if the source or application requires it.
|
|
158
|
+
|
|
159
|
+
## Record Processing and Environment
|
|
160
|
+
|
|
161
|
+
Capture only details that can change the result:
|
|
162
|
+
|
|
163
|
+
- application version and source commit, including dirty-tree status;
|
|
164
|
+
- decoder and scientific library versions;
|
|
165
|
+
- runtime and operating-system or architecture details when relevant;
|
|
166
|
+
- command arguments or structured parameters;
|
|
167
|
+
- unit conversions, QC decisions, interpolation, coordinate selection, and
|
|
168
|
+
aggregation rules;
|
|
169
|
+
- deterministic random seed when randomness is present;
|
|
170
|
+
- container image digest or environment lock-file hash when available.
|
|
171
|
+
|
|
172
|
+
Do not dump an entire environment full of unrelated packages merely because it
|
|
173
|
+
is easy. Prefer a lock file plus the versions of software that actually touched
|
|
174
|
+
the data.
|
|
175
|
+
|
|
176
|
+
## Hash and Write Atomically
|
|
177
|
+
|
|
178
|
+
The Python standard library is sufficient for local artifact hashes and a
|
|
179
|
+
canonical, atomically replaced JSON manifest:
|
|
180
|
+
|
|
181
|
+
```python
|
|
182
|
+
import hashlib
|
|
183
|
+
import json
|
|
184
|
+
import os
|
|
185
|
+
from pathlib import Path
|
|
186
|
+
|
|
187
|
+
|
|
188
|
+
def sha256_file(path, chunk_size=1024 * 1024):
|
|
189
|
+
digest = hashlib.sha256()
|
|
190
|
+
with Path(path).open("rb") as handle:
|
|
191
|
+
for chunk in iter(lambda: handle.read(chunk_size), b""):
|
|
192
|
+
digest.update(chunk)
|
|
193
|
+
return digest.hexdigest()
|
|
194
|
+
|
|
195
|
+
|
|
196
|
+
def write_manifest(path, manifest):
|
|
197
|
+
path = Path(path)
|
|
198
|
+
temporary = path.with_suffix(path.suffix + ".tmp")
|
|
199
|
+
payload = json.dumps(
|
|
200
|
+
manifest, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
|
201
|
+
) + "\n"
|
|
202
|
+
temporary.write_text(payload, encoding="utf-8", newline="\n")
|
|
203
|
+
os.replace(temporary, path)
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Hash artifacts only after their writers are closed. If the manifest itself must
|
|
207
|
+
be signed, sign the canonical bytes using the project's established signing
|
|
208
|
+
workflow rather than inventing a custom scheme.
|
|
209
|
+
|
|
210
|
+
## Verify and Replay
|
|
211
|
+
|
|
212
|
+
Before replay:
|
|
213
|
+
|
|
214
|
+
1. Validate the manifest schema version and required fields.
|
|
215
|
+
2. Re-resolve every immutable object identity. Fail if an object changed unless
|
|
216
|
+
the reproducibility contract explicitly permits an equivalent replacement.
|
|
217
|
+
3. Download or locate inputs and compare size plus a real checksum.
|
|
218
|
+
4. Reapply the recorded selection and confirm the materialized subset hash.
|
|
219
|
+
5. Recreate the pinned processing environment and transformations.
|
|
220
|
+
6. Produce artifacts in a new output directory.
|
|
221
|
+
7. Compare artifact hashes for bitwise claims or documented scientific metrics
|
|
222
|
+
and tolerances for scientific claims.
|
|
223
|
+
8. Write a new run manifest that references, but does not overwrite, the
|
|
224
|
+
original.
|
|
225
|
+
|
|
226
|
+
Report the first mismatch with its role, expected value, and actual value.
|
|
227
|
+
|
|
228
|
+
## Temporary Data Lifecycle
|
|
229
|
+
|
|
230
|
+
- Write the final artifact and its manifest before deleting request-owned input
|
|
231
|
+
files.
|
|
232
|
+
- Keep cleanup in `finally` so failures and cancellation do not leak large
|
|
233
|
+
GRIB2, NetCDF, radar, or observation files.
|
|
234
|
+
- Never delete a shared user cache as request cleanup.
|
|
235
|
+
- If raw inputs are deleted and the remote source is mutable or short-lived,
|
|
236
|
+
label the run traceable rather than replayable.
|
|
237
|
+
- A manifest is small and should normally outlive transient downloads.
|
|
238
|
+
|
|
239
|
+
## Verification Checklist
|
|
240
|
+
|
|
241
|
+
- Request intent and resolved data identity are separate.
|
|
242
|
+
- Every timestamp is UTC and has explicit semantics.
|
|
243
|
+
- Every input has an exact locator, identity metadata, and content hash when
|
|
244
|
+
bytes were materialized.
|
|
245
|
+
- Subset rules, inventory identity, byte ranges, and selected coordinates are
|
|
246
|
+
recorded.
|
|
247
|
+
- Software versions and result-changing transformations are explicit.
|
|
248
|
+
- Final artifacts have size, media type, and SHA-256.
|
|
249
|
+
- The claimed reproducibility level matches what can actually be replayed.
|
|
250
|
+
- The manifest contains no secrets or machine-specific private information.
|
|
251
|
+
|
|
252
|
+
## Security & Safety Notes
|
|
253
|
+
|
|
254
|
+
- Never record credentials, authorization headers, cookies, signed-query
|
|
255
|
+
strings, private bucket names, or secret environment variables.
|
|
256
|
+
- Redact user names and unnecessary absolute local paths before sharing a
|
|
257
|
+
manifest.
|
|
258
|
+
- Treat a hash as an integrity value, not proof that an untrusted input is safe.
|
|
259
|
+
- Validate manifest paths before opening files; do not allow `..` traversal or
|
|
260
|
+
writes outside the intended output directory.
|
|
261
|
+
- Keep TLS verification enabled when re-fetching inputs.
|
|
262
|
+
- Preserve license, attribution, redistribution, and access restrictions.
|
|
263
|
+
|
|
264
|
+
## Common Pitfalls
|
|
265
|
+
|
|
266
|
+
- **ETag was labeled SHA-256 or MD5:** ETag semantics depend on the provider and
|
|
267
|
+
upload method. Store it as identity and compute a real local hash.
|
|
268
|
+
- **A model result cannot be recreated:** Initialization and forecast lead were
|
|
269
|
+
collapsed into valid time. Record all three.
|
|
270
|
+
- **The same URI returns different bytes:** The remote object was mutable and no
|
|
271
|
+
version or checksum was pinned. Downgrade the claim or retain the input.
|
|
272
|
+
- **A GRIB subset differs:** The inventory version, selected rows, or inclusive
|
|
273
|
+
byte ranges were omitted.
|
|
274
|
+
- **The artifact hash changes across systems:** The process is scientifically,
|
|
275
|
+
but not bitwise, reproducible. Define and test a numerical tolerance.
|
|
276
|
+
|
|
277
|
+
## Limitations
|
|
278
|
+
|
|
279
|
+
- A manifest cannot restore data that was deleted from a mutable source unless
|
|
280
|
+
the input bytes or an immutable archive were retained.
|
|
281
|
+
- Exact replay may require licensed software, hardware, or provider access not
|
|
282
|
+
captured in a public manifest.
|
|
283
|
+
- Provenance establishes lineage and integrity; it does not establish forecast
|
|
284
|
+
skill, observational truth, or scientific validity.
|
|
285
|
+
|
|
286
|
+
## Additional Resources
|
|
287
|
+
|
|
288
|
+
- [W3C PROV overview](https://www.w3.org/TR/prov-overview/)
|
|
289
|
+
- [Amazon S3 object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)
|
|
290
|
+
- [Amazon S3 checksum guidance](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
|
|
291
|
+
- [CF metadata conventions](https://cfconventions.org/)
|
|
292
|
+
- [NetCDF Climate and Forecast conventions](https://www.ncei.noaa.gov/products/netcdf)
|
|
@@ -0,0 +1,241 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: weather-model-run-discovery
|
|
3
|
+
description: "Resolve the newest complete numerical weather prediction cycle and forecast objects across provider mirrors without downloading full payloads."
|
|
4
|
+
category: data
|
|
5
|
+
risk: safe
|
|
6
|
+
source: self
|
|
7
|
+
source_type: self
|
|
8
|
+
date_added: "2026-09-24"
|
|
9
|
+
author: ShianMike
|
|
10
|
+
tags: [weather, nwp, model-cycle, aws-s3, nomads, herbie]
|
|
11
|
+
tools: [claude, cursor, gemini, codex]
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Weather Model Run Discovery
|
|
15
|
+
|
|
16
|
+
## Overview
|
|
17
|
+
|
|
18
|
+
Resolve a numerical weather prediction run before downloading model fields.
|
|
19
|
+
Probe exact objects or narrow prefixes, verify a request-specific completion
|
|
20
|
+
contract, and return the newest qualifying initialization with its source
|
|
21
|
+
metadata.
|
|
22
|
+
|
|
23
|
+
This skill decides *which run and objects exist*. Use
|
|
24
|
+
`weather-model-data-fetching` afterward to transfer or subset the data.
|
|
25
|
+
|
|
26
|
+
## When to Use This Skill
|
|
27
|
+
|
|
28
|
+
- An application needs the newest usable GFS, GEFS, HRRR, RAP, NAM, or similar
|
|
29
|
+
model cycle.
|
|
30
|
+
- The nominal latest cycle returns `404`, has only early forecast hours, or is
|
|
31
|
+
still changing.
|
|
32
|
+
- Equivalent AWS, NOMADS, Google, Azure, or other mirrors must be compared
|
|
33
|
+
without silently mixing runs.
|
|
34
|
+
- A scheduled job needs a bounded fallback to an older complete cycle.
|
|
35
|
+
- The resolved run and provider must be recorded for later reproduction.
|
|
36
|
+
|
|
37
|
+
Do not activate this skill when the initialization is already fixed or when the
|
|
38
|
+
task is to download, decode, or interpret model variables.
|
|
39
|
+
|
|
40
|
+
## Define Complete Before Probing
|
|
41
|
+
|
|
42
|
+
Collect the following request contract:
|
|
43
|
+
|
|
44
|
+
- model, domain, product, and ensemble member;
|
|
45
|
+
- legal UTC cycle hours and cycle interval;
|
|
46
|
+
- required forecast hours;
|
|
47
|
+
- required companion objects, such as pressure and surface products;
|
|
48
|
+
- required inventory or index sidecars;
|
|
49
|
+
- provider priority and whether cross-provider fallback is allowed;
|
|
50
|
+
- maximum lookback, maximum probes, and freshness requirement.
|
|
51
|
+
|
|
52
|
+
A run is complete only when every sentinel required by the consumer exists.
|
|
53
|
+
Examples include:
|
|
54
|
+
|
|
55
|
+
- one product at one forecast hour for a single-image job;
|
|
56
|
+
- both pressure-level and surface files for a sounding;
|
|
57
|
+
- the terminal forecast hour needed by a time series;
|
|
58
|
+
- every required member for an ensemble statistic.
|
|
59
|
+
|
|
60
|
+
The presence of forecast hour zero does not prove that a run is complete.
|
|
61
|
+
|
|
62
|
+
## Discovery Workflow
|
|
63
|
+
|
|
64
|
+
1. Normalize the current time and all candidate initialization times to UTC.
|
|
65
|
+
2. Enumerate only legal cycles, newest first, within the lookback bound.
|
|
66
|
+
3. Build documented object keys for the exact model, product, member, and
|
|
67
|
+
forecast hour. Never recursively scan an entire bucket.
|
|
68
|
+
4. Probe exact objects with `HeadObject`, HTTP `HEAD`, or a narrow paginated
|
|
69
|
+
prefix listing. A probe should retrieve metadata, not the model payload.
|
|
70
|
+
5. Require every sentinel and sidecar in the completion contract. Reject zero-
|
|
71
|
+
length or obviously placeholder objects.
|
|
72
|
+
6. For a live mirror that may expose objects before publication finishes,
|
|
73
|
+
repeat the metadata probe after a short bounded interval and require stable
|
|
74
|
+
size and object identity.
|
|
75
|
+
7. Select the first complete candidate. A fallback provider must refer to the
|
|
76
|
+
same initialization, product, member, and forecast hour.
|
|
77
|
+
8. Emit a resolution record before starting the download.
|
|
78
|
+
|
|
79
|
+
Use a provider's documented cycle status page as supporting evidence, not as a
|
|
80
|
+
substitute for probing the exact objects the consumer requires.
|
|
81
|
+
|
|
82
|
+
## Minimal Resolver Example
|
|
83
|
+
|
|
84
|
+
Keep provider-specific key construction separate from the selection rule. This
|
|
85
|
+
example accepts a metadata-only `probe` callback so it can be used with S3,
|
|
86
|
+
HTTP, or a test fixture.
|
|
87
|
+
|
|
88
|
+
```python
|
|
89
|
+
from datetime import datetime, timedelta, timezone
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
def candidate_cycles(now, cycle_hours, lookback):
|
|
93
|
+
now = now.astimezone(timezone.utc).replace(minute=0, second=0, microsecond=0)
|
|
94
|
+
allowed = sorted(set(cycle_hours), reverse=True)
|
|
95
|
+
candidates = []
|
|
96
|
+
for days_back in range((lookback // 24) + 2):
|
|
97
|
+
day = (now - timedelta(days=days_back)).date()
|
|
98
|
+
for hour in allowed:
|
|
99
|
+
cycle = datetime(day.year, day.month, day.day, hour, tzinfo=timezone.utc)
|
|
100
|
+
if cycle <= now and now - cycle <= timedelta(hours=lookback):
|
|
101
|
+
candidates.append(cycle)
|
|
102
|
+
return sorted(set(candidates), reverse=True)
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
def resolve_latest(now, cycle_hours, lookback, required_objects, probe):
|
|
106
|
+
"""Return one complete run; probe(uri) returns metadata or None."""
|
|
107
|
+
attempts = []
|
|
108
|
+
for cycle in candidate_cycles(now, cycle_hours, lookback):
|
|
109
|
+
requested = required_objects(cycle)
|
|
110
|
+
found = {name: probe(uri) for name, uri in requested.items()}
|
|
111
|
+
missing = [name for name, metadata in found.items() if not metadata]
|
|
112
|
+
attempts.append({"cycle": cycle.isoformat(), "missing": missing})
|
|
113
|
+
if not missing:
|
|
114
|
+
return {
|
|
115
|
+
"initialization": cycle.isoformat(),
|
|
116
|
+
"objects": [
|
|
117
|
+
{"role": name, "uri": requested[name], **found[name]}
|
|
118
|
+
for name in requested
|
|
119
|
+
],
|
|
120
|
+
"attempts": attempts,
|
|
121
|
+
}
|
|
122
|
+
raise LookupError(f"no complete run within {lookback} hours: {attempts}")
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
The caller must make `required_objects` express the real completion contract;
|
|
126
|
+
checking a single convenient file defeats the purpose of discovery.
|
|
127
|
+
|
|
128
|
+
## Public S3 Diagnostics
|
|
129
|
+
|
|
130
|
+
For public NOAA Open Data buckets, inspect exact objects without loading AWS
|
|
131
|
+
credentials:
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
aws s3api head-object --no-sign-request \
|
|
135
|
+
--bucket BUCKET \
|
|
136
|
+
--key "EXACT/DOCUMENTED/OBJECT"
|
|
137
|
+
|
|
138
|
+
aws s3api list-objects-v2 --no-sign-request \
|
|
139
|
+
--bucket BUCKET \
|
|
140
|
+
--prefix "MODEL.DATE/DOMAIN/PRECISE-RUN-PREFIX" \
|
|
141
|
+
--max-items 100
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Paginate narrow listings. S3 returns at most one page at a time, and a
|
|
145
|
+
successful listing still needs defensive response parsing. `403` and `404`
|
|
146
|
+
from `HeadObject` can be intentionally nonspecific; distinguish a missing
|
|
147
|
+
public object from a permission or endpoint error before falling back.
|
|
148
|
+
|
|
149
|
+
## Resolution Record
|
|
150
|
+
|
|
151
|
+
Return enough information for the fetcher and provenance layer to use the exact
|
|
152
|
+
same objects:
|
|
153
|
+
|
|
154
|
+
```json
|
|
155
|
+
{
|
|
156
|
+
"requested_at_utc": "2026-09-18T08:45:00Z",
|
|
157
|
+
"model": "hrrr",
|
|
158
|
+
"product": "prs",
|
|
159
|
+
"initialization": "2026-09-18T06:00:00Z",
|
|
160
|
+
"required_forecast_hours": [0, 6, 18],
|
|
161
|
+
"provider": "aws",
|
|
162
|
+
"objects": [
|
|
163
|
+
{
|
|
164
|
+
"role": "pressure_f018",
|
|
165
|
+
"uri": "s3://bucket/exact-key",
|
|
166
|
+
"size_bytes": 123456,
|
|
167
|
+
"etag": "opaque-object-identity",
|
|
168
|
+
"last_modified": "2026-09-18T07:12:34Z"
|
|
169
|
+
}
|
|
170
|
+
],
|
|
171
|
+
"fallback_cycles_rejected": []
|
|
172
|
+
}
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
Keep ETags as opaque identity values. They are not guaranteed to be full-file
|
|
176
|
+
MD5 checksums.
|
|
177
|
+
|
|
178
|
+
## Reliability and Caching
|
|
179
|
+
|
|
180
|
+
- Cache positive results only briefly enough for the model cadence and
|
|
181
|
+
publication latency. Cache negative probes for a shorter interval.
|
|
182
|
+
- Retry timeouts, `408`, `429`, and transient `5xx` responses with bounded
|
|
183
|
+
exponential backoff and jitter; honor `Retry-After`.
|
|
184
|
+
- Bound lookback and total probes so a provider outage cannot create an
|
|
185
|
+
unbounded bucket walk.
|
|
186
|
+
- Do not treat one mirror's lag as proof that the run is absent everywhere.
|
|
187
|
+
- Do not mix objects from different providers unless their run identity and
|
|
188
|
+
product semantics have been verified equivalent.
|
|
189
|
+
- Record rejected cycles and missing sentinels so fallback is visible.
|
|
190
|
+
|
|
191
|
+
## Verification Checklist
|
|
192
|
+
|
|
193
|
+
- Candidate cycles are legal for the requested model and expressed in UTC.
|
|
194
|
+
- Every required product, member, forecast hour, and sidecar is present.
|
|
195
|
+
- Object sizes are nonzero and stable when a stabilization check is required.
|
|
196
|
+
- The resolved provider and exact object identities are recorded.
|
|
197
|
+
- The selected run satisfies the user's freshness and lookback bounds.
|
|
198
|
+
- No model payload was downloaded merely to discover availability.
|
|
199
|
+
- A failure reports which cycles were checked and which sentinels were absent.
|
|
200
|
+
|
|
201
|
+
## Security & Safety Notes
|
|
202
|
+
|
|
203
|
+
- Probe only public datasets or resources the user is authorized to access.
|
|
204
|
+
- Never embed cloud credentials, signed URLs, session tokens, or private
|
|
205
|
+
endpoint details in a resolution record.
|
|
206
|
+
- Keep TLS verification enabled. Unsigned public S3 access is not the same as
|
|
207
|
+
disabling certificate verification.
|
|
208
|
+
- Bound prefix width, pagination, retries, and request rate before probing.
|
|
209
|
+
- Treat provider object names and metadata as untrusted input when writing logs
|
|
210
|
+
or constructing local paths.
|
|
211
|
+
|
|
212
|
+
## Common Pitfalls
|
|
213
|
+
|
|
214
|
+
- **Newest cycle has `f000`:** Later required forecast hours are still
|
|
215
|
+
publishing. Test the actual terminal sentinels.
|
|
216
|
+
- **A provider fallback changes the forecast:** The fallback changed a product,
|
|
217
|
+
member, or cycle as well as the mirror. Reject it.
|
|
218
|
+
- **A listing found an object but the download changes:** The publisher was
|
|
219
|
+
still updating it. Require stable metadata or conditional reads.
|
|
220
|
+
- **Discovery is slow and expensive:** The prefix is too broad or pagination is
|
|
221
|
+
unbounded. Construct exact keys wherever possible.
|
|
222
|
+
- **Local time selected the wrong day:** Perform cycle arithmetic in UTC and
|
|
223
|
+
convert only for display.
|
|
224
|
+
|
|
225
|
+
## Limitations
|
|
226
|
+
|
|
227
|
+
- Publication schedules, provider retention, key layouts, and mirror status can
|
|
228
|
+
change; verify them against current provider documentation.
|
|
229
|
+
- Metadata stability reduces the chance of a partial publication but does not
|
|
230
|
+
prove that the model contents are scientifically valid.
|
|
231
|
+
- This skill does not fetch GRIB2 fields, decode model data, or assess forecast
|
|
232
|
+
quality.
|
|
233
|
+
|
|
234
|
+
## Additional Resources
|
|
235
|
+
|
|
236
|
+
- [NCEP NOMADS data availability](https://nomads.ncep.noaa.gov/)
|
|
237
|
+
- [NCEP NOMADS status](https://nomads.ncep.noaa.gov/status.php)
|
|
238
|
+
- [Herbie latest model run guidance](https://herbie.readthedocs.io/en/latest/user_guide/tutorial/latest.html)
|
|
239
|
+
- [Amazon S3 `HeadObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html)
|
|
240
|
+
- [Amazon S3 `ListObjectsV2`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)
|
|
241
|
+
- [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
|