opencode-skills-collection 4.0.71 → 4.0.73

package/bundled-skills/marlin-bed-leveling/SKILL.md

@@ -0,0 +1,181 @@
+---
+name: marlin-bed-leveling
+description: "Calibrates Marlin 2.x 3D printer firmware bed leveling: Unified Bed Leveling (UBL), Bilinear ABL, M420 S1 post-homing, Z-probe offsets, G26 mesh prints, and EEPROM slots. Trigger phrases: marlin bed leveling, ubl calibration, m420 s1, z probe offset, g29 bed level, first layer adhesion."
+category: development
+risk: safe
+source: community
+source_repo: wwewtech/marlin-bed-leveling
+source_type: community
+date_added: "2026-09-22"
+author: wwewtech
+tags: [3d-printing, marlin-firmware, gcode, hardware-calibration, fdm, additive-manufacturing]
+tools: [claude, cursor, gemini, windsurf]
+license: "MIT"
+---
+
+# Marlin Bed Leveling: Deterministic Calibration & First-Layer Mesh Engineering
+
+Master Marlin 2.x bed leveling architectures (Unified Bed Leveling UBL, Bilinear ABL, Manual Mesh MBL) to guarantee flawless first-layer adhesion through rigorous G-code phase workflows, probe offset math, and persistent EEPROM compensation.
+
+## When to Use This Skill
+
+Activate this skill when:
+- Calibrating, tuning, or troubleshooting automatic bed leveling on Marlin 2.x 3D printers (Ender 3, CR-10, Prusa clones, custom CoreXY / Cartesian machines).
+- The user asks: "How do I set up UBL in Marlin?", "My printer homes with G28 and ignores the mesh", "Calculate BLTouch Z-probe offset", "Generate G26 validation print G-code", or "Edit warped mesh points with M421".
+- Diagnosing first-layer defects: nozzle dragging across PEI sheets, corner detachment, uneven squish, or compensation disabling after homing.
+- Sizing slicer start G-code sequences (Cura, PrusaSlicer, OrcaSlicer, Bambu Studio) with thermal soaking and mesh restoration.
+
+Do NOT use this skill when:
+- Calibrating Klipper or RepRapFirmware printers (different firmware syntax and mesh compensation systems).
+- Fixing gross mechanical skew: loose V-slot rollers, bent Z leadscrews, or wobbly gantry arms (hardware faults cannot be solved by software leveling).
+- Delta printer kinematics setup requiring radius and tower endstop adjustments before mesh probing.
+
+## Core Mental Models & Non-Negotiable Rules
+
+1. **The Post-Homing Enable Mandate (`G28` Disables Leveling)**:
+   - In Marlin firmware, executing `G28` (Home All Axes) automatically **DISABLES bed leveling compensation by default** (`RESTORE_LEVELING_AFTER_G28` is often disabled in vendor firmware).
+   - In every print start G-code, the leveling restoration command MUST be issued **strictly AFTER `G28`**:
+     ```gcode
+     G28      ; Home all axes (silently turns leveling OFF)
+     M420 S1  ; Re-enable bed leveling compensation from EEPROM mesh
+     ```
+   - Failing to include `M420 S1` (or `G29 A` for UBL) after `G28` causes the printer to execute the entire print with zero leveling compensation.
+
+2. **The 5-Phase UBL Calibration Loop**:
+   Unified Bed Leveling (UBL) is a state machine requiring five sequential phases:
+   - **Phase 1 (Probe Reachable Grid)**: `G29 P1` probes all coordinates physically reachable by the probe offset.
+   - **Phase 2 (Manual Edge Probing)**: `G29 P2` uses a paper feeler gauge for points the BLTouch cannot reach due to gantry geometry.
+   - **Phase 3 (Mathematical Extrapolation)**: `G29 P3 T0.0` fills unprobed edge coordinates by extrapolating slopes from adjacent measured points.
+   - **Phase 4 (Validation & Micro-Tuning)**: `G26 B60 H210 F1.75` prints a single-layer validation test pattern; inspect and edit coordinates with `M421 I... J... Q...`.
+   - **Phase 5 (Storage & Activation)**: `G29 S1` saves to EEPROM mesh slot 1; `G29 A` activates UBL; `M500` commits to permanent EEPROM storage.
+
+3. **Thermal Equilibrium Pre-Flight Invariant**:
+   - Aluminum build plates, magnetic stickers, and spring-steel PEI sheets expand and warp non-linearly under thermal expansion.
+   - Probing a bed at room temperature (20°C) and then printing at 60°C or 100°C bakes a $0.08\text{--}0.20\text{mm}$ error into every mesh point.
+   - **Mandatory Pre-Flight**: Beds MUST be heated to target operating temperature and soaked for $\ge 5\text{ minutes}$ prior to running `G29` probing.
+
+4. **Z-Probe Offset Sign Convention (`M851 Z...`)**:
+   - For downward-deploying probes (BLTouch, CRTouch, inductive sensors) where the probe trigger point triggers *below* the nozzle tip:
+     $$\mathbf{Z\text{-}Offset\ is\ strictly\ NEGATIVE}\ (e.g.,\ \text{M851 Z-1.85})$$
+   - A *positive* Z-offset on a downward-deploying probe instructs the firmware that the nozzle is lower than the probe trigger, driving the nozzle directly into the bed on the first move.
+   - Adjusting Z-offset with paper gauge:
+     - More resistance / tighter squish $\implies$ more negative (e.g., `-1.80` $\to$ `-1.85`).
+     - Less resistance / looser gap $\implies$ less negative (e.g., `-1.80` $\to$ `-1.75`).
+
+5. **Mesh Fade Height Geometry (`M420 Z...`)**:
+   - Leveling compensation must gradually fade out as Z-height increases so top surface layers print geometrically flat rather than conforming to the bed warp.
+   - Set fade height to `M420 Z10.0` (fades compensation smoothly over the first 10mm of print height). Setting `Z0` disables fade (perpetuating bed ripples to the top of the model); setting `Z > 25` is unnecessary ballast.
+
+## Named Sins & Anti-Patterns (Что категорически ЗАПРЕЩЕНО)
+
+| Anti-Pattern | Manifestation in Code/Workflow | Mandatory Production Counter-Rule |
+| :--- | :--- | :--- |
+| **The Forgotten `M420 S1`** | `G28` followed immediately by print moves. | Always place `M420 S1` (or `G29 A`) immediately after `G28`. |
+| **Cold Bed Mesh Probing** | Running `G29 P1` with bed heater at 20°C. | Heat bed to printing temperature and soak for $\ge 5$ minutes first. |
+| **Positive Z-Probe Offset** | Setting `M851 Z+1.5` on a BLTouch probe. | Downward-deploying probes MUST have a negative Z-offset (`M851 Z-...`). |
+| **Unsaved Mesh Discard** | Probing a 7x7 mesh without running `M500`. | Always commit mesh to EEPROM via `G29 S1` followed by `M500`. |
+| **Software-Fixing Physical Tilt** | Compensating for a 2.5mm physical gantry tilt with UBL. | Physically tram bed corners (`G35` or thumb screws) before probing. |
+| **Blind Unchecked Extrapolation** | Running `G29 P3` without verifying boundary values. | Inspect extrapolated border points with `M420 V` or `G29 T`. |
+| **Cold Extruder G26 Validation** | Running G26 test without nozzle heating. | Heat nozzle to printing temperature (`G26 B60 H210 F1.75`). |
+| **Loose Probe Mount Blindness** | Probing while BLTouch bracket has physical wobble. | Check physical probe mount rigidity; probe repeatability requires $s \le 0.005\text{mm}$ (`M48`). |
+| **Excessive Fade Height** | Setting `M420 Z0` or `M420 Z50`. | Use `M420 Z10.0` to smoothly transition from warped bed to true flat geometry. |
+| **Blind Point Over-Editing** | Manually offsetting a single mesh point by 0.5mm via `M421`. | Clean the build plate or check for debris under PEI sheet before altering matrix. |
+
+## Concrete Archetypes / Presets
+
+### Archetype 1: End-to-End Unified Bed Leveling (UBL) Calibration Macro
+```gcode
+; ==============================================================================
+; MARLIN 2.x UBL FULL CALIBRATION SEQUENCE
+; ==============================================================================
+M140 S60         ; Start heating bed to 60°C
+M190 S60         ; Wait for bed to reach 60°C
+G4 S300          ; Thermal soak for 300 seconds (5 minutes)
+
+G28              ; Home all axes
+M117 Probing Reachable Points...
+G29 P1           ; Phase 1: Automated probe of reachable grid points
+
+M117 Extrapolating Mesh Edges...
+G29 P3 T0.0      ; Phase 3: Mathematically extrapolate unprobed perimeter points
+G29 P3 T0.0      ; Second pass to ensure corner fill
+
+G29 T            ; Print ASCII topology matrix to terminal for verification
+G29 S1           ; Save calibrated mesh to EEPROM slot 1
+G29 A            ; Activate UBL leveling system
+M500             ; Save EEPROM permanently to board flash
+M117 UBL Mesh 1 Saved and Activated!
+```
+
+### Archetype 2: Production Slicer Start G-Code (Cura / PrusaSlicer / Orca)
+```gcode
+; ==============================================================================
+; PRODUCTION SLICER START G-CODE (MARLIN UBL / BILINEAR)
+; ==============================================================================
+M140 S[first_layer_bed_temperature]    ; Start heating bed
+M104 S150                              ; Pre-heat nozzle to 150°C (prevent oozing)
+M190 S[first_layer_bed_temperature]    ; Wait for bed to stabilize
+
+G28                                    ; Home all axes (turns leveling OFF)
+M420 S1 Z10.0                          ; Restore mesh from EEPROM & set 10mm fade
+; (OR for UBL: G29 A followed by G29 L1)
+
+G1 Z50 F3000                           ; Move nozzle up to safe clearance
+M109 S[first_layer_temperature]        ; Heat nozzle to final printing temp
+
+; Priming Purge Line on bed edge
+G92 E0                                 ; Reset extruder
+G1 X0.1 Y20 Z0.28 F5000.0              ; Move to start position
+G1 X0.1 Y200.0 Z0.28 F1500.0 E15       ; Draw first purge line
+G1 X0.4 Y200.0 Z0.28 F5000.0           ; Move to side
+G1 X0.4 Y20 Z0.28 F1500.0 E30          ; Draw second purge line
+G92 E0                                 ; Reset extruder
+G1 Z2.0 F3000                          ; Z-hop before moving to print
+```
+
+### Archetype 3: Probe Repeatability & Offset Verification (`M48` & `M851`)
+```gcode
+; ==============================================================================
+; PROBE REPEATABILITY & Z-OFFSET CALIBRATION
+; ==============================================================================
+G28                     ; Home axes
+G1 X117.5 Y117.5 Z10 F5000 ; Move to center of 235x235 bed
+M48 P10 X117.5 Y117.5 V2 E L2 ; Run 10-point repeatability test
+; Standard deviation must be <= 0.005 mm. If > 0.010 mm, tighten probe mount!
+
+; Manual Z-Offset adjustment with paper feeler gauge:
+; 1. Move to Z=0: G1 Z0 F1000
+; 2. Baby-step Z until paper drags with slight friction
+; 3. If nozzle moved down 0.15mm: M851 Z-1.85 (subtract 0.15 from current -1.70)
+; 4. Commit: M500
+```
+
+## The Pre-Emit Quality Gate Checklist
+
+Before emitting any Marlin leveling advice, macro, or slicer start code:
+- [ ] **Post-Homing Verification**: `M420 S1` (or `G29 A`) placed strictly after `G28` in the execution flow.
+- [ ] **Thermal Soak Pre-Flight**: Bed heating and 5-minute soak mandated prior to `G29` probing.
+- [ ] **Negative Offset Math**: Downward-deploying probes have verified negative `M851 Z-...` values.
+- [ ] **EEPROM Commit**: `G29 S1` followed immediately by `M500` to preserve the mesh.
+- [ ] **Fade Height Limit**: `M420 Z10.0` configured for smooth transition to flat geometry.
+- [ ] **Physical Tramming Precedence**: Confirmed that manual corner leveling is performed before software compensation.
+- [ ] **Probe Repeatability**: Confirmed that `M48` probe standard deviation is $\le 0.005\text{mm}$.
+
+## Self-Correction & Verification Loop
+
+### Bad (Vibe-Coded AI Slop Output):
+> "Put `G29` before `G28` in your start G-code. If your BLTouch is scraping, set `M851 Z1.5`. You don't need to heat the bed during leveling because Marlin compensates for it automatically."
+*Problems: Fatal G28 homing wipes out the preceding G29 mesh, positive Z-offset crashes nozzle into bed, dangerous cold probing advice.*
+
+### Good (Production Marlin Bed Leveling Output):
+> 1. Tram the four bed corners manually using `G35` or thumb screws to get mechanical variance under $\pm 0.1\text{mm}$.
+> 2. Heat bed to 60°C and nozzle to 150°C; let soak for 5 minutes so thermal expansion stabilizes.
+> 3. Verify probe repeatability using `M48 P10` (standard deviation must be $\le 0.005\text{mm}$).
+> 4. Probe grid with `G29 P1`, extrapolate unprobed corners with `G29 P3 T0.0`, save to slot 1 with `G29 S1`, and commit to flash with `M500`.
+> 5. In your slicer start G-code, place `M420 S1 Z10.0` immediately after `G28` to restore the active mesh with a 10mm fade height.
+
+## Limitations
+
+- Marlin commands and mesh behavior vary with firmware version, configuration, probe, bed geometry, and slicer. Check the exact firmware build and printer documentation before applying a macro or offset.
+- G-code can move a hot toolhead and damage equipment or cause burns. Supervise calibration, keep hands clear, use the printer's emergency stop when motion is unsafe, and verify travel limits before running generated sequences.
+- Example coordinates, temperatures, tolerances, and fade values are illustrative; measure the actual printer and follow the manufacturer’s limits.

package/bundled-skills/noaa-radar-satellite-fetching/SKILL.md

@@ -0,0 +1,253 @@
+---
+name: noaa-radar-satellite-fetching
+description: "Retrieve NOAA NEXRAD and GOES products from public cloud archives using verified site, product, channel, sector, and scan-time selection."
+category: data
+risk: safe
+source: self
+source_type: self
+date_added: "2026-09-24"
+author: ShianMike
+tags: [weather, noaa, nexrad, goes, radar, satellite, aws-s3]
+tools: [claude, cursor, gemini, codex]
+---
+
+# NOAA Radar and Satellite Fetching
+
+## Overview
+
+Retrieve NOAA NEXRAD radar and GOES satellite objects from public cloud
+archives while preserving the product, platform, spatial sector, and scan-time
+identity. Discover through narrow prefixes, choose by observation time rather
+than upload time, and validate the downloaded scientific file before use.
+
+This skill covers raw or provider-produced radar and satellite products. It
+does not retrieve numerical model output or ordinary station observations.
+
+## When to Use This Skill
+
+- A task needs NEXRAD Level II archive volumes or real-time chunks.
+- A task needs selected NEXRAD Level III products.
+- GOES ABI, GLM, or another GOES-R product is needed for a time and region.
+- A download must select the nearest completed scan within a tolerance.
+- A public S3 key, satellite assignment, radar site, channel, or sector is
+  uncertain or has changed.
+
+Do not activate this skill merely to display a third-party map tile or to answer
+a general weather question.
+
+## Define the Product Contract
+
+For NEXRAD, specify:
+
+- Level II archive, Level II real-time chunks, or a named Level III product;
+- four-character radar site, such as `KTLX`;
+- desired scan time in UTC and nearest/previous/next selection policy;
+- maximum time tolerance;
+- full volume or exact product code;
+- raw file retention and decoder.
+
+For GOES, specify:
+
+- physical satellite number or operational role, such as East or West;
+- instrument and product level/short name;
+- Full Disk, CONUS, Mesoscale 1, or Mesoscale 2 sector where applicable;
+- ABI channel or derived product;
+- desired scan time in UTC and tolerance;
+- calibration form needed, such as radiance or brightness temperature.
+
+Resolve an operational role to a physical satellite using current NOAA
+documentation at request time. East and West assignments change over a
+satellite program's lifetime.
+
+## Current Public AWS Routes
+
+Consult the AWS Registry of Open Data before hard-coding a bucket. As of this
+skill's publication:
+
+- NEXRAD Level II archive: `s3://unidata-nexrad-level2/`
+- NEXRAD Level II real-time chunks: `s3://unidata-nexrad-level2-chunks/`
+- selected NEXRAD Level III real-time data: `s3://unidata-nexrad-level3/`
+- GOES imagery and metadata: satellite-specific buckets such as
+  `s3://noaa-goes18/` and `s3://noaa-goes19/`
+
+The former `noaa-nexrad-level2` bucket was deprecated and scheduled to become
+unavailable on September 1, 2025. Do not copy that retired route from old
+tutorials.
+
+## Discover with Narrow Prefixes
+
+NEXRAD Level II archive objects are organized by UTC date and site. GOES
+objects are organized by product, year, day of year, and hour. List only the
+smallest prefix that can contain the requested scan.
+
+```bash
+aws s3 ls --no-sign-request \
+  s3://unidata-nexrad-level2/YYYY/MM/DD/KXXX/
+
+aws s3 ls --no-sign-request \
+  s3://noaa-goes19/ABI-L2-CMIPC/YYYY/DDD/HH/
+```
+
+Replace every placeholder with a validated value. Include the adjacent hour or
+UTC day only when the time tolerance crosses that boundary. Paginate listings
+and bound the total keys inspected.
+
+Use real-time chunk buckets only when the consumer is designed to assemble and
+validate chunks. Prefer completed archive volumes for ordinary historical or
+post-event processing.
+
+## Select by Scan Time
+
+Object `Last-Modified` is publication metadata, not the measurement time.
+
+- For NEXRAD, parse the site and scan timestamp from the documented object name,
+  then confirm time and site from the decoded volume header.
+- For GOES-R, filenames encode platform plus scan start (`s`), end (`e`), and
+  creation (`c`) timestamps. Select by scan start or interval overlap, not file
+  creation time.
+- Default to the closest scan at or before the requested instant for a
+  retrospective view. Allow a future scan only when the caller explicitly asks
+  for nearest-in-either-direction behavior.
+- Reject every candidate outside the stated tolerance.
+- When sectors overlap, require the requested sector instead of choosing solely
+  by time.
+
+This standard-library helper extracts GOES-R scan timestamps without opening
+the NetCDF payload:
+
+```python
+import re
+from datetime import datetime, timezone
+from pathlib import PurePosixPath
+
+
+GOES_TIMES = re.compile(
+    r"_s(?P<start>\d{14})_e(?P<end>\d{14})_c(?P<created>\d{14})"
+)
+
+
+def parse_goes_times(object_key):
+    match = GOES_TIMES.search(PurePosixPath(object_key).name)
+    if not match:
+        raise ValueError(f"unrecognized GOES-R filename: {object_key}")
+
+    def parse(value):
+        return datetime.strptime(value, "%Y%j%H%M%S%f").replace(
+            tzinfo=timezone.utc
+        )
+
+    return {name: parse(value) for name, value in match.groupdict().items()}
+```
+
+Treat filename parsing as discovery. After download, verify the corresponding
+time coverage and platform attributes inside the file.
+
+## Download and Validate
+
+1. Record bucket, exact key, size, ETag, and `Last-Modified` before transfer.
+2. Download to a request-owned temporary path. For public buckets, use unsigned
+   S3 access rather than supplying credentials.
+3. If the object might still be publishing, require stable metadata or pin a
+   conditional request to the observed object identity.
+4. Verify the received byte count and a provider checksum when one is supplied.
+   Otherwise compute and record a local SHA-256 digest.
+5. Open the object with an existing format-aware decoder.
+6. Verify site/platform, product, scan interval, and expected dimensions or
+   radar sweeps before renaming it into the cache.
+
+Reuse an installed NEXRAD or NetCDF reader. Do not build a binary radar or
+satellite decoder merely to fetch a file.
+
+## Radar-Specific Checks
+
+- Confirm that the radar site is the requested site and was operational at the
+  scan time.
+- Treat Level II volumes, real-time chunks, and Level III products as different
+  contracts; they are not interchangeable encodings.
+- Verify volume start time, end time when available, sweep count, moments, and
+  elevation angles after decoding.
+- Keep range folding, missing gates, quality masks, and velocity ambiguity
+  explicit.
+- A radar's nominal coverage radius does not guarantee useful low-level data at
+  a location. Consider distance, beam height, terrain blockage, and outages.
+
+## Satellite-Specific Checks
+
+- Confirm the physical satellite, instrument, product short name, sector, scan
+  mode, channel, and time coverage from NetCDF attributes.
+- Do not treat ABI fixed-grid `x` and `y` coordinates as latitude and longitude.
+  Use the file's geostationary projection metadata.
+- Apply scale/offset, fill values, data-quality flags, and product-specific
+  calibration according to NOAA documentation.
+- A Mesoscale file is not identified only by its small dimensions; distinguish
+  Mesoscale 1 and 2 explicitly.
+- Do not blend files from different scan modes, sectors, satellites, or product
+  levels without recording the transformation.
+
+## Reliability, Cache, and Cleanup
+
+- Cache by bucket, key, and object identity, not a friendly timestamp alone.
+- Retry timeouts, `408`, `429`, and transient `5xx` responses with bounded
+  backoff and jitter.
+- Bound time tolerance, adjacent-hour listings, pagination, concurrency, and
+  disk usage.
+- Keep partial downloads separate from valid cache entries.
+- Use event notifications for continuous real-time ingestion when appropriate;
+  do not repeatedly scan broad prefixes.
+- For a one-shot image or export, delete request-owned raw objects in `finally`
+  only after the final artifact and provenance manifest are durable.
+- Never delete a shared user cache as request cleanup.
+
+## Verification Checklist
+
+- The source bucket is current and documented by NOAA or its data distributor.
+- Radar site or physical satellite and operational role are explicit.
+- Product level, code/short name, sector, channel, and scan mode match the
+  request.
+- The selected scan satisfies the direction policy and maximum time tolerance.
+- Object size and identity match the downloaded bytes.
+- The format-aware decoder opens the file and confirms internal metadata.
+- Geolocation uses the product's real coordinate reference system.
+- Temporary data cleanup preserves the requested artifact and manifest.
+
+## Security & Safety Notes
+
+- Fetch only public data or resources the user is authorized to access.
+- Never embed AWS credentials, signed URLs, private endpoints, or notification
+  subscription tokens in code or logs.
+- Keep TLS verification enabled; `--no-sign-request` does not disable TLS.
+- Validate site, product, channel, sector, and time inputs against allowlists
+  before constructing object prefixes or local paths.
+- Bound response size and available disk space before downloading high-rate
+  radar or satellite streams.
+- Preserve NOAA attribution and do not imply NOAA endorsement.
+
+## Common Pitfalls
+
+- **Old NEXRAD examples return no data:** They reference the retired Level II
+  bucket. Resolve the current bucket from the registry.
+- **The right minute shows the wrong image:** Creation time was used instead of
+  scan start/end time.
+- **GOES East points to the wrong bucket:** An operational role was hard-coded
+  to a satellite number. Resolve the assignment for the requested date.
+- **The map is displaced:** Fixed-grid scan angles were treated as geographic
+  coordinates. Apply the geostationary projection.
+- **A real-time radar volume will not decode:** Individual chunks were treated
+  as a completed archive volume. Assemble and validate them or use the archive.
+
+## Limitations
+
+- Buckets, satellite assignments, product availability, scan modes, and naming
+  conventions can change; verify current official documentation.
+- This skill retrieves source products but does not guarantee meteorological
+  suitability, perform mosaicking, or implement scientific calibration.
+- Historical gaps and instrument or site outages cannot be repaired by provider
+  fallback.
+
+## Additional Resources
+
+- [NEXRAD on the AWS Registry of Open Data](https://registry.opendata.aws/noaa-nexrad/)
+- [NOAA GOES on the AWS Registry of Open Data](https://registry.opendata.aws/noaa-goes/)
+- [NOAA GOES-R documents](https://www.ospo.noaa.gov/resources/documents/goes-r.html)
+- [NOAA STAR guide to satellite filenames](https://www.star.nesdis.noaa.gov/atmospheric-composition-training/satellite_data_decoding_data_file_names.php)
+- [NOAA Radar Operations Center interface documents](https://www.roc.noaa.gov/wsr88d/BuildInfo/Files.aspx)

package/bundled-skills/oneroster-csv-validator/SKILL.md

@@ -0,0 +1,205 @@
+---
+name: oneroster-csv-validator
+description: "Validates 1EdTech / IMS Global OneRoster v1.1 and v1.2 CSV roster sets: manifest integrity, bulk vs delta strictness, foreign key references, and encoding sanitization. Trigger phrases: oneroster csv validator, validate oneroster zip, clever roster error, oneroster sourcedid."
+category: testing
+risk: safe
+source: community
+source_repo: wwewtech/oneroster-csv-validator
+source_type: community
+date_added: "2026-09-22"
+author: wwewtech
+tags: [edtech, oneroster, csv-validation, ims-global, 1edtech, data-engineering]
+tools: [claude, cursor, gemini, windsurf]
+license: "MIT"
+---
+
+# OneRoster CSV Validator: Strict 1EdTech Data Integrity & Pre-Flight Linter
+
+Perform rigorous pre-flight validation on OneRoster v1.1 and v1.2 CSV roster bundles prior to SIS/Clever/ClassLink ingestion: audit foreign key referential integrity, eliminate orphan enrollments, enforce bulk vs delta partitioning, and sanitize encoding traps.
+
+## When to Use This Skill
+
+Activate this skill when:
+- Validating or debugging student information system (SIS) roster export packages (`.zip` archives or CSV directories) for OneRoster v1.1 or v1.2 compliance.
+- The user asks: "Why is Clever rejecting our OneRoster zip file?", "Check for orphaned userSourcedIds in enrollments.csv", "Validate bulk vs delta mode in OneRoster CSVs", or "Sanitize UTF-8 BOM characters from school rosters".
+- Auditing relational integrity across `manifest.csv`, `orgs.csv`, `users.csv`, `courses.csv`, `classes.csv`, `enrollments.csv`, and `academicSessions.csv`.
+- Preparing automated pre-flight CI/CD pipelines to catch corrupted student or teacher records before nightly SIS sync runs.
+
+Do NOT use this skill when:
+- Integrating OneRoster REST / OAuth 2.0 API endpoints (this skill focuses strictly on CSV table bindings).
+- Generating synthetic or fabricated student PII without authorization.
+- Working on generic non-standard CSV spreadsheets that do not follow 1EdTech specifications.
+
+## Core Mental Models & Non-Negotiable Rules
+
+1. **The Manifest Root Authority Law**:
+   - `manifest.csv` MUST reside strictly at the root level of the ZIP archive (never nested inside a subfolder).
+   - It MUST declare the exact specification version: `oneroster.version,1.1` or `oneroster.version,1.2`.
+   - It MUST list every single file present in the exchange. If a CSV file is present in the archive but omitted from `manifest.csv`, or listed in `manifest.csv` but missing from the zip, the package is rejected.
+
+2. **The Strict Bulk vs Delta Partition**:
+   - One exchange bundle = strictly ONE operational mode.
+   - **Bulk Exchange Mode**: Represents a complete master snapshot. In OneRoster 1.1, rows MUST NOT contain `dateLastModified` or `status` columns. In 1.2, all status fields MUST be `active`.
+   - **Delta Exchange Mode**: Represents incremental changes. Every row MUST contain a valid ISO 8601 UTC timestamp (`YYYY-MM-DDTHH:MM:SS.sssZ`) in `dateLastModified` and a valid status (`active` or `tobedeleted`).
+   - Mixing bulk and delta formats within the same archive is a fatal validation failure.
+
+3. **The Foreign Key Referential Integrity Hierarchy**:
+   - Relational dependency chain:
+     $$\mathbf{orgs.csv} \longleftarrow \mathbf{courses.csv} \longleftarrow \mathbf{classes.csv} \longleftarrow \mathbf{enrollments.csv} \longrightarrow \mathbf{users.csv}$$
+   - Every `schoolSourcedId` in `users.csv` and `classes.csv` MUST resolve to an existing `sourcedId` in `orgs.csv`.
+   - Every `courseSourcedId` in `classes.csv` MUST resolve to `courses.csv`.
+   - Every `userSourcedId` and `classSourcedId` in `enrollments.csv` MUST resolve to active primary keys in `users.csv` and `classes.csv`.
+   - **Zero Orphan Tolerance**: A single orphan link (e.g. enrolling a deleted student ID) invalidates the batch.
+
+4. **Strict Role Enum Constraint Axioms**:
+   - In `users.csv`: `role` MUST strictly match the OneRoster enum:
+     - v1.1: `administrator`, `proctor`, `student`, `teacher`.
+     - v1.2: adds `aide`, `guardian`, `parent`, `staff`.
+   - In `enrollments.csv`: `role` is strictly constrained to `administrator`, `proctor`, `student`, `teacher`. Custom vendor roles (`substitute`, `dean`) must be mapped to valid specification enums.
+   - Primary `sourcedId` values MUST be case-sensitively unique strings (RFC 4122 UUID format strongly recommended).
+
+5. **RFC 4180 CSV & UTF-8 BOM Sanitization**:
+   - Files MUST be encoded in **UTF-8 without Byte Order Mark (BOM)**.
+   - Microsoft Excel frequently prepends `\xEF\xBB\xBF` to CSV exports, corrupting the first column name (`sourcedId` $\neq$ `sourcedId`).
+   - Fields containing commas, double-quotes, or newlines MUST be enclosed in double quotes (`"`). Literal quotes inside fields must be escaped as `""`.
+
+## Named Sins & Anti-Patterns (Что категорически ЗАПРЕЩЕНО)
+
+| Anti-Pattern | Manifestation in CSV Packages | Mandatory Production Counter-Rule |
+| :--- | :--- | :--- |
+| **Nested Archive Packaging** | Zipping a parent folder (`district_roster/manifest.csv`). | Manifest and CSVs MUST reside at the archive root (`/manifest.csv`). |
+| **The UTF-8 BOM Trap** | Excel export injecting `\xEF\xBB\xBF` into `sourcedId`. | Strip BOM headers during pre-flight sanitization pass. |
+| **Orphan Enrollment Links** | `enrollments.csv` pointing to non-existent user IDs. | Enforce foreign key validation across all entity files before import. |
+| **Bulk / Delta Column Bleed** | Including `dateLastModified` or `tobedeleted` in bulk. | Enforce schema separation: bulk files must not contain delta headers. |
+| **Circular Org Hierarchies** | School A listed as parent of School B, and vice-versa. | Perform directed acyclic graph (DAG) cycle detection on `orgs.csv`. |
+| **Illegal Role Strings** | Using non-standard strings like `counselor` or `sub`. | Map all roles to standard enums (`administrator`, `teacher`, `student`). |
+| **Malformed ISO 8601 Timestamps** | Writing `"09/22/2026 14:00"` instead of ISO 8601. | Enforce strict RFC 3339 UTC format: `YYYY-MM-DDTHH:MM:SS.sssZ`. |
+| **Unescaped CSV Commas** | Unquoted commas in user names (`Doe, John`) shifting columns. | Wrap text fields containing commas in standard RFC 4180 quotes. |
+| **Missing Mandatory Core Files** | Exporting a bulk zip without `academicSessions.csv`. | Confirm all 7 mandatory core files exist in zip and manifest. |
+| **Hallucinated SourcedIDs** | Inventing arbitrary IDs during reconciliation. | Preserve authoritative SIS primary keys without synthetic generation. |
+
+## Concrete Archetypes / Presets
+
+### Archetype 1: Pre-Flight Python Referential Integrity Linter
+```python
+import csv
+import io
+import re
+import zipfile
+from typing import Dict, List, Set
+
+CORE_FILES = [
+    "manifest.csv", "orgs.csv", "users.csv", "courses.csv",
+    "classes.csv", "enrollments.csv", "academicSessions.csv"
+]
+VALID_ROLES_1_1 = {"administrator", "proctor", "student", "teacher"}
+
+def validate_oneroster_archive(zip_path: str) -> dict:
+    errors = []
+    warnings = []
+
+    with zipfile.ZipFile(zip_path, 'r') as z:
+        names = z.namelist()
+
+        # 1. Manifest Root Check
+        if "manifest.csv" not in names:
+            return {"status": "FATAL", "errors": ["manifest.csv missing from zip root."]}
+
+        # 2. Extract and Parse Core Sets
+        data: Dict[str, List[dict]] = {}
+        keys: Dict[str, Set[str]] = {}
+
+        for filename in CORE_FILES:
+            if filename in names:
+                raw_bytes = z.read(filename)
+                # Strip UTF-8 BOM if present
+                if raw_bytes.startswith(b'\xef\xbb\xbf'):
+                    warnings.append(f"{filename} contains UTF-8 BOM; stripped automatically.")
+                    raw_bytes = raw_bytes[3:]
+
+                reader = csv.DictReader(io.StringIO(raw_bytes.decode('utf-8', errors='replace')))
+                rows = list(reader)
+                data[filename] = rows
+                if rows and "sourcedId" in rows[0]:
+                    keys[filename] = {r["sourcedId"] for r in rows if "sourcedId" in r}
+            else:
+                errors.append(f"Mandatory core file missing: {filename}")
+
+        # 3. Foreign Key Checks
+        if "orgs.csv" in keys and "users.csv" in data:
+            for r in data["users.csv"]:
+                org_ref = r.get("orgSourcedIds") or r.get("schoolSourcedId")
+                if org_ref and org_ref not in keys["orgs.csv"]:
+                    errors.append(f"users.csv: user {r.get('sourcedId')} references non-existent org {org_ref}")
+
+        if "users.csv" in keys and "classes.csv" in keys and "enrollments.csv" in data:
+            for r in data["enrollments.csv"]:
+                uid = r.get("userSourcedId")
+                cid = r.get("classSourcedId")
+                if uid not in keys["users.csv"]:
+                    errors.append(f"enrollments.csv: orphan enrollment references missing user {uid}")
+                if cid not in keys["classes.csv"]:
+                    errors.append(f"enrollments.csv: orphan enrollment references missing class {cid}")
+
+                role = r.get("role")
+                if role not in VALID_ROLES_1_1:
+                    errors.append(f"enrollments.csv: illegal role '{role}' on user {uid}")
+
+    return {
+        "status": "FAILED" if errors else "PASSED",
+        "errors": errors,
+        "warnings": warnings,
+        "total_records_checked": sum(len(v) for v in data.values())
+    }
+```
+
+### Archetype 2: Minimal Valid OneRoster v1.1 Manifest
+```csv
+propertyName,value
+oneroster.version,1.1
+file.orgs,bulk
+file.users,bulk
+file.courses,bulk
+file.classes,bulk
+file.enrollments,bulk
+file.academicSessions,bulk
+```
+
+### Archetype 3: Delta Synchronization Row Example
+```csv
+sourcedId,status,dateLastModified,userSourcedId,classSourcedId,role,primary
+enr_99201,active,2026-09-22T08:30:00.000Z,usr_0421,cls_bio101,student,true
+enr_99202,tobedeleted,2026-09-22T08:30:00.000Z,usr_0884,cls_bio101,student,false
+```
+
+## The Pre-Emit Quality Gate Checklist
+
+Before emitting validation results, CSV transforms, or roster scripts:
+- [ ] **Root Location**: Confirmed `manifest.csv` is located strictly at `/` without folder nesting.
+- [ ] **BOM Scrubbing**: Verified all CSV headers are clean UTF-8 without `\xEF\xBB\xBF`.
+- [ ] **Mode Consistency**: Verified no delta columns exist in bulk packages, and timestamps exist in deltas.
+- [ ] **Referential Integrity**: 100% check of `userSourcedId`, `classSourcedId`, `orgSourcedIds`, and `courseSourcedId`.
+- [ ] **Role Enum Validation**: Confirmed all role values match 1EdTech specifications.
+- [ ] **Cycle-Free Orgs**: DAG validation run on `orgs.csv` to ensure no circular parent references.
+- [ ] **RFC 4180 Escaping**: Commas, quotes, and multi-line values enclosed in double quotes.
+
+## Self-Correction & Verification Loop
+
+### Bad (Vibe-Coded AI Slop Output):
+> "Zip your CSV folder up and upload it. If Clever says there is a missing manifest, just add a blank manifest.csv inside the folder. If some student IDs are missing in enrollments, just delete those rows or invent temporary user IDs."
+*Problems: Nested zip breaks root discovery, blank manifest violates schema, inventing fake IDs corrupts student records, ignores UTF-8 BOM.*
+
+### Good (Production OneRoster CSV Validator Output):
+> 1. Extract zip to verify `/manifest.csv` exists at the root path; confirm `oneroster.version` is declared as `1.1`.
+> 2. Scan all CSV headers: detect and strip UTF-8 BOM (`\xEF\xBB\xBF`) from `users.csv` to prevent header corruption.
+> 3. Perform foreign key referential integrity pass:
+>    - Flag 4 orphan rows in `enrollments.csv` referencing deleted student ID `usr_9914`.
+>    - Flag invalid role `"substitute_teacher"` on row 14 of `enrollments.csv` (must map to `"teacher"`).
+> 4. Verify bulk purity: confirm no `dateLastModified` or `tobedeleted` status flags are present in bulk mode.
+> 5. Emit clean, line-numbered pre-flight audit report before committing to SIS ingestion.
+
+## Limitations
+
+- OneRoster version details and required files can differ by exchange profile and vendor implementation. Validate against the exact 1EdTech specification and recipient's current import contract before rejecting or transforming a roster.
+- Student and staff roster files contain sensitive education records. Process only with authorization, minimize retained data, redact identifiers in diagnostics, and do not upload source CSVs to external services.
+- The included snippets are illustrative, not a complete certified validator; for example, large files, namespaces, delta semantics, and vendor extensions require additional handling and tests.