ecological-agent-skills 3.1.0

package/skills/ecological-data-foundation/SKILL.md

@@ -0,0 +1,129 @@
+---
+name: ecological-data-foundation
+description: "Cleans, validates, and standardizes ecological occurrence records and downloads biodiversity data from global repositories. Use this skill when the user needs data cleaning, coordinate validation, duplicate removal, outlier detection, taxonomic harmonization, or downloads from GBIF, iNaturalist, eBird, OBIS, or IUCN Red List. Also triggers for Darwin Core formatting, data quality reports, flagged records, and biodiversity data preparation."
+skill_version: 1.0.0
+---
+
+# Skill: ecological-data-foundation
+
+**Domain:** Data ingestion · QA · Schema · Metadata  
+**Phase:** 1 — Foundation  
+**Used by:** All workflows
+
+---
+
+## Purpose
+
+This skill guides the agent through the first mandatory step of any quantitative ecology project: getting raw data into a clean, well-documented, analysis-ready state. It covers ingestion of heterogeneous sources, structural validation, duplicate and outlier detection, schema standardisation, and metadata generation.
+
+---
+
+## When to Invoke
+
+- A new dataset arrives (CSV, XLSX, GDB, Shapefile, database export, API pull)
+- Before any spatial operation, statistical test, or model fitting
+- When the user asks to "clean", "validate", "check", or "prepare" ecological data
+- When merging datasets from different sources or institutions
+
+---
+
+## Inputs
+
+| Input | Format | Required |
+|-------|--------|----------|
+| Raw occurrence or survey data | CSV, XLSX, TSV, GDB | Yes |
+| Data dictionary or field protocol | PDF, DOCX, TXT | Recommended |
+| Environmental layers (if applicable) | GeoTIFF, NetCDF | Optional |
+| Existing metadata record | EML, JSON, YAML | Optional |
+
+---
+
+## Outputs
+
+| Output | Description |
+|--------|-------------|
+| `data_clean.csv` | Validated, deduplicated, standardised dataset |
+| `qa_report.md` | Summary of issues found and actions taken |
+| `schema.yaml` | Field definitions, types, valid ranges, units |
+| `metadata.xml` | EML or Dublin Core metadata record |
+| `flagged_records.csv` | Records removed or flagged, with reason codes |
+
+---
+
+## Steps
+
+### 1. Ingest and Inspect
+- Load all source files; report file sizes, row counts, column names, and data types
+- Identify encoding issues, BOM characters, delimiter inconsistencies
+- Document source provenance (institution, date, version, license)
+
+### 2. Schema Standardisation
+- Map field names to a canonical schema (Darwin Core preferred for biodiversity)
+- Enforce consistent data types (dates as ISO-8601, coordinates as decimal degrees WGS84)
+- Flag or convert non-standard units
+
+### 3. Duplicate Detection
+- Identify exact duplicates (all fields identical)
+- Identify spatial-temporal near-duplicates (same species, same coordinates, same date ± N days)
+- Document resolution strategy (keep first, keep most complete, remove all)
+
+### 4. Coordinate and Spatial QA
+- Check coordinate ranges (latitude: −90 to 90; longitude: −180 to 180)
+- Flag records with coordinates at country centroids, capital cities, or institution headquarters (likely georeferencing errors)
+- Flag records with zero coordinates (0,0)
+- Validate records fall within the stated country/region polygon
+
+### 5. Taxonomic QA
+- Check species names against a reference taxonomy (GBIF Backbone, Catalogue of Life)
+- Flag synonyms, misspellings, and higher-rank-only identifications
+- Resolve to accepted name + authorship
+
+### 6. Temporal QA
+- Check for dates in the future or before plausible survey era
+- Flag records with only year-level precision when finer precision is needed
+
+### 7. Attribute QA
+- Check numeric fields for out-of-range values (e.g., abundance < 0, DBH > 500 cm)
+- Check categorical fields for invalid entries
+- Assess missing value rates per field; flag fields above threshold (default 20%)
+
+### 8. Generate Outputs
+- Write `data_clean.csv` with action codes in an appended `QA_status` column
+- Write `qa_report.md` summarising each issue category, count, and resolution
+- Write `schema.yaml` with field definitions
+- Write `metadata.xml` in EML format
+
+---
+
+## Key Decisions to Document
+
+- Duplicate resolution strategy
+- Coordinate uncertainty threshold for exclusion
+- Temporal precision requirement
+- Missing value threshold per field
+- Taxonomy backbone version used
+
+---
+
+## Tools and Libraries
+
+**R:** `dplyr`, `readr`, `janitor`, `CoordinateCleaner`, `taxize`, `EML`  
+**Python:** `pandas`, `pyjanitor`, `geopy`, `pycountry`, `dwca-reader`  
+**CLI:** `csvkit`, `miller (mlr)`
+
+---
+
+## Resources
+
+- `resources/darwin-core-glossary.md` — Darwin Core field reference
+- `resources/qa-checklist.md` — printable QA checklist
+- `resources/coordinate-cleaning-flags.md` — flag code reference
+- `examples/` — example prompt invocations
+
+---
+
+## Notes
+
+- Always preserve the original raw file; never overwrite it
+- Record the exact software versions used for reproducibility
+- Large raster datasets should be handled by the `geoprocessing-for-ecology` skill after this step

package/skills/ecological-data-foundation/examples/example-prompts.md

@@ -0,0 +1,40 @@
+# Example Invocation Prompts — ecological-data-foundation
+
+## Basic Cleaning
+
+```
+Load skill: ecological-data-foundation
+Task: Clean and validate the occurrence dataset at data/raw/occurrences_raw.csv.
+Apply all standard QA checks. The target taxon is mammals. Use GBIF Backbone for taxonomy.
+Output results to data/processed/.
+```
+
+## Merging Multiple Sources
+
+```
+Load skill: ecological-data-foundation
+Task: I have three occurrence datasets from different institutions:
+  - data/raw/mnrj_mammals.csv (MNRJ herbarium export)
+  - data/raw/gbif_download.csv (GBIF Darwin Core)
+  - data/raw/fieldwork_2023.xlsx (our field data)
+Merge them into a single Darwin Core dataset. Remove duplicates and apply full QA.
+Report how many records each source contributed after cleaning.
+```
+
+## Targeted Check
+
+```
+Load skill: ecological-data-foundation
+Task: I already cleaned my data but want to run just the coordinate checks.
+File: data/processed/occ_v1.csv. Apply CoordinateCleaner flags and report.
+Do NOT modify the file; just produce a flag report.
+```
+
+## Schema Validation
+
+```
+Load skill: ecological-data-foundation
+Task: Validate that data/processed/occ_clean.csv conforms to Darwin Core.
+List any fields that are missing, misnamed, or have incorrect data types.
+Generate schema.yaml from the current file.
+```

package/skills/ecological-data-foundation/resources/coordinate-cleaning-flags.md

@@ -0,0 +1,66 @@
+# Coordinate Cleaning Flag Reference
+
+Based on the `CoordinateCleaner` R package flag system.
+
+## Flag Codes and Descriptions
+
+| Flag | Description | Default threshold | Action |
+|------|-------------|------------------|--------|
+| `.cap` | Coordinates at capital city | 0.1° radius | Flag and investigate |
+| `.cen` | Coordinates at country centroid | 0.1° radius | Flag and investigate |
+| `.gbif` | Coordinates at GBIF headquarters | 0.1° radius | Remove |
+| `.inst` | Coordinates at known herbarium/museum | 0.1° radius | Flag and investigate |
+| `.sea` | Coordinates in the ocean (for terrestrial taxa) | — | Flag and investigate |
+| `.val` | Coordinates outside valid range | lat [-90,90], lon [-180,180] | Remove |
+| `.zero` | Coordinates exactly at (0, 0) | — | Remove |
+| `.equ` | Identical lat and lon values | — | Flag and investigate |
+| `.dup` | Identical coordinates to another record | — | Flag; keep one |
+| `.env` | Environmental outlier (extreme value in predictor space) | Mahalanobis distance p < 0.025 | Flag and investigate |
+| `.out` | Spatial outlier (geographic distance from main cluster) | 7 MADs from median | Flag and investigate |
+
+## Recommended Workflow (R)
+
+```r
+library(CoordinateCleaner)
+
+flags <- clean_coordinates(
+  x = occ_df,
+  lon = "decimalLongitude",
+  lat = "decimalLatitude",
+  species = "species",
+  tests = c("capitals", "centroids", "equal", "gbif",
+            "institutions", "seas", "urban", "validity", "zeros"),
+  capitals_rad = 10000,   # 10 km radius
+  centroids_rad = 1000,   # 1 km radius
+  seas_ref = "buffland"   # use buffered land polygon
+)
+
+# Inspect flags
+summary(flags)
+occ_clean <- occ_df[flags$.summary, ]
+occ_flagged <- occ_df[!flags$.summary, ]
+```
+
+## Known Country Centroid Coordinates (South America)
+
+| Country | Approx centroid lat | Approx centroid lon |
+|---------|--------------------|--------------------|
+| Brazil | -10.333 | -53.200 |
+| Colombia | 4.099 | -72.888 |
+| Peru | -9.190 | -75.016 |
+| Bolivia | -16.290 | -63.589 |
+| Argentina | -34.000 | -64.000 |
+| Paraguay | -23.442 | -58.444 |
+
+## Ocean Check for Terrestrial Taxa
+
+Use `cc_sea()` with a buffered land polygon (0.5–1° buffer) to avoid incorrectly flagging coastal records:
+
+```r
+occ_sea_checked <- cc_sea(
+  x = occ_df,
+  lon = "decimalLongitude",
+  lat = "decimalLatitude",
+  ref = buffland  # load from CoordinateCleaner package
+)
+```

package/skills/ecological-data-foundation/resources/darwin-core-glossary.md

@@ -0,0 +1,91 @@
+# Darwin Core Field Glossary
+
+Essential fields for biodiversity occurrence data following the [Darwin Core standard](https://dwc.tdwg.org/).
+
+## Identity Fields
+
+| Field | Type | Description | Example |
+|-------|------|-------------|---------|
+| `occurrenceID` | string | Globally unique identifier for the occurrence | `urn:uuid:a948571f-...` |
+| `catalogNumber` | string | Institution-assigned identifier | `MNRJ-12345` |
+| `recordedBy` | string | Observer name(s) | `"Silva, J.R."` |
+| `recordNumber` | string | Field number assigned by the observer | `JRS-2023-001` |
+
+## Taxonomic Fields
+
+| Field | Type | Description | Example |
+|-------|------|-------------|---------|
+| `scientificName` | string | Full scientific name with authorship | `Panthera onca (Linnaeus, 1758)` |
+| `kingdom` | string | | `Animalia` |
+| `phylum` | string | | `Chordata` |
+| `class` | string | | `Mammalia` |
+| `order` | string | | `Carnivora` |
+| `family` | string | | `Felidae` |
+| `genus` | string | | `Panthera` |
+| `specificEpithet` | string | Species epithet only | `onca` |
+| `taxonRank` | string | Lowest rank of the name | `species` |
+| `vernacularName` | string | Common name | `jaguar` |
+| `taxonID` | string | Taxon identifier in a reference system | `gbif:5219243` |
+
+## Occurrence Fields
+
+| Field | Type | Description | Example |
+|-------|------|-------------|---------|
+| `basisOfRecord` | string | Nature of the record | `HumanObservation`, `PreservedSpecimen`, `MachineObservation` |
+| `occurrenceStatus` | string | Presence or absence | `present`, `absent` |
+| `individualCount` | integer | Number of individuals | `3` |
+| `sex` | string | | `male`, `female`, `undetermined` |
+| `lifeStage` | string | | `adult`, `juvenile`, `larva` |
+| `behavior` | string | Observed behavior | `foraging` |
+
+## Location Fields
+
+| Field | Type | Description | Example |
+|-------|------|-------------|---------|
+| `decimalLatitude` | float | Latitude in decimal degrees (WGS84) | `-15.7801` |
+| `decimalLongitude` | float | Longitude in decimal degrees (WGS84) | `-47.9292` |
+| `geodeticDatum` | string | Datum for coordinates | `WGS84` |
+| `coordinateUncertaintyInMeters` | float | Radius of coordinate uncertainty | `100` |
+| `coordinatePrecision` | float | Decimal places of coordinates | `0.0001` |
+| `countryCode` | string | ISO 3166-1 alpha-2 | `BR` |
+| `stateProvince` | string | State or province | `Mato Grosso` |
+| `county` | string | County or municipality | `Cáceres` |
+| `locality` | string | Specific location description | `"Fazenda São José, margem do rio"` |
+| `verbatimElevation` | string | Original elevation text | `"850-900 m"` |
+| `minimumElevationInMeters` | float | | `850` |
+| `maximumElevationInMeters` | float | | `900` |
+
+## Event Fields
+
+| Field | Type | Description | Example |
+|-------|------|-------------|---------|
+| `eventDate` | string | ISO-8601 date or date range | `2023-07-15`, `2023-07/2023-08` |
+| `year` | integer | | `2023` |
+| `month` | integer | | `7` |
+| `day` | integer | | `15` |
+| `eventTime` | string | Time of observation | `14:30:00-03:00` |
+| `samplingProtocol` | string | Method used | `"point count"`, `"camera trap"` |
+| `samplingEffort` | string | Effort description | `"3 nights, 1 trap"` |
+
+## Data Quality Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `identificationQualifier` | string | Uncertainty in identification (`cf.`, `aff.`) |
+| `identifiedBy` | string | Who identified the specimen |
+| `dateIdentified` | string | When identification was made |
+| `datasetName` | string | Source dataset name |
+| `institutionCode` | string | Institution acronym |
+| `license` | string | Data license (CC BY 4.0, etc.) |
+| `rightsHolder` | string | Owner of the data rights |
+
+## Common Issues and Fixes
+
+| Issue | Detection | Fix |
+|-------|-----------|-----|
+| Coordinates swapped (lat/lon) | lat > 90 or < -90 | Swap columns |
+| Comma as decimal separator | `"-15,78"` | Replace `,` → `.` |
+| DMS instead of decimal | `"15°46'48"S"` | Convert to decimal |
+| Missing datum | No `geodeticDatum` | Assume WGS84; flag |
+| Future date | `eventDate` > today | Flag and investigate |
+| Country centroid | Coordinates = known centroid | Flag as `COUNTRY_CENTROID` |

package/skills/ecological-data-foundation/resources/data-citation-guide.md

@@ -0,0 +1,265 @@
+# Data Citation Guide for Ecological Occurrence Data
+
+Reference for citing biodiversity data sources correctly in publications, reports, and analytical workflows. Covers citation formats, data-use policies, and licence requirements for the six primary sources supported by this skill library.
+
+---
+
+## 1. GBIF — Global Biodiversity Information Facility
+
+**Website**: https://www.gbif.org
+**API**: https://api.gbif.org/v1/
+
+### Citation formats
+
+#### `occ_download` / `occ.download` (preferred for publications)
+GBIF issues a citable DOI for every download submitted via `occ_download` (R) or `occ.download` (Python). The DOI resolves to a stable dataset snapshot.
+
+```
+GBIF.org (YEAR) GBIF Occurrence Download.
+https://doi.org/10.15468/dl.XXXXXXX
+Accessed on YYYY-MM-DD.
+```
+
+Example (R):
+```r
+meta <- occ_download_meta(dl_key)
+doi  <- meta$doi
+cat("Citation: GBIF.org (", format(Sys.Date(), "%Y"), ") GBIF Occurrence Download. ",
+    "https://doi.org/", doi, " Accessed on ", Sys.Date(), ".\n", sep = "")
+```
+
+#### `occ_search` / `occ.search` (no DOI)
+`occ_search` does not generate a persistent DOI and **should not be used in publications**. For exploratory analysis only. Record the query date and parameters manually.
+
+```
+GBIF.org (YEAR) Occurrence records for [species]. Query via GBIF API (occ_search).
+Accessed on YYYY-MM-DD. [Not citable — re-run with occ_download for publication.]
+```
+
+### Data use policy
+- Most records: **CC BY 4.0** or **CC0**
+- Some datasets: **CC BY-NC 4.0** (check individual dataset licences at https://www.gbif.org/dataset)
+- Always attribute the original data provider, not only GBIF
+- Use the GBIF citation widget: https://www.gbif.org/citation-guidelines
+
+---
+
+## 2. iNaturalist
+
+**Website**: https://www.inaturalist.org
+**API**: https://api.inaturalist.org/v1/
+
+### Citation format
+
+iNaturalist does **not** issue download DOIs. Record the access date precisely.
+
+```
+iNaturalist contributors and the California Academy of Sciences (YEAR).
+iNaturalist Research-grade Observations. iNaturalist.org.
+Accessed on YYYY-MM-DD. https://www.inaturalist.org
+```
+
+Or, if re-exported through GBIF:
+```
+iNaturalist (YEAR) Occurrence records exported via GBIF.
+https://doi.org/10.15468/ab3s5x
+Accessed on YYYY-MM-DD.
+```
+
+### Quality grades
+| Grade | Meaning | SDM recommendation |
+|-------|---------|-------------------|
+| **Research** | ID agreed by ≥2 users, has coordinates, is not captive | Use by default |
+| **Needs ID** | Single ID or disagreement | Avoid for SDM |
+| **Casual** | Missing fields, captive, or cultivated | Exclude |
+
+### Data use policy
+- Research-grade observations: **CC BY-NC** by default (individual users may choose CC0 or CC BY)
+- Commercial use requires CC0 or CC BY records only
+- Bulk data exports for research: permitted; cite as above
+
+---
+
+## 3. eBird (Cornell Lab of Ornithology)
+
+**Website**: https://ebird.org
+**Data portal**: https://ebird.org/data/download
+
+### Citation format
+
+eBird data requires a **signed Data Use Agreement**. Access is free but must be requested.
+
+```
+eBird Basic Dataset (EBD). Version: YEAR-MM.
+Cornell Lab of Ornithology, Ithaca, New York.
+https://ebird.org/data/download
+Accessed on YYYY-MM-DD.
+```
+
+### Data use policy
+- **Non-commercial research** only under the eBird Data Access Agreement
+- Data may not be redistributed or used to create competing products
+- Publications must acknowledge eBird and include the citation above
+- Restricted species data (sensitive species, breeding season) may be withheld
+
+### Filtering recommendations for SDM
+| Filter | Recommended value | Rationale |
+|--------|------------------|-----------|
+| Protocol | Stationary, Traveling | Standardised effort |
+| Approved | TRUE | Quality-reviewed by eBird |
+| Complete checklist | TRUE | Absence implied for non-detected species |
+| Effort distance | ≤5 km | Reduces localisation uncertainty |
+| Duration | 5–300 min | Avoids very short/long checklists |
+
+---
+
+## 4. OBIS — Ocean Biodiversity Information System
+
+**Website**: https://obis.org
+**API**: https://api.obis.org/v3/
+
+### Citation format
+
+```
+OBIS (YEAR). Ocean Biodiversity Information System.
+Intergovernmental Oceanographic Commission of UNESCO.
+www.obis.org. Accessed on YYYY-MM-DD.
+```
+
+For specific datasets within OBIS, also cite the original dataset DOI available in each record's `datasetName` and `resourceID` fields.
+
+### Data use policy
+- **CC0 1.0** (public domain dedication)
+- Users encouraged (but not required) to share results with OBIS
+- Do not use OBIS data to identify precise locations of sensitive species (use aggregated records)
+
+### Quality flags to exclude
+| Flag | Meaning |
+|------|---------|
+| `NO_COORD` | Missing coordinates |
+| `ZERO_COORD` | Coordinates are 0,0 |
+| `ON_LAND` | Marine record mapped to land |
+| `DEPTH_EXCEEDS_BATH` | Depth exceeds bathymetry |
+| `COORDINATE_MISMATCH` | Textual and coordinate locations conflict |
+
+---
+
+## 5. IUCN Red List
+
+**Website**: https://www.iucnredlist.org
+**API**: https://apiv3.iucnredlist.org
+**API key**: Required — apply at https://apiv3.iucnredlist.org/
+
+### Citation format
+
+```
+IUCN (YEAR). The IUCN Red List of Threatened Species. Version YEAR-N.
+https://www.iucnredlist.org. Accessed on YYYY-MM-DD.
+```
+
+For species-specific assessments:
+```
+[Author(s)] (YEAR). [Species name]. The IUCN Red List of Threatened Species YEAR:
+[Category]. https://dx.doi.org/10.2305/IUCN.UK.[version].RLTS.[TXID].en.
+Accessed on YYYY-MM-DD.
+```
+
+### Data use policy
+- **CC BY 4.0**
+- Distribution maps and species data may not be used to create competing databases
+- API key must not be shared; each user must register independently
+- Sensitive species (CR, EN) distribution data may be partially obscured
+
+### Red List categories
+| Code | Category |
+|------|---------|
+| EX | Extinct |
+| EW | Extinct in the Wild |
+| CR | Critically Endangered |
+| EN | Endangered |
+| VU | Vulnerable |
+| NT | Near Threatened |
+| LC | Least Concern |
+| DD | Data Deficient |
+| NE | Not Evaluated |
+
+---
+
+## 6. WorldClim / CHELSA (Predictor Data)
+
+### WorldClim v2.1
+
+```
+Fick, S.E. & Hijmans, R.J. (2017). WorldClim 2: new 1-km spatial resolution climate surfaces
+for global land areas. International Journal of Climatology 37(12): 4302–4315.
+https://doi.org/10.1002/joc.5086
+```
+
+**Licence**: CC BY 4.0
+
+### CHELSA v2.1
+
+```
+Karger, D.N. et al. (2021). Global climate data at high spatial resolution (CHELSA v2.1).
+Scientific Data 8: 282. https://doi.org/10.1038/s41597-021-01084-7
+```
+
+**Licence**: CC BY 4.0
+
+---
+
+## Combining Multiple Sources
+
+When merging occurrence data from multiple sources, include the `source` and `datasetName` columns in your output so records can be traced back to their origin.
+
+### Recommended merge workflow
+```r
+library(dplyr)
+occ_all <- bind_rows(
+  read_csv("output/gbif/occurrences_raw_GBIF_Panthera_onca_20260101.csv"),
+  read_csv("output/inat/occurrences_raw_iNat_Panthera_onca_20260101.csv"),
+  read_csv("output/obis/occurrences_raw_OBIS_Chelonia_mydas_20260101.csv")
+) |>
+  distinct(species, decimalLatitude, decimalLongitude, eventDate, .keep_all = TRUE)
+```
+
+### Master data citation block (for Methods section)
+```
+Occurrence data were downloaded from GBIF (GBIF.org YEAR, doi:...), iNaturalist
+(iNaturalist contributors and CAS YEAR, accessed YYYY-MM-DD), and OBIS (OBIS YEAR,
+accessed YYYY-MM-DD). Records were merged and spatially thinned to one record per
+[resolution] grid cell. Final dataset: [n] records of [n_species] species, [year_range].
+```
+
+---
+
+## Licence Compatibility Matrix
+
+| Source | Licence | Commercial use | Redistribute | Attribution required |
+|--------|---------|----------------|-------------|----------------------|
+| GBIF (CC0) | CC0 | Yes | Yes | Strongly recommended |
+| GBIF (CC BY) | CC BY 4.0 | Yes | Yes | Yes |
+| GBIF (CC BY-NC) | CC BY-NC 4.0 | **No** | Yes | Yes |
+| iNaturalist | CC BY-NC | **No** | Yes | Yes |
+| eBird | Custom DUA | **No** | **No** | Yes |
+| OBIS | CC0 | Yes | Yes | Recommended |
+| IUCN | CC BY 4.0 | Yes | Yes* | Yes |
+| WorldClim | CC BY 4.0 | Yes | Yes | Yes |
+| CHELSA | CC BY 4.0 | Yes | Yes | Yes |
+
+*IUCN: redistribution of the full database is not permitted; individual species data may be shared with attribution.
+
+---
+
+## Quick Reference: occ_search vs occ_download (GBIF)
+
+| Aspect | `occ_search` | `occ_download` |
+|--------|-------------|---------------|
+| Speed | Immediate | Minutes to hours |
+| Record limit | 100,000 | Unlimited |
+| DOI generated | **No** | **Yes** |
+| Reproducible | **No** | **Yes** |
+| Recommended for | Exploration | Publication |
+| Credentials needed | No | Yes (GBIF account) |
+
+**Rule of thumb**: Use `occ_search` for initial exploration and data assessment. Switch to `occ_download` before any analysis intended for publication.