ecological-agent-skills 3.1.0

package/skills/ecological-data-foundation/resources/gbif-data-citation-guide.md

@@ -0,0 +1,193 @@
+# GBIF Data Citation Guide
+
+How to download, cite, and document GBIF occurrence data correctly for scientific publications.
+
+---
+
+## 1. Why Citation Method Matters
+
+GBIF provides two download mechanisms with fundamentally different citability:
+
+| Method | DOI generated | Citable for publication | Reproducible | Use case |
+|---|---|---|---|---|
+| `occ_search()` / `pygbif.occurrences.search()` | **No** | No — not recommended for peer review | No (results may change) | Exploration, pilot analysis, dashboards |
+| `occ_download()` / `pygbif.occurrences.download()` | **Yes** | Yes — required for publication | Yes (snapshot frozen) | All published analyses |
+
+**Rule:** If your analysis will appear in a publication or technical report, always
+use the download API (`occ_download` / `occurrences.download`) to obtain a citable DOI.
+
+---
+
+## 2. How to Cite GBIF Data Correctly
+
+### Standard GBIF citation format (APA-style)
+
+```
+GBIF.org (YEAR) GBIF Occurrence Download. https://doi.org/10.15468/dl.XXXXXXX
+Accessed on YYYY-MM-DD.
+```
+
+**Example:**
+
+```
+GBIF.org (2024) GBIF Occurrence Download.
+https://doi.org/10.15468/dl.abc123
+Accessed on 2024-03-15.
+```
+
+### Fields required in every citation
+
+| Field | Example | Notes |
+|---|---|---|
+| Portal name | `GBIF.org` | Always "GBIF.org", not the institution name |
+| Year of download | `(2024)` | Year the download was created |
+| Record type | `GBIF Occurrence Download` | Fixed string |
+| DOI | `https://doi.org/10.15468/dl.XXXXXXX` | Full URL, not just the suffix |
+| Access date | `Accessed on 2024-03-15` | ISO 8601 date format |
+
+---
+
+## 3. Retrieving the DOI from a Download in R
+
+```r
+suppressPackageStartupMessages(library(rgbif))
+
+# Initiate download (runs asynchronously on GBIF servers)
+download_key <- occ_download(
+  pred("taxonKey", 2435098),           # GBIF taxon key for your species
+  pred("hasCoordinate", TRUE),
+  pred("occurrenceStatus", "PRESENT"),
+  pred_in("basisOfRecord", c("HUMAN_OBSERVATION",
+                              "MACHINE_OBSERVATION",
+                              "PRESERVED_SPECIMEN")),
+  pred_lt("coordinateUncertaintyInMeters", 10000),
+  format = "SIMPLE_CSV"
+)
+
+# Wait for completion (polls every 10 seconds)
+occ_download_wait(download_key)
+
+# Retrieve metadata (includes DOI)
+meta <- occ_download_meta(download_key)
+doi  <- meta$doi   # e.g., "10.15468/dl.abc123"
+cat("Download DOI:", doi, "\n")
+
+# Save DOI to metadata file for citation
+writeLines(
+  c(
+    paste("GBIF download key:", download_key),
+    paste("DOI:", doi),
+    paste("Citation: GBIF.org (", format(Sys.Date(), "%Y"), ") GBIF Occurrence Download.",
+          paste0("https://doi.org/", doi),
+          "Accessed on", format(Sys.Date(), "%Y-%m-%d")),
+    paste("Download date:", Sys.Date()),
+    paste("Species:", "Panthera onca")   # replace with your species
+  ),
+  "download_metadata.txt"
+)
+
+# Import the data
+occ_data <- occ_download_get(download_key) |> occ_download_import()
+```
+
+---
+
+## 4. Retrieving the DOI from a Download in Python
+
+```python
+import pygbif.occurrences as occ
+import time
+from pathlib import Path
+from datetime import date
+
+# Initiate download
+download_key = occ.download(
+    "taxonKey = 2435098",       # replace with actual taxon key
+    "hasCoordinate = TRUE",
+    "occurrenceStatus = PRESENT",
+    "basisOfRecord in HUMAN_OBSERVATION,MACHINE_OBSERVATION,PRESERVED_SPECIMEN",
+    "coordinateUncertaintyInMeters <= 10000"
+)
+
+# Poll until complete
+while True:
+    status = occ.download_meta(download_key[0])["status"]
+    print(f"Status: {status}")
+    if status == "SUCCEEDED":
+        break
+    elif status == "FAILED":
+        raise RuntimeError("GBIF download failed")
+    time.sleep(30)
+
+# Get DOI from metadata
+meta = occ.download_meta(download_key[0])
+doi  = meta.get("doi", "")
+print(f"Download DOI: {doi}")
+
+# Save metadata
+output_dir = Path("output/gbif")
+output_dir.mkdir(parents=True, exist_ok=True)
+
+with open(output_dir / "download_metadata.txt", "w") as f:
+    f.write(f"GBIF download key: {download_key[0]}\n")
+    f.write(f"DOI: {doi}\n")
+    f.write(f"Citation: GBIF.org ({date.today().year}) GBIF Occurrence Download. "
+            f"https://doi.org/{doi} Accessed on {date.today().isoformat()}\n")
+    f.write(f"Download date: {date.today().isoformat()}\n")
+```
+
+---
+
+## 5. Registering the DOI in data_provenance.md
+
+Every project using GBIF data must have a `data_provenance.md` at the project root.
+Add an entry like:
+
+```markdown
+## GBIF Occurrence Data
+
+| Species | GBIF Taxon Key | Download Key | DOI | Download Date | Filters Applied |
+|---|---|---|---|---|---|
+| *Panthera onca* | 2435098 | 0001234-240101 | [10.15468/dl.abc123](https://doi.org/10.15468/dl.abc123) | 2024-03-15 | hasCoordinate, PRESENT, uncertainty < 10 km |
+| *Chrysocyon brachyurus* | 2441050 | 0001235-240101 | [10.15468/dl.def456](https://doi.org/10.15468/dl.def456) | 2024-03-15 | hasCoordinate, PRESENT, uncertainty < 10 km |
+```
+
+---
+
+## 6. occ_search vs occ_download — Decision Guide
+
+| Situation | Use |
+|---|---|
+| Exploring data availability, checking record counts | `occ_search` (no DOI needed) |
+| Pilot/exploratory analysis not for publication | `occ_search` acceptable |
+| Analysis to be included in a paper, report, or thesis | **`occ_download` required** |
+| Dataset with > 100,000 records | **`occ_download` required** (occ_search limited to 100k) |
+| Reproducible analysis shared with collaborators | **`occ_download` required** |
+| Training an SDM for conservation planning | **`occ_download` required** |
+
+---
+
+## 7. Common Pitfalls
+
+- **Citing the GBIF portal URL instead of the DOI:** `https://www.gbif.org/occurrence/search?...` is
+  not citable. Always use the download DOI.
+- **Using `occ_search` for final analysis:** results from `occ_search` are not frozen;
+  re-running the same query months later may return different records. Only `occ_download`
+  creates a reproducible, citeable snapshot.
+- **Forgetting to record the download date:** required even when DOI is present.
+- **Not saving `download_metadata.txt`:** always save alongside the occurrence CSV.
+- **Using taxon name instead of taxon key:** names can be ambiguous. Use the GBIF
+  backbone taxon key (`occ_search(scientificName=...)$key`) for unambiguous queries.
+- **Not filtering `coordinateUncertaintyInMeters`:** records with large uncertainty
+  (> 10 km) should be excluded or handled explicitly.
+
+---
+
+## 8. References
+
+| Resource | URL |
+|---|---|
+| GBIF citation guidelines | https://www.gbif.org/citation-guidelines |
+| rgbif R package | https://docs.ropensci.org/rgbif/ |
+| pygbif Python package | https://pygbif.readthedocs.io/ |
+| GBIF DOI minting policy | https://www.gbif.org/faq?question=what-doi-does-gbif-assign-to-downloaded-data |

package/skills/ecological-data-foundation/resources/qa-checklist.md

@@ -0,0 +1,83 @@
+# QA Checklist — Ecological Data Foundation
+
+Use this checklist for every new dataset before proceeding to analysis.
+
+## 1. File and Ingest
+
+- [ ] Source file preserved in `data/raw/` (never overwrite)
+- [ ] File encoding confirmed (UTF-8 preferred)
+- [ ] Delimiter and quoting confirmed
+- [ ] Row count matches expected (check for truncation)
+- [ ] Column names documented
+
+## 2. Schema and Types
+
+- [ ] All fields mapped to Darwin Core (or equivalent standard)
+- [ ] Dates parsed as ISO-8601 strings
+- [ ] Coordinates as decimal degrees (float)
+- [ ] Country codes as ISO 3166-1 alpha-2
+- [ ] Categorical fields enumerated and valid entries listed
+
+## 3. Duplicates
+
+- [ ] Exact duplicates identified and counted
+- [ ] Spatial-temporal near-duplicates checked (same species, same coords ± X km, same date ± Y days)
+- [ ] Resolution strategy documented and applied
+
+## 4. Coordinates
+
+- [ ] Latitude in range [-90, 90]
+- [ ] Longitude in range [-180, 180]
+- [ ] Zero coordinates (0, 0) flagged
+- [ ] Country centroid coordinates flagged
+- [ ] Capital city coordinates flagged
+- [ ] Coordinates fall within stated country/region polygon
+- [ ] Coordinate uncertainty recorded where available
+
+## 5. Taxonomy
+
+- [ ] Species names checked against reference backbone (GBIF / Catalogue of Life)
+- [ ] Synonyms resolved to accepted name
+- [ ] Misspellings corrected (with original preserved)
+- [ ] Higher-rank-only identifications flagged
+- [ ] Hybrids and cultivars handled according to study scope
+
+## 6. Temporal
+
+- [ ] No dates in the future
+- [ ] No dates before plausible survey era for the taxon
+- [ ] Temporal precision meets study requirements
+- [ ] Records with year-only precision flagged if day-level is needed
+
+## 7. Attribute Ranges
+
+- [ ] Numeric fields checked for biologically impossible values
+- [ ] Missing value rate per field computed and documented
+- [ ] Fields exceeding missing value threshold (default 20%) flagged for decision
+
+## 8. Outputs
+
+- [ ] `data_clean.csv` written with `QA_status` column
+- [ ] `flagged_records.csv` written with reason codes
+- [ ] `qa_report.md` summarises issue counts and resolutions
+- [ ] `schema.yaml` documents all field definitions
+- [ ] `metadata.xml` (EML or Dublin Core) completed
+
+## QA Status Codes
+
+| Code | Meaning |
+|------|---------|
+| `OK` | Record passed all checks |
+| `COORD_CENTROID` | Coordinates at country/institution centroid |
+| `COORD_ZERO` | Coordinates are (0, 0) |
+| `COORD_OUT_OF_RANGE` | lat or lon outside valid bounds |
+| `COORD_OUTSIDE_COUNTRY` | Point falls outside declared country polygon |
+| `DATE_FUTURE` | Event date is in the future |
+| `DATE_UNLIKELY` | Event date before plausible survey era |
+| `DUPLICATE_EXACT` | Identical to another record |
+| `DUPLICATE_SPATIOTEMPORAL` | Near-duplicate (spatial-temporal proximity) |
+| `TAXON_SYNONYM` | Name is a synonym; resolved to accepted name |
+| `TAXON_MISSPELLING` | Misspelling detected and corrected |
+| `TAXON_HIGH_RANK` | Identified only to genus or higher |
+| `MISSING_COORDS` | No coordinate information |
+| `REMOVED` | Record excluded from clean dataset |

package/skills/ecological-data-foundation/scripts/clean_occurrences.R

@@ -0,0 +1,230 @@
+# ecological-agent-skills / Copyright (C) 2026 Francisco Diego Barros Barata
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+# Usage: Rscript clean_occurrences.R <raw_occurrences.csv> <output_dir> [country_code]
+
+# ── Inline logger ─────────────────────────────────────────────────────────────
+SKILL_NAME <- "ecological-data-foundation"
+.log_ts  <- function() format(Sys.time(), "[%Y-%m-%d %H:%M:%S]")
+log_info <- function(...) message(.log_ts(), " [INFO]  ", sprintf(...))
+log_warn <- function(...) message(.log_ts(), " [WARN]  ", sprintf(...))
+log_error<- function(...) message(.log_ts(), " [ERROR] ", sprintf(...))
+log_step <- function(n, d) log_info("-- STEP %d: %s", n, d)
+log_decision <- function(v, val, why) log_info("DECISION | %s = %s | %s", v, val, why)
+dir.create("logs", recursive=TRUE, showWarnings=FALSE)
+
+# Standard occurrence cleaning pipeline
+# Usage: Rscript clean_occurrences.R <input_csv> <output_dir>
+# Requires: dplyr, readr, CoordinateCleaner, taxize, janitor
+
+suppressPackageStartupMessages({
+  library(dplyr)
+  library(readr)
+  library(CoordinateCleaner)
+  library(janitor)
+})
+
+args <- commandArgs(trailingOnly = TRUE)
+input_file  <- ifelse(length(args) >= 1, args[1], "data/raw/occurrences.csv")
+output_dir  <- ifelse(length(args) >= 2, args[2], "data/processed")
+
+log_info("Script: clean_occurrences.R | Skill: %s", SKILL_NAME)
+log_info("Input file : %s", input_file)
+log_info("Output dir : %s", output_dir)
+
+# ── Input precondition check ──────────────────────────────────────────────────
+if (!file.exists(input_file)) {
+  log_error(
+    "Input nao encontrado: %s\nCausa provavel: arquivo nao gerado pelo passo anterior.\nVerifique a saida de: ecological-data-foundation (download_from_gbif)\nSkill anterior: ecological-data-foundation",
+    input_file
+  )
+  stop("Missing: ", input_file)
+}
+
+dir.create(output_dir, recursive = TRUE, showWarnings = FALSE)
+
+log_decision("input_file", input_file, "caminho passado como args[1] ou padrao")
+log_decision("output_dir", output_dir, "caminho passado como args[2] ou padrao")
+
+# ── 1. Ingest ──────────────────────────────────────────────────────────────
+log_step(1, "Ingerir dados brutos de ocorrencias")
+tryCatch({
+  raw <- read_csv(input_file, show_col_types = FALSE) |>
+    clean_names()
+  log_info("Registros brutos lidos: %d | Colunas: %d", nrow(raw), ncol(raw))
+}, error = function(e) {
+  log_error(
+    "Falha ao ler CSV de entrada: %s\nCausa provavel: arquivo corrompido ou nao e CSV valido.\nVerifique: %s\nSkill anterior: ecological-data-foundation",
+    conditionMessage(e), input_file
+  )
+  stop(e)
+})
+
+if (nrow(raw) == 0) {
+  log_warn("Arquivo de entrada nao contem registros: %s", input_file)
+}
+
+# ── 2. Require minimum columns ─────────────────────────────────────────────
+log_step(2, "Verificar colunas obrigatorias")
+tryCatch({
+  required_cols <- c("decimal_latitude", "decimal_longitude", "species")
+  missing_req <- setdiff(required_cols, names(raw))
+  if (length(missing_req) > 0) {
+    log_error(
+      "Colunas obrigatorias ausentes: %s\nCausa provavel: CSV gerado por fonte diferente ou com nomes de colunas alterados.\nVerifique o schema do arquivo: %s\nSkill anterior: ecological-data-foundation",
+      paste(missing_req, collapse = ", "), input_file
+    )
+    stop("Missing required columns: ", paste(missing_req, collapse = ", "))
+  }
+  log_info("Todas as colunas obrigatorias presentes: %s", paste(required_cols, collapse = ", "))
+}, error = function(e) {
+  log_error("Falha na verificacao de colunas: %s", conditionMessage(e))
+  stop(e)
+})
+
+# ── 3. Remove records with missing coordinates ─────────────────────────────
+log_step(3, "Remover registros sem coordenadas e converter para numerico")
+tryCatch({
+  n_before_na <- nrow(raw)
+  raw <- raw |>
+    filter(!is.na(decimal_latitude), !is.na(decimal_longitude)) |>
+    mutate(
+      decimal_latitude  = as.numeric(decimal_latitude),
+      decimal_longitude = as.numeric(decimal_longitude)
+    )
+  n_removed_na <- n_before_na - nrow(raw)
+  if (n_removed_na > 0) {
+    log_warn("Registros removidos por coordenadas ausentes: %d", n_removed_na)
+  } else {
+    log_info("Nenhum registro removido por coordenadas ausentes.")
+  }
+}, error = function(e) {
+  log_error(
+    "Falha ao filtrar coordenadas ausentes: %s\nCausa provavel: tipos de coluna inesperados no CSV.\nVerifique: %s\nSkill anterior: ecological-data-foundation",
+    conditionMessage(e), input_file
+  )
+  stop(e)
+})
+
+if (nrow(raw) < 30) {
+  log_warn("Poucos registros apos remocao de NAs (%d). Minimo recomendado para SDM: 30.", nrow(raw))
+}
+
+# ── 4. Coordinate cleaning ─────────────────────────────────────────────────
+log_step(4, "Limpeza de coordenadas com CoordinateCleaner")
+log_decision(
+  "cc_tests",
+  "capitals,centroids,equal,gbif,institutions,validity,zeros",
+  "conjunto padrao de testes para detectar registros suspeitos"
+)
+tryCatch({
+  flags <- clean_coordinates(
+    x       = raw,
+    lon     = "decimal_longitude",
+    lat     = "decimal_latitude",
+    species = "species",
+    tests   = c("capitals", "centroids", "equal", "gbif",
+                "institutions", "validity", "zeros"),
+    verbose = FALSE
+  )
+
+  clean    <- raw[flags$.summary, ] |> mutate(QA_status = "OK")
+  flagged  <- raw[!flags$.summary, ] |>
+    mutate(QA_status = apply(
+      flags[!flags$.summary, grep("^\\.", names(flags))], 1,
+      function(r) paste(names(r)[!r], collapse = "|")
+    ))
+
+  log_info("Registros limpos: %d | Registros sinalizados: %d", nrow(clean), nrow(flagged))
+  if (nrow(flagged) > 0) {
+    log_warn("%d registros sinalizados pelo CoordinateCleaner (%.1f%% do total).",
+             nrow(flagged), 100 * nrow(flagged) / nrow(raw))
+  }
+}, error = function(e) {
+  log_error(
+    "Falha na limpeza de coordenadas (CoordinateCleaner): %s\nCausa provavel: dados malformados ou pacote CoordinateCleaner nao instalado.\nVerifique: install.packages('CoordinateCleaner')\nSkill anterior: ecological-data-foundation",
+    conditionMessage(e)
+  )
+  stop(e)
+})
+
+# ── 5. Remove exact duplicates ─────────────────────────────────────────────
+log_step(5, "Remover duplicatas exatas")
+tryCatch({
+  n_before <- nrow(clean)
+  clean <- clean |> distinct(species, decimal_latitude, decimal_longitude,
+                              event_date, .keep_all = TRUE)
+  n_dup <- n_before - nrow(clean)
+  log_decision(
+    "dedup_cols",
+    "species,decimal_latitude,decimal_longitude,event_date",
+    "combinacao padrao para identificar duplicatas espaciotemporais"
+  )
+  if (n_dup > 0) {
+    log_warn("Duplicatas exatas removidas: %d", n_dup)
+  } else {
+    log_info("Nenhuma duplicata exata encontrada.")
+  }
+}, error = function(e) {
+  log_error(
+    "Falha ao remover duplicatas: %s\nCausa provavel: coluna event_date ausente ou mal formatada.\nVerifique o schema do CSV.\nSkill anterior: ecological-data-foundation",
+    conditionMessage(e)
+  )
+  stop(e)
+})
+
+if (nrow(clean) < 30) {
+  log_warn(
+    "Apenas %d registros limpos apos todas as filtragens. SDMs requerem >= 30 registros confiaveis.",
+    nrow(clean)
+  )
+}
+
+# ── 6. Write outputs ───────────────────────────────────────────────────────
+log_step(6, "Escrever arquivos de saida")
+tryCatch({
+  write_csv(clean,   file.path(output_dir, "data_clean.csv"))
+  write_csv(flagged, file.path(output_dir, "flagged_records.csv"))
+  log_info("Gravado: %s", file.path(output_dir, "data_clean.csv"))
+  log_info("Gravado: %s", file.path(output_dir, "flagged_records.csv"))
+}, error = function(e) {
+  log_error(
+    "Falha ao gravar arquivos de saida: %s\nCausa provavel: sem permissao de escrita em '%s'.\nVerifique permissoes do diretorio.\nSkill anterior: ecological-data-foundation",
+    conditionMessage(e), output_dir
+  )
+  stop(e)
+})
+
+# ── 7. QA report ──────────────────────────────────────────────────────────
+log_step(7, "Gerar relatorio de QA")
+tryCatch({
+  report <- c(
+    "# QA Report — Occurrence Cleaning",
+    "",
+    paste("- Input file:", input_file),
+    paste("- Raw records:", nrow(raw) + nrow(flagged)),
+    paste("- Exact duplicates removed:", n_dup),
+    paste("- Records flagged by CoordinateCleaner:", nrow(flagged)),
+    paste("- Clean records written:", nrow(clean)),
+    "",
+    "## Flag Counts",
+    ""
+  )
+
+  flag_cols <- grep("^\\.", names(flags), value = TRUE)
+  for (fc in flag_cols) {
+    n_fail <- sum(!flags[[fc]], na.rm = TRUE)
+    if (n_fail > 0) report <- c(report, paste0("- `", fc, "`: ", n_fail))
+  }
+
+  writeLines(report, file.path(output_dir, "qa_report.md"))
+  log_info("Gravado: %s", file.path(output_dir, "qa_report.md"))
+}, error = function(e) {
+  log_error(
+    "Falha ao gerar relatorio QA: %s\nCausa provavel: problema ao escrever no diretorio '%s'.\nSkill anterior: ecological-data-foundation",
+    conditionMessage(e), output_dir
+  )
+  stop(e)
+})
+
+log_info("Concluido. Registros limpos: %d | Sinalizados: %d", nrow(clean), nrow(flagged))