@nuvlore/extension-bds 0.1.1

package/skills/bds-prism-service-proxy-timeout-investigation/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: bds-prism-service-proxy-timeout-investigation
+description: Use for Prism Spark on K8s driver BdsCipherClient generateKey/decrypt read timeouts against BDS bds-service-proxy, especially PRISM Delta commit failures with SocketTimeoutException or KMSClientException.
+allowed-tools:
+  - Read
+  - Glob
+  - Grep
+  - mcp__devops_tool__bds_prism_service_proxy_timeout_query_pack
+  - mcp__devops_tool__jira_api_read
+  - mcp__devops_tool__kubernetes_status_snapshot
+  - mcp__devops_tool__remote_readonly_command
+---
+
+# BDS Prism Service-Proxy Timeout Investigation
+
+Use this skill when a Prism ticket, Slack thread, or Chiron attachment shows
+Prism Spark on K8s failing with `BdsCipherClient.generateKey`,
+`BdsCipherClient.decrypt`, `java.net.SocketTimeoutException`,
+`KMSClientException: Read timed out`, or Delta commit aborts while calling BDS
+`bds-service-proxy`.
+
+## System Boundary
+
+The normal request path is:
+
+`Prism Spark on K8s driver -> BdsCipherClient -> bds-service-proxy -> tenant key/decrypt backend`
+
+A Prism driver `SocketTimeoutException` proves the client waited too long for a
+response. It does not by itself prove `bds-service-proxy` was down or slow.
+Correlate it with service-proxy logs before assigning BDS service impact.
+
+Load `references/service-proxy-timeout-runbook.md` only for SPC/Jira access,
+historical log retention, Prism/OMS RBAC, and output-table details.
+
+## Rules
+
+- Stay read-only. Do not mutate Jira, Kubernetes, pods, services, tenant state,
+  bucket data, or local cluster files.
+- This skill defines the flow; the atomic helper tool only emits validated read-only
+  Jira/Kubernetes query inputs.
+- Use Jira GET only and never print Jira tokens or Keychain payloads.
+- For customer SPC sessions, use the plan/runbook path, verify Kubernetes
+  context, `auth whoami`, and namespace `auth can-i` before reading evidence.
+- Do not rely on `kubectl logs` as negative evidence until the first returned
+  timestamp proves it covers the incident window.
+- If Kubernetes logs start after the incident window, use the runbook's bounded
+  pod-local rotated-log reads. Do not copy logs out or write files.
+- Separate client-side Prism timeout, service-side errors, same-tenant
+  traffic, and Prism/OMS wrapper errors.
+
+## Workflow
+
+1. Parse Jira key, cluster, tenant, CID, Spark appId, Prism pod label, activity
+   id, and incident UTC from Slack/Jira/Chiron.
+2. Generate validated read-only query inputs with the atomic helper tool.
+3. Read Jira issue metadata, comments, and bounded Chiron attachment snippets.
+4. Connect through SPC and verify Kubernetes context/RBAC.
+5. Check `bds-service-proxy` deployment, pods, endpoints, image, restarts, and
+   which pod covers the incident window.
+6. Check `kubectl logs` retention; use pod-local rotated logs when needed.
+7. Grep the incident minute for tenant, CID/appId where available,
+   `generatedek`, `generateKey`, `decrypt`, `SocketTimeout`, `KMSClientException`,
+   `503`, `ERROR`, and `WARN`.
+8. Check Prism/OMS RBAC before trying to read Prism pod logs or OMS dump files.
+9. Draft the Slack response. State whether BDS logs show service-side errors or
+   successful same-tenant traffic, then ask Prism to inspect Spark
+   driver/network/socket timeout behavior when BDS-side evidence is clean.
+
+## Output Standard
+
+Return Markdown with incident context, the system boundary, evidence table,
+timeline, conclusion, suggested Slack reply, and evidence gaps. The conclusion
+must distinguish client-side timeout evidence from service-side outage evidence.

package/skills/bds-prism-service-proxy-timeout-investigation/references/service-proxy-timeout-runbook.md

@@ -0,0 +1,48 @@
+# BDS Prism Service-Proxy Timeout Runbook
+
+This reference holds detailed access and log rules for
+`bds-prism-service-proxy-timeout-investigation/SKILL.md`.
+
+## Access Details
+
+- Use Jira GET only. If the default Jira Keychain route is absent, the legacy
+  Jira Keychain alias emitted by the plan may be used as a secret source. Never
+  print the token.
+- For customer SPC sessions, use `AWS_PROFILE=sso-cust`. If `spc connect` fails
+  before connecting with `ExpiredToken`, refresh with
+  `sso-cli aws --force --profile sso-cust`.
+- Use `AWS_PROFILE=sso-cust spc connect cust <cluster>`, then verify
+  `kubectl --kubeconfig=$HOME/.kube/spc-config config current-context`,
+  `auth whoami`, and namespace `auth can-i` before reading evidence.
+- Prefer `kubectl --kubeconfig=$HOME/.kube/spc-config -n bds ...` explicit
+  commands when `kcsb` is unavailable.
+
+## Historical Logs And RBAC
+
+- If Kubernetes logs start after the incident window, inspect pod-local rotated
+  logs inside the long-running `bds-service-proxy` pod:
+  `/opt/bds-service-proxy/log/bds-service-proxy.log` and
+  `/opt/bds-service-proxy/log/bds-service-proxy-audit.log`.
+- Use bounded pod exec reads only: `ls`, `grep`, `head`, `tail`, `sed -n`. Do
+  not copy logs out or write files.
+- Match timestamps using the file format. Pod-local service-proxy logs use ISO
+  timestamps such as `2026-06-05T15:38:02.028Z`, while Prism attachments may use
+  `2026-06-05 15:38:02,937`.
+- For Prism/OMS dump references such as
+  `/usr/local/workday-oms/logs/dumps/<id>.dump`, first check RBAC for `prism`
+  and `oms` namespaces. If RBAC denies access, say direct dump inspection is
+  blocked and rely only on the Jira attachment excerpt.
+
+## Output Details
+
+- Jira/Chiron metadata: Delta commit path, manifest file, Spark appId, activity
+  id, OMS dump path, tenant, CID, and incident UTC.
+- SPC/RBAC: current context, identity, `bds`, `prism`, and `oms` namespace
+  access checks.
+- Service-proxy state: deployment, pods, endpoints, image, restart count, and
+  which pod covers the incident window.
+- Service logs: Kubernetes retention boundary, pod-local log file checked,
+  tenant/CID/appId grep terms, service-side errors, successful same-tenant
+  traffic, and downstream wrapper errors.
+- Conclusion should distinguish Prism Spark driver client-side timeout evidence
+  from BDS service-proxy service-side outage evidence.

package/skills/bds-scylla-tenant-bucket-investigation/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: bds-scylla-tenant-bucket-investigation
+description: Use for Scylla BDS tenant health alerts and read-only tenant bucket checks where BDM pods, bucketdetails-config, and hadoop fs S3A inspection are needed.
+allowed-tools:
+  - Read
+  - Glob
+  - Grep
+  - mcp__devops_tool__bds_scylla_tenant_bucket_command_pack
+  - mcp__devops_tool__bitbucket_api_read
+  - mcp__devops_tool__git_repo_read
+  - mcp__devops_tool__repo_text_index
+  - mcp__devops_tool__kubernetes_status_snapshot
+  - mcp__devops_tool__remote_readonly_command
+  - mcp__devops_tool__pharos_alert_read
+  - mcp__devops_tool__bds_dashboard_links
+---
+
+# BDS Scylla Tenant Bucket Investigation
+
+Use this skill for Scylla BDS tenant health alerts, tenant bucket path checks,
+missing tenant data questions, or BDM / bds-service-proxy image behavior after
+HDFS CLI removal.
+
+Load `references/tenant-bucket-runbook.md` only after generating the plan and
+identifying the incident type.
+
+## Hard Rules
+
+- Stay read-only. Do not mutate Kubernetes resources, bucket objects, local pod
+  files, tenant state, or BDS service state.
+- This skill defines the investigation flow. Use
+  `mcp__devops_tool__bds_scylla_tenant_bucket_command_pack` only to normalize
+  fields and produce validated read-only source/Kubernetes/S3A command inputs.
+- Treat BDM source code as the first source of truth for tenant-health meaning.
+  Prefer `mcp__devops_tool__bitbucket_api_read` against
+  `https://bitbucket.workday.com/projects/BDS/repos/sirius/browse`; use
+  `git_repo_read`, `repo_text_index`, and local read-only grep only for a
+  human-reviewed cache.
+- Build and verify the SPC/Kubernetes connection before remote checks. If
+  `kcsb` is unavailable, use the explicit read-only
+  `kubectl --kubeconfig=$HOME/.kube/spc-config -n bds ...` equivalent.
+- Use the BDM pod and `hadoop fs` for S3A reads. Do not use `hdfs`, `aws s3`,
+  KDC pods, or `hadoop fs -get` for this workflow.
+- Do not print tenant key material or decrypted tenant file contents unless the
+  operator explicitly asks and confirms the sensitivity boundary.
+- Before claiming SIR, Galahad, OMS, or ORS pods/logs are down or missing,
+  verify namespace RBAC with `kubectl auth can-i`.
+
+## Evidence Order
+
+1. Parse alert name, cluster, source, environment, tenant, severity, and firing
+   or resolved time from Slack, PagerDuty, Solas, or Pharos context.
+2. Confirm the operator already connected to the target Scylla environment; do
+   not run login retry loops.
+3. Generate the validated read-only command pack.
+4. Read BDM `BDS/sirius` source and record exact tenant-health file paths,
+   functions/classes, and predicates before interpreting bucket data.
+5. Collect cluster context with `kcsb get pod` and
+   `kcsb get cm bucketdetails-config -o yaml`.
+6. Count active BDS tenants separately from bucket tenant folders.
+7. Map each source predicate to S3A evidence from `hadoop fs` read commands.
+8. For decrypt/SIR failures, correlate BDM logs, bds-service-proxy logs,
+   front-door reachability, and backend namespace RBAC.
+9. Compare with another active healthy tenant when available.
+10. Return a concise RCA/current-state summary with evidence gaps and read-only
+    follow-ups.
+
+## Output
+
+Return alert context, source logic, evidence table, active-tenant versus
+bucket-folder census, code-to-data comparison, key/decrypt logs, SIR/Galahad/OMS
+boundary, healthy comparison, bucket path checked, confidence, and read-only next
+checks. Separate direct command evidence from inference.

package/skills/bds-scylla-tenant-bucket-investigation/references/tenant-bucket-runbook.md

@@ -0,0 +1,91 @@
+# BDS Scylla Tenant Bucket Runbook
+
+This reference holds scenario details for
+`bds-scylla-tenant-bucket-investigation/SKILL.md`.
+
+## Source And Tenant Health Logic
+
+- Search BDM source for `tenant health`, `TenantHealth`, `tenant_health`,
+  `unhealthy`, `brokenDsInfo`, `consistency_check`, and `operationsLog`; record
+  exact file paths, functions/classes, and predicates before interpreting data.
+- Distinguish active BDS tenant count from tenant folders in the bucket. Bucket
+  folders can be stale, disabled, or test leftovers; active tenant evidence
+  should come from BDM metrics such as `bds_stats_tenant_*{isActiveTenant="true"}`
+  or from BDM/EMS active-tenant health-check behavior.
+- For alerts on envops/test tenants, do not downplay the alert if the
+  environment is tier-0/customer-like. State both facts: environment criticality
+  and whether any real customer BDS tenant data impact was observed.
+
+## Kubernetes And Bucket Evidence
+
+- Use `kcsb get pod` and `kcsb get cm bucketdetails-config -o yaml` to anchor
+  pod names and bucket name before checking S3A paths.
+- Tenant folders are discoverable by the rule
+  `data/tenants/<ENV_NAME>.<tenantName>`, for example
+  `data/tenants/WD504-IMPL.schellman1`.
+- A normally discoverable tenant folder can contain `operationsLog.csv`, `ted`,
+  and `tmp`; use those as path-shape evidence, not as health proof.
+- Inside the BDM pod, use only read commands such as `hadoop fs -ls`,
+  `hadoop fs -cat`, `hadoop fs -head`, `hadoop fs -du`, `hadoop fs -count`,
+  `hadoop fs -stat`, `hadoop fs -test`, and `hadoop fs -text`.
+- Prefer status-only read/decrypt probes such as redirecting `hadoop fs -cat` to
+  `/dev/null` and reporting only exit status and sanitized errors.
+- Do not use `aws s3` inside BDM or bds-service-proxy pods; observed behavior is
+  `NoCredentials`.
+- Do not assume `hdfs` exists in BDM or bds-service-proxy pods. New images may
+  remove it; use `hadoop fs`.
+- Do not use `hadoop fs -get` for listing or debugging. It downloads to local
+  disk and can fail with local mkdir errors.
+- Do not use KDC pods as the default S3A inspection point. They may have `hdfs`
+  but lack `org.apache.hadoop.fs.s3a.S3AFileSystem`.
+
+## Key Fetch And SIR Boundary
+
+- For read/decrypt failures, correlate BDM logs with bds-service-proxy logs:
+  look for the SIR key endpoint, `Response code 503`, `Response status: 503`,
+  `BdsCipherClient Error in decrypt`, and `KMSClientException`.
+- For SIR key-fetch 503 cases, separate front-door reachability from the real
+  authenticated business request. From a `bds-service-proxy` pod, verify DNS and
+  HTTP reachability to the SIR service such as `sir-<env>.sir:11710`.
+- Treat manual unauthenticated 401 responses as reachability/auth-boundary
+  evidence only, not as the BDS business-flow result.
+- Use `curl -k` only for HTTPS; the observed in-cluster SIR endpoint is HTTP.
+- For SIR key-fetch 503, run bds-service-proxy-side reachability probes and RBAC
+  checks for `sir`, `galahad`, `ors`, and `oms`. If RBAC denies access, mark
+  direct backend inspection as blocked and route the tenant-specific 503 to the
+  owning channel.
+
+## Interpretation
+
+- Map each source-code predicate to concrete S3A evidence: object existence,
+  timestamp, content, empty/non-empty state, parse result, or missing path.
+- Compare with at least one other active tenant when available. A successful
+  predefined-file read and `Tenant [...] is healthy` log on another tenant is
+  evidence against a cluster-wide BDM/bucket/service-proxy outage.
+- Correlate with Pharos alert state and BDS dashboard links when the question is
+  alert-driven.
+
+## Output Details
+
+- Alert context: cluster, environment, tenant, source, severity, firing/resolved
+  window.
+- Source logic: repo URL, commit/ref when known, file path, function/class, and
+  exact unhealthy predicates.
+- Evidence table: command, target, result, interpretation.
+- Tenant census: active BDS tenants versus bucket tenant folders, clearly
+  labeled with the source of each count.
+- Code-to-data comparison: one row per predicate from source with matching S3A
+  evidence.
+- Key/decrypt logs: concrete timestamped lines for the BDM read failure,
+  bds-service-proxy authenticated SIR key-fetch failure, decrypt failure, and
+  downstream 500/KMS error when present.
+- SIR/Galahad/OMS boundary: SIR front-door DNS/HTTP reachability from
+  bds-service-proxy, manual unauthenticated probe result labeled as such, and
+  RBAC status for direct backend namespace inspection.
+- Healthy comparison: another active tenant passing the same read/decrypt path
+  when available.
+- Bucket path checked: bucket source, data root, tenant path, and consistency
+  file path when used.
+- Cause or current hypothesis, with confidence.
+- Read-only next checks and manual remediation notes, clearly marked as not
+  executed.

package/skills/bds-tenant-data-operation-log-investigation/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: bds-tenant-data-operation-log-investigation
+description: Use for read-only BDS tenant data operation log checks where TSS/OMS tenant state must be separated from PRISM/BDS HDFS data readiness.
+allowed-tools:
+  - Read
+  - Glob
+  - Grep
+  - mcp__devops_tool__bds_tenant_data_operation_log_command_pack
+  - mcp__devops_tool__bitbucket_api_read
+  - mcp__devops_tool__git_repo_read
+  - mcp__devops_tool__repo_text_index
+  - mcp__devops_tool__bds_environment_host_lookup
+  - mcp__devops_tool__bds_machine_inventory
+  - mcp__devops_tool__remote_readonly_command
+  - mcp__devops_tool__workday_boundary_login
+---
+
+# BDS Tenant Data Operation Log Investigation
+
+Use this skill to check tenant BDS data, `operationsLog.csv`, PRISM
+enable/clone history, or release-state versus BDS HDFS data mismatch.
+
+## Rules
+
+- Stay read-only. Do not mutate tenant state, HDFS paths, crypto zones, BDM,
+  Kubernetes, local files, or Keychain.
+- This skill defines the flow. Use
+  `mcp__devops_tool__bds_tenant_data_operation_log_command_pack` only for validated
+  read-only HDFS/source command inputs.
+- Keep state domains separate: TSS/OMS state comes from tenant status APIs;
+  BDS/PRISM data state comes from HDFS tenant paths and `operationsLog.csv`.
+- Never conclude BDS/PRISM data is ready only because TSS says
+  `RELEASED` / `health: OK`.
+- Operation log path: `data/tenants/<ENV_NAME>.<tenantName>/operationsLog.csv`.
+  Normal BDS tenant paths commonly contain `operationsLog.csv`, `ted`, and
+  `tmp`; missing `operationsLog.csv` or `tmp` is important evidence.
+- Read Sirius BDM source when endpoint or path semantics are unclear. Search
+  for `operationLog`, `operationLogPath`,
+  `getOperationLogRaw`, `/api/v3/bds/log`, `/v2/bds/log`, and
+  `TenantDataManager`.
+- If BDM HTTP log routes return 403 from a jump host, verify the direct HDFS
+  `operationsLog.csv` path from an authorized datanode or BDM pod before
+  drawing conclusions.
+- Before Boundary login or remote HDFS commands, resolve and confirm the
+  concrete datanode or BDM evidence host with
+  `mcp__devops_tool__bds_environment_host_lookup`. If a datanode host is
+  provided, confirm it belongs to the target cluster; if not, resolve
+  `datanode`, `bdm`, or `namenode` roles and choose the narrowest host that can
+  perform read-only HDFS checks.
+- Do not invent host tools such as `workday_host_info`, `host_info`,
+  `host_query`, or `bds_enterprise_host_lookup`. Do not use local `grep`, `ls`,
+  `Read`, `Glob`, repository search, local `/hadoop` paths, local `hdfs`, or
+  local `curl` as a substitute for host lookup. If host lookup fails, stop and
+  report the exact tool/access gap.
+- Keychain LDAP entries may be JSON. Extract only the top-level `token` field
+  locally and pass it through stdin. Never print passwords, tokens, raw
+  Keychain payloads, or sudo stdin.
+- Use `sudo su -` and `kinit-workdayadmin` only when explicitly approved by the
+  operator and only for read commands.
+- Allowed HDFS reads: `hdfs crypto -listZones`, `hdfs dfs -ls`, `-du`, `-cat`,
+  `-tail`, `-test`, `-count`, and `-stat`.
+- Do not run `hdfs dfs -rm`, `-put`, `-get`, `-cp`, `-mv`, `-mkdir`, `-touchz`,
+  `-chmod`, or `-chown`.
+
+## Workflow
+
+1. Parse environment, tenant, target BDS cluster, datanode host, HDFS authority,
+   and related comparison tenants from the request.
+2. Generate validated read-only command inputs with the atomic helper tool.
+3. If source semantics are unclear, read Sirius BDM source and confirm the
+   operation log route and HDFS path construction.
+4. Read TSS/OMS tenant state only as state-domain evidence.
+5. Resolve and confirm the datanode or BDM evidence host with
+   `bds_environment_host_lookup`.
+6. Establish datanode access and Kerberos/root context only through approved
+   credential handling.
+7. Check crypto zones for the target and related tenants.
+8. List the exact tenant data path and test for `operationsLog.csv`.
+9. Tail `operationsLog.csv` only when it exists; otherwise report the missing
+   file as direct evidence.
+10. Compare with related preview, gold, or numbered tenants when provided.
+11. Separate direct HDFS evidence from inference.
+
+## Output Standard
+
+Return Markdown with scope, separated state domains, evidence table, path-shape
+comparison, operation log timeline with exact `prism,Enable` / `prism,Clone`
+rows when present, direct-evidence conclusion, inference confidence, and
+read-only next checks. Remediation must be manual and human-reviewed.

package/skills/bds-yarn-app-health-investigation/SKILL.md

@@ -0,0 +1,77 @@
+---
+name: bds-yarn-app-health-investigation
+description: Read-only BDS YARN application host health workflow for Prism timeout requests, aggregated log host discovery, NodeManager health, and HDFS DataNode cross-checks.
+allowed-tools:
+  - Read
+  - Glob
+  - Grep
+  - mcp__devops_tool__bds_yarn_app_health_command_pack
+  - mcp__devops_tool__bds_environment_host_lookup
+  - mcp__devops_tool__bds_machine_inventory
+  - mcp__devops_tool__remote_readonly_command
+  - mcp__devops_tool__workday_boundary_login
+---
+
+# BDS YARN App Health Investigation
+
+Use this skill to cross-check YARN application host health, especially Prism
+timeout cases with an app ID such as `application_1778917335863_0195`, a BDS
+cluster, driver host, tenant, and incident UTC window.
+
+Load `references/yarn-app-health-runbook.md` only for historical-window,
+ResourceManager, aggregated-log, and output-detail rules.
+
+## Hard Rules
+
+- Strictly read-only. Do not kill YARN apps, decommission nodes, delete HDFS
+  files, copy/download full logs, restart services, edit configs, or mutate host
+  state.
+- This skill defines the investigation flow. Use `bds_yarn_app_health_command_pack`
+  only to normalize inputs and produce bounded read-only command inputs.
+- Before Boundary login or remote commands, resolve and confirm the concrete
+  driver/control host with `mcp__devops_tool__bds_environment_host_lookup`.
+  If a driver host is provided, confirm it belongs to the requested cluster. If
+  not, resolve a ResourceManager/NameNode/NodeManager/DataNode host suitable for
+  read-only YARN/HDFS status commands.
+- Check Boundary/DC readiness with `workday_boundary_login action=status` when
+  host access is needed. Run `action=login` only after explicit operator
+  approval to use LDAP PROD from Keychain.
+- Never ask for or print LDAP passwords, Keychain JSON, tokens, keytabs, or
+  sudo stdin.
+- Use MCP-safe single-command inputs from the plan. Do not combine commands with
+  semicolons, `&&`, command substitution, sudo, shell redirection, or local
+  writes.
+- Do not invent host tools such as `workday_host_info`, `host_info`,
+  `host_query`, or `bds_enterprise_host_lookup`. Do not pivot to disk alerts,
+  Jira, Bamboo, Pharos, local repository search, dashboards, local `yarn`, local
+  `hdfs`, or local `/hadoop` paths when host lookup or access is blocked.
+  Report the exact evidence gap.
+
+## Evidence Order
+
+1. Parse BDS cluster, driver host, app ID, tenant, prid/cid, and incident UTC
+   window.
+2. Resolve and confirm the driver/control host with
+   `bds_environment_host_lookup`.
+3. Check Boundary/DC readiness; login only with explicit approval.
+4. Generate the validated read-only command pack with
+   `bds_yarn_app_health_command_pack`.
+5. Confirm driver host identity, UTC time, and available `yarn`, `hdfs`, and
+   `kinit` commands.
+6. Run Kerberos initialization from the existing keytab using `kinit -kt`.
+7. Check `yarn application -status <appId>`.
+8. Prefer `yarn logs -show_application_log_info` before any full `yarn logs`
+   read.
+9. Use HDFS aggregated log file names as the app host list.
+10. Cross-check each app host with `yarn node -status <host>:45454`.
+11. Use `hdfs dfsadmin -report` for HDFS counters and to identify which app
+    hosts are also DataNodes.
+12. If incident UTC is available, check the historical window with read-only
+    kernel, NodeManager, DataNode, and long-GC log filters.
+
+## Output
+
+Return app host list, current YARN NodeManager health, HDFS cluster counters,
+DataNode matches, historical-window findings when checked, and evidence gaps.
+If only current state was checked, clearly say current health is not proof of
+historical health at the incident timestamp.

package/skills/bds-yarn-app-health-investigation/references/yarn-app-health-runbook.md

@@ -0,0 +1,53 @@
+# BDS YARN App Health Runbook
+
+This reference holds detailed rules for
+`bds-yarn-app-health-investigation/SKILL.md`.
+
+## Application State And Aggregated Logs
+
+- Treat RM/Timeline not-found separately from HDFS aggregated log availability.
+- Historical apps may be removed from RM/Timeline state. If
+  `yarn application -status` cannot find an old app, parse ResourceManager
+  `ApplicationSummary` logs instead of treating the app as absent.
+- For ResourceManager failed-app spikes, filter `ApplicationSummary` lines in
+  the incident UTC window to `applicationType=SPARK`, `finalStatus=FAILED`, and
+  diagnostics containing `exitCode: 0`; return `finishTimeUtc`, `appId`, `name`,
+  `user`, `queue`, `finalStatus`, and `appMasterHost`.
+- If owner or suffix is unknown, search only the YARN aggregated log root with
+  `hdfs dfs -find <logRoot> -name '*<appId>*'`.
+- List and size the resolved app log directory with `hdfs dfs -ls` and
+  `hdfs dfs -du -h`.
+- The tenant in Prism logs may be `WD1-PROD.<tenant>`, but the YARN app owner
+  can be a service account such as `wd1-prod_prism`.
+- Aggregated logs may use `bucket-logs-tfile/<last4>/<applicationId>` rather
+  than `<owner>/logs/<applicationId>`.
+- For `application_1778917335863_0195`, the bucket segment is `0195`.
+- A host log file around tens of GB is a driver-log outlier and can explain
+  slow full `yarn logs` reads.
+
+## Historical Incident Window
+
+- If the requester asks whether checks were current-state or around the log
+  time, run a historical incident-window check before answering.
+- Historical checks must be read-only host log reads: `journalctl -k` for the
+  exact UTC window plus bounded `grep`/`zgrep` over NodeManager and DataNode
+  logs.
+- Filter routine noise such as `DFSUtil.*duration=0`, block deletion INFO,
+  `DataNode.clienttrace`, short GC pauses, and `PacketResponder terminating`.
+- Do not include minor unrelated WARNs in requester-facing replies unless they
+  are in the incident window and plausibly tied to the HDFS/DataNode path.
+- If a broad historical scan drops SSH, retry only incomplete hosts with
+  narrower file targets and no SSH multiplexing before marking the result
+  complete.
+
+## Output Details
+
+- App host list.
+- Current YARN NodeManager state and health report summary for those hosts.
+- Current HDFS cluster counters: live DataNodes, under-replicated blocks,
+  missing blocks, low-redundancy block groups, and corrupt replicas if present.
+- Which app hosts also appear as HDFS DataNodes, with decommission status and
+  last contact when available.
+- If historical logs were checked, say so explicitly and summarize whether
+  kernel / YARN NodeManager / HDFS DataNode health errors, disk/network errors,
+  DataNode unhealthy signals, or long GC pauses were found in that window.

package/skills/pharos-alert-investigation/SKILL.md

@@ -0,0 +1,80 @@
+---
+name: pharos-alert-investigation
+description: Read Pharos firing alerts, alert rules, active silences, and BDS observability links for alert detail investigation.
+allowed-tools:
+  - Read
+  - Glob
+  - Grep
+  - mcp__devops_tool__bds_pharos_query_pack
+  - mcp__devops_tool__pharos_alert_read
+  - mcp__devops_tool__pharos_prometheus_read
+  - mcp__devops_tool__bds_dashboard_links
+  - mcp__devops_tool__bds_dashboard_query_knowledge
+  - mcp__devops_tool__bds_dashboard_query_read
+  - mcp__devops_tool__dashboard_query_knowledge
+  - mcp__devops_tool__bds_scylla_tenant_bucket_command_pack
+  - mcp__devops_tool__bds_disk_io_alert_command_pack
+---
+
+# Pharos Alert Investigation
+
+Use this skill when an operator asks what BDS alerts are firing, wants Pharos
+alert details, asks whether an alert is silenced, or needs BDS dashboard links
+for a firing alert. Also use it for natural-language BDS Pharos questions such
+as disk IO, YARN alerts, NameNode latency, or visible BDS clusters.
+
+## Rules
+
+- Use our own `mcp__devops_tool__pharos_alert_read` tool. Do not shell out to
+  `pharos_alerts_cli.py` for read workflows.
+- Treat Pharos reads as read-only. Never create, delete, or update silences,
+  routes, rules, dashboards, or alertmanager config from this skill.
+- Credentials must come from env or Keychain. Never ask for or print
+  `PHAROS_CLIENT_SECRET`.
+- Use exact label filters from the alert when available:
+  `alertname`, `source`, `bds_cluster`, `severity`, and `wd_env_physical`.
+- For natural-language requests, call `mcp__devops_tool__bds_pharos_query_pack`
+  first, then execute read-only steps until direct Pharos evidence is sufficient.
+- Use `mcp__devops_tool__pharos_prometheus_read` for arbitrary BDS metric reads:
+  instant PromQL, range PromQL, label values, or series discovery.
+- Use `mcp__devops_tool__bds_dashboard_links` for Grafana, Pharos alerting,
+  silence, and Solas links.
+- Use `mcp__devops_tool__bds_dashboard_query_read` when the user asks what a
+  dashboard panel checks.
+- Use `mcp__devops_tool__bds_dashboard_query_knowledge` to learn BDS metric
+  families from source-controlled dashboards; use generic
+  `mcp__devops_tool__dashboard_query_knowledge` for non-BDS dashboard repos.
+- For `bds_bdm_tenant_tenant_health_status_alert` or Scylla tenant unhealthy
+  alerts, use `mcp__devops_tool__bds_scylla_tenant_bucket_command_pack` before
+  bucket drill-down. Inspect BDM source predicates, then map them to BDM pod plus
+  `hadoop fs` S3A reads. Do not use `aws s3`, `hdfs`, or `hadoop fs -get`.
+- For DataNode failed-volume or BDS storage alerts, use Pharos only to confirm
+  alert/rule/metric context. Then hand the drill-down to
+  `bds-disk-io-alert-investigation` and its disk command pack, because the first
+  evidence boundary is the source host storage state.
+
+## Workflow
+
+1. Parse environment, namespace, alert name, source host, cluster, severity,
+   time range, and intent.
+2. Call `bds_pharos_query_pack` unless the user already supplied an exact tool
+   action and PromQL query.
+3. Read `firing_alerts` first with narrow filters when the request is alert-like.
+4. Read `active_silences` with the same filters to identify suppression state.
+5. Read `alert_rules` by `alertName` when the user asks why the alert fires or
+   needs expression/details.
+6. Run `pharos_prometheus_read` when the request is metric-like or exploratory.
+7. Generate BDS dashboard links from the same labels.
+8. Return a compact summary with direct evidence, missing data, and next
+   read-only drill-down.
+
+## Output Standard
+
+Return concise Markdown:
+
+- Alert: name, state, severity, source, cluster, activeAt.
+- Rule: expression, duration, labels, annotations when requested or relevant.
+- Silence: active silence id, matcher scope, createdBy, endsAt, comment.
+- Links: BDS dashboard, alert rule search, active silence query, Solas.
+- Metrics: PromQL used, result count, top values or trend summary.
+- Evidence gaps: explicitly state what was not found or not read.