@nuvlore/extension-bds 0.1.1

package/skills/bds-airflow-job-status-investigation/references/airflow-access-runbook.md

@@ -0,0 +1,35 @@
+# Airflow Access And Component Health Runbook
+
+Use for Boundary access, kubeconfig checks, Airflow component health, namespace
+events, scheduler/web logs, and generic DAG/config health evidence. Owner:
+`bds-airflow-job-status-investigation/SKILL.md`.
+
+## Access Details
+
+- Private cloud access is `bauth init` followed by ordinary `ssh <host>`.
+  Use the explicit hostname from the refreshed environment config unchanged.
+- For ATL, use the latest `BDS_ATL_CUST_AZ1` config before selecting a
+  control-plane host. Old hosts can disappear after config refresh; the learned
+  current ATL access path used `s-3m1d12123f.sys.az1.cust.atl.wd`.
+- If the default SSH config routes through a stale `Atl-Nprd-Mgmt` bastion or
+  times out during banner exchange, explicitly use `-J proxy-jump.atl.bnd`
+  rather than inventing `.bnd` service-host aliases.
+- If `kubectl get pod` as the SSH user fails with `localhost:8080 refused`,
+  treat it as missing normal-user kubeconfig, not cluster health evidence.
+- If `sudo -n` needs a password, use
+  `workday_private_cloud_sudo_readonly_command`; agents and command packs must
+  not read Keychain directly or print the saved credential.
+
+## Component Health
+
+- Capture `airflow-scheduler` and `airflow-web` readiness, status, restarts,
+  age, node placement, namespace events, and bounded scheduler/web logs.
+- Always query `-n airflow`; the default namespace can correctly show no pods.
+- `GET /airflow/health ... 200` supports webserver health.
+- `DagBag Failed to import` plus `FileNotFoundError` for
+  `/etc/hadoop/conf/bds_config.json` or
+  `/data/workday-bds-airflow/dr_to_dr_config/dr_to_dr_config.json` is a
+  DAG/config risk even when Airflow pods are healthy.
+- Prometheus `superfluous response.WriteHeader` and sensu-client read-only
+  filesystem `chown` messages are noise unless paired with pod restarts, failed
+  readiness, or Airflow container errors.

package/skills/bds-airflow-job-status-investigation/references/airflow-bas-dr-runbook.md

@@ -0,0 +1,42 @@
+# Airflow BAS DR SLA And Bad Worker Runbook
+
+Use for `bas_dr_sla_miss`, tenant-data DR copy SLA alerts, stale DR pods, bad
+worker runtime, and BAS recovery validation. Owner:
+`bds-airflow-job-status-investigation/SKILL.md`.
+
+- Treat BAS mirror snapshot lag as the symptom. Prove whether the cause is
+  DistCp/copy failure, Airflow DAG/config failure, or Kubernetes worker runtime
+  cleanup failure by collecting DR task pod placement/status,
+  `deletionTimestamp`, finalizers, pod events, task logs, and bad-node runtime
+  evidence.
+- Do not delete pods, stop host processes, restart kubelet/containerd, or
+  uncordon a bad node from this read-only workflow.
+- If remediation is requested, recommend only an approved operator path: cordon
+  the bad worker, force-delete stale completed or terminating DR task pods after
+  logs show `Command exited with return code 0` and `Marking task as SUCCESS`,
+  let replacement pods schedule on healthy workers, and keep the bad node
+  cordoned for DC follow-up.
+- Do not recommend force-deleting an actively progressing `Running` pod. First
+  read logs for source/destination snapshot ids, YARN app id, DistCp progress,
+  and success markers.
+- A Kubernetes pod can remain `Running` on a degraded node after the Airflow task
+  has already completed successfully; classify that as a stale pod only with log
+  evidence.
+- For a suspected bad worker, collect read-only node runtime evidence:
+  `kubectl get node -o wide`, pods by node, host `uptime`, `D`/`Z`/zombie
+  process counts, bounded `ps` lines for `defunct`, `runc`, `containerd`,
+  `kubelet`, `mount`, and `cgroup`, plus bounded `systemctl is-active kubelet
+  containerd` with timeout.
+- Interpret kubelet/containerd service-query timeouts, stuck `runc`/mount/cgroup
+  cleanup, and many zombie processes as node-level runtime degradation.
+- For recovery verification after approved cleanup, confirm new DR task pods on
+  healthy nodes, newer mirror snapshots such as `sYYYYMMDDHHMMr`, task logs show
+  success, and BAS lag is below the `3600s` threshold. The shorthand contract is
+  BAS lag below `3600s`.
+- For BDS ASH BAS DR SLA verification, read the live Pharos script exporter
+  source on `s-11rdzb3.sys.az1.cust.ash.wd`: scrape config
+  `/etc/pharos_script_exporter/conf.d/chef_scrapes/bds_bas_dr_sla.yaml` and run
+  `sudo /data/pharos_script_exporter/scripts/bas_dr_sla_miss_metric.rb BDS_ASH_CUST_AZ1`.
+- Count metric lines where `bds_bas_dr_sla_miss` is greater than `3600`, and
+  report `total_metrics`, `sla_missed_gt_3600`, `max_lag_seconds`,
+  `max_lag_tenant`, and top lagging tenants.

package/skills/bds-airflow-job-status-investigation/references/airflow-copy-and-migration-runbook.md

@@ -0,0 +1,63 @@
+# Airflow Copy And Migration Runbook
+
+Use for on-demand tenant copy, migration-service Slack incidents, and S3 VPCE
+copy connectivity evidence. Owner:
+`bds-airflow-job-status-investigation/SKILL.md`.
+
+## On-Demand Tenant Copy
+
+- If the control panel provides an ID like
+  `dag_tenant_data_copy_on_demand--1780713387370568`, treat the numeric suffix
+  as epoch microseconds when direct Airflow metadata does not match.
+- Convert that suffix to UTC, then inspect the scheduler container task log:
+  `/data/workday-bds-airflow/logs/dag_tenant_data_copy_on_demand/manual__<UTC_ISO>/run_tenant_data_ondemand_restore_copy/1.log`.
+- Use `Running command`, `--src-tenant`, `--dst-tenant`,
+  `Copy tenant data from cluster`, `Obtained Source Snapshot`,
+  `Latest remote matching Snapshot`, `Submitted application`, `map <percent>%`,
+  `Command exited with return code`, and `Marking task as SUCCESS` as the
+  authoritative evidence.
+- In tenant-data namespace migration, the `tenant-data-copy` command shows
+  `--src-cluster`, `--dst-cluster`, `--src-env`, `--dst-env`, `--src-tenant`,
+  and `--dst-tenant`; `Copy tenant data from cluster ...` confirms the process.
+  `Obtained Source Snapshot` and `Starting: Building listing for
+  hdfs://.../.snapshot/<snapshot>` identify the source snapshot/path.
+- A task pod that is `Ready=True`, has zero restarts, no pod events, and has
+  DistCp/YARN progress still advancing is a slow running copy rather than
+  Airflow component failure.
+
+## Migration-Service Incidents
+
+- Start from Slack thread facts: `migrationEventId`, source cluster,
+  source/target environments, tenant, migration run type, snapshot id, DAG run
+  id, failure timestamp, cutover window, and linked Jira. Treat Slack as event
+  source and Jira as tracking/follow-up evidence.
+- Validate the state chain from read-only records before recommending action:
+  `FIRST_MIGRATION`, `DIFF_MIGRATION`, `CUTOVER_MIGRATION`, bucket sync,
+  `PRISM_READY`, and event `COMPLETED`.
+- Derive the correct HDFS authority from cluster evidence. Do not guess by
+  shortening names; ATL CUST AZ1 tenant records may use
+  `hdfs://bdsatlcustaz11`, not `hdfs://bdsatlcustaz1`.
+- Read-only tenant record files commonly include `operationsLog.csv`,
+  `s3MigrationCopyRecords.txt`, `s3MigrationDagRunRecords.txt`, and
+  `s3MigrationVerificationRecords.txt`.
+- If a retry is requested during cutover, verify whether migration-service is
+  accepting only `CUTOVER_MIGRATION`; `Only cutover run allowed ... Discarding
+  message` for a discarded `DIFF_MIGRATION` is state-machine evidence.
+- Recovery needs cutover success, verification source/target file count match,
+  bucket sync completion, `PRISM_READY`, and event `COMPLETED`, not only one
+  successful copy log.
+
+## S3 VPCE Copy Failure
+
+- Compare the failing S3 VPCE endpoint and any proposed alternate S3 VPCE from
+  the same worker/source IP before canceling, retriggering, or declaring the
+  migration unblocked.
+- Use only DNS lookup, route lookup, and
+  `curl -k -sS -o /dev/null --connect-timeout ...` connect probes. Do not use
+  `aws s3 cp`, object writes, or migration retries as connectivity tests.
+- If multiple ATL workers time out to one `bucket.vpce-...s3.<region>.vpce.amazonaws.com`
+  endpoint but the same worker/source IP reaches another VPCE, treat the issue
+  as endpoint/path-specific rather than a general worker or S3 outage.
+- Interpret `http=000` plus connect timeout as failure before HTTP/S3
+  authorization; interpret an HTTP status such as `307` from the alternate
+  endpoint as proof that the worker can reach that S3 endpoint path.

package/skills/bds-airflow-job-status-investigation/references/airflow-job-status-runbook.md

@@ -0,0 +1,15 @@
+# BDS Airflow Job Status Runbook Router
+
+This file is only a router for `bds-airflow-job-status-investigation/SKILL.md`.
+Do not paste all Airflow procedure detail here. Load the smallest reference that
+matches the incident class.
+
+- Access, kubeconfig, component pods, events, scheduler/web logs:
+  `airflow-access-runbook.md`.
+- On-demand tenant copy, migration-service Slack incidents, and S3 VPCE copy
+  failures: `airflow-copy-and-migration-runbook.md`.
+- Snapshot cleanup, NameNode write locks, and PRISM `CreateDataFrame` latency:
+  `airflow-snapshot-runbook.md`.
+- `bas_dr_sla_miss`, stale DR pods, bad worker runtime, and BAS recovery
+  validation: `airflow-bas-dr-runbook.md`.
+- Requester-facing answer fields by scenario: `airflow-output-runbook.md`.

package/skills/bds-airflow-job-status-investigation/references/airflow-output-runbook.md

@@ -0,0 +1,28 @@
+# Airflow Output Runbook
+
+Use only when composing the requester-facing answer for an Airflow incident.
+Owner: `bds-airflow-job-status-investigation/SKILL.md`.
+
+- Slack event source and migration identifiers when the incident came from a
+  Slack migration-service thread.
+- Control-plane host used and whether normal-user kubectl lacked kubeconfig.
+- Airflow namespace pod readiness/status/restarts/age, events, and scheduler/web
+  log findings.
+- DAG/run state if provided.
+- Long-running task pod tenant/process, source cluster/path, destination
+  cluster/path, snapshot id, YARN app id, and latest progress when available.
+- For `bas_dr_sla_miss`: affected tenants, last stale snapshot time, stuck pod
+  status and node placement, task success/failure markers, bad worker zombie and
+  runtime evidence, cordon/remediation recommendation, replacement pod
+  placement, new mirror snapshot id, and BAS lag versus the `3600s` threshold.
+- S3 VPCE evidence when relevant: old/new endpoint hostnames, resolved IPs,
+  source worker IP, route gateway/device, curl `time_connect`, `http_code`, and
+  `errormsg`.
+- Migration-service recovery evidence when relevant: cutover status, snapshot,
+  verification file counts, bucket sync status, `PRISM_READY`, and event
+  `COMPLETED`.
+- Snapshot-cleanup latency evidence when relevant: cleanup cron/timeout from
+  cookbook_bitbucket, Airflow deleteSnapshot start/end, NameNode write-lock
+  duration/stack, fast-case metadata audit RPCs, and application timeout
+  boundary.
+- Evidence gaps and blocked checks, if any.

package/skills/bds-airflow-job-status-investigation/references/airflow-snapshot-runbook.md

@@ -0,0 +1,38 @@
+# Airflow Snapshot Cleanup And NameNode Lock Runbook
+
+Use for snapshot-cleanup latency, NameNode write-lock evidence, and PRISM
+`CreateDataFrame` timeout investigations. Owner:
+`bds-airflow-job-status-investigation/SKILL.md`.
+
+- Read the cleanup schedule from
+  `/Users/boqiang.liang/500-work/cookbook_bitbucket/bds_kube/recipes/helm_deploy.rb`.
+  For `BDS_ASH_CUST_AZ1`, the learned schedule is
+  `SCHEDULE_ASH_CUST = '0 2,3,5 * * 3-5'` and the timeout is `3600` seconds.
+- Keep the investigation read-only and collect Airflow `deleteSnapshot`
+  start/end lines, active NameNode `Longest write-lock held` lines, and the
+  `FSNamesystem.deleteSnapshot -> NameNodeRpcServer.deleteSnapshot` stack when
+  available.
+- Bound log scans by evidence windows. First extract cleaner run start/end and
+  top delete durations from `dag_bulk_snapshot_cleaner` logs with `awk` over
+  `Deleting snapshot` and `Successfully deleted, duration(ms):`; then inspect
+  only the active NameNode log shard and hourly
+  `hdfs-audit.log.<yyyy-mm-dd>-<hour>.*.gz` files that overlap those
+  timestamps.
+- Keep snapshot-cleanup searches bounded. The older shorthand
+  `hdfs-audit.log.<date>-<hour>` means the same hourly audit shard pattern as
+  `hdfs-audit.log.<yyyy-mm-dd>-<hour>.*.gz`.
+- When Airflow shows repeated snapshot deletions for the same tenant, elevated
+  per-snapshot delete durations, matching NameNode `deleteSnapshot` long
+  write-locks, and read RPCs waiting behind the lock, classify the cause as
+  snapshot backlog or large diff cleanup.
+- Do not claim exact diff/deleted-file counts unless a log line or an approved
+  bounded diff/count command proves it.
+- For Spark/Delta metadata-read evidence, use fast-case NameNode audit logs
+  before or after the lock window to prove the path issues read-lock RPCs such
+  as target `_delta_log` `getfileinfo`.
+- Treat `getFileInfo/getfileinfo`, `listStatus`, `open`, and
+  `getBlockLocations` as NameNode read-lock RPCs that wait behind the
+  `deleteSnapshot` write lock.
+- During snapshot cleanup, report `deleteSnapshot` as expected lock behavior,
+  then separately document cleanup duration, configured low-risk schedule, and
+  whether the application timeout boundary was reached.

package/skills/bds-disk-io-alert-investigation/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: bds-disk-io-alert-investigation
+description: Read-only BDS storage alert RCA workflow for kernel medium errors, HDFS DataNode failed volumes, PERC virtual disks, physical disk predictive failure, and Hadoop disk handoff.
+allowed-tools:
+  - Read
+  - Glob
+  - Grep
+  - mcp__devops_tool__bds_disk_io_alert_command_pack
+  - mcp__devops_tool__bds_environment_host_lookup
+  - mcp__devops_tool__bds_machine_inventory
+  - mcp__devops_tool__workday_boundary_login
+  - mcp__devops_tool__remote_readonly_command
+---
+
+# BDS Disk IO Alert Investigation
+
+Use this skill when the incident is a BDS storage failure: disk IO errors,
+kernel medium errors, bad blocks, HDFS DataNode failed volumes, PERC virtual
+disk issues, or physical disk predictive failure on a BDS host. For example, a
+DataNode failed-volume alert for a specific source host belongs here because
+the first RCA boundary is the host's filesystem, kernel, and controller state.
+
+Do not route DataNode failed-volume incidents to Scylla tenant-bucket, Airflow
+job-status, or generic YARN health workflows unless disk investigation evidence
+shows the symptom belongs to those domains.
+
+## Rules
+
+- Resolve the BDS evidence target host before any access or SSH attempt.
+  Preserve `active datanode`, alert `Source`, BDS cluster, namespace, and disk
+  path as separate fields. For DataNode failed-volume alerts, the active
+  DataNode host is usually the storage evidence target; the alert `Source`
+  label may be a Pharos/exporter label and must not be used as the SSH target
+  until verified.
+- The required first host lookup tool is exactly
+  `mcp__devops_tool__bds_environment_host_lookup`. Use it with the BDS cluster
+  and `datanodes` role to confirm the target host belongs to the cluster. Use
+  `mcp__devops_tool__bds_machine_inventory` only when broader machine-type
+  inventory is needed.
+- Do not invent host tools such as `workday_host_info`, `host_info`,
+  `host_query`, or `bds_enterprise_host_lookup`. Do not search the repo with
+  `grep`, `ls`, `Read`, or `Glob` to discover host information. The host
+  resolution path is the explicit MCP tool call to
+  `mcp__devops_tool__bds_environment_host_lookup`. If asked which tool to use,
+  name that exact MCP tool.
+- Treat disk paths such as `/hadoop/disk5/dfs/dn` as remote paths on the
+  resolved DataNode. Never run local `execute`, Bash, `ls`, `test`, `stat`,
+  `find`, or file reads against those paths.
+- If `mcp__devops_tool__bds_environment_host_lookup` is unavailable or fails,
+  stop and report the exact tool error as a tooling/access gap. Do not fall back
+  to local filesystem checks, repo search, or guessed host metadata.
+- For private-cloud/DC hosts, always pass the Workday infra access gate before
+  host reads: call `workday_boundary_login action=status`; if not ready, use
+  `action=login` only after explicit operator approval. Login prepares local
+  Boundary/SSHCA auth state with `bauth init`; it must not mutate remote hosts.
+- Only collect evidence. Do not unmount, repair, format, stop services, restart
+  services, mark disks offline, clear logs, or modify RAID state.
+- This skill defines the investigation flow. Use
+  `mcp__devops_tool__bds_disk_io_alert_command_pack` only as an atomic helper to
+  normalize inputs and generate the read-only command pack.
+- Prefer read-only non-sudo commands first: `lsblk`, `df`, `findmnt`, `iostat`,
+  and Dell `omreport` reads.
+- If sudo is required for `journalctl -k` or hardware health commands, use it
+  only after explicit operator approval.
+- Never print LDAP passwords, Keychain JSON, tokens, or sudo stdin.
+- Do not put secrets in SSH command arguments. Pass secrets via stdin only.
+
+## Evidence Order
+
+1. Parse alert fields: alert name, active DataNode host, alert `Source` label,
+   BDS cluster, namespace, disk path, severity, and alert time.
+2. Generate the validated read-only command pack with
+   `bds_disk_io_alert_command_pack`.
+3. Resolve and confirm the evidence target host with
+   `bds_environment_host_lookup` using the cluster and `datanodes` role. Do not
+   run SSH, Boundary login, Bash, `grep`, `ls`, or tool-discovery searches in
+   this step. If host lookup fails, stop with the exact tool error.
+4. Check Workday Boundary readiness with `workday_boundary_login action=status`.
+   Run `action=login` only after explicit operator approval, then use ordinary
+   SSH to the explicit private-cloud/DC hostname.
+5. Confirm host identity, UTC time, boot time, kernel, and sudo timestamp state
+   on the resolved remote host only.
+6. Map block devices to mount points with remote `lsblk`, `df`, and `findmnt`.
+7. Capture current IO state with `iostat -xz 1 3`.
+8. Read bounded kernel and DataNode-adjacent evidence for failed volumes,
+   `critical medium error`, `Sense Key`, `Unrecovered read error`, `megaraid`,
+   OS device, sector, virtual disk, and physical disk identifiers.
+9. Read Dell controller status with `omreport storage controller`.
+10. Read virtual disk status and bad-block state with
+   `omreport storage vdisk controller=0 vdisk=<id>`.
+11. Read physical disk health with
+   `omreport storage pdisk controller=0 pdisk=<id>`.
+12. Read Dell alert log entries for `PDR64`, `VDR58`, physical disk ID, and
+    virtual disk ID.
+
+## RCA Standard
+
+Return concise Markdown:
+
+- Summary with alert, host, cluster, impacted OS device, mount point, virtual
+  disk, and physical disk.
+- Evidence table for VD status, RAID layout, bad-block state, physical disk
+  `Failure Predicted`, serial number, and part number.
+- Timeline table in UTC.
+- Current state: host reachability, filesystem mounted or not, iostat state, and
+  redundancy/risk.
+- Recommendation as manual human-reviewed action only: drain, decommission, or
+  isolate before disk replacement.
+
+## Interpretation
+
+- `Virtual Disk Bad Blocks: Yes` plus kernel `critical medium error` is a real
+  storage fault, not merely alert noise.
+- `Failure Predicted: Yes` on the physical disk is replacement evidence.
+- RAID-0 data disks have no disk redundancy. Treat continued writes as risky
+  until the platform runbook isolates or drains the affected disk/node.

package/skills/bds-hadoop-review/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: bds-hadoop-review
+description: Use when reviewing Workday BDS Hadoop upgrade work, Hadoop release-line PRs, BDS cluster impacts, Hadoop CLI removals, or BDS Hadoop-specific Bitbucket review preparation.
+tools:
+  - mcp__devops_tool__bitbucket_api_read
+  - mcp__devops_tool__jira_api_read
+  - mcp__devops_tool__bds_machine_inventory
+  - mcp__devops_tool__bds_scylla_tenant_bucket_command_pack
+  - mcp__devops_tool__git_repo_read
+  - mcp__devops_tool__repo_text_index
+  - mcp__devops_tool__apache_project_update
+  - mcp__devops_tool__pr_finding_validate
+  - mcp__devops_tool__pr_diff_grounding
+  - mcp__devops_tool__pr_changes_only_filter
+  - mcp__devops_tool__pr_comment_dedup
+  - mcp__devops_tool__pr_comment_footer
+---
+
+# BDS Hadoop Review
+
+Ground review in PR diff, linked Jira scope, BDS cluster inventory, and official
+Apache Hadoop update context when version changes are involved. Validate
+findings with `pr_finding_validate`, ground them with `pr_diff_grounding`, and
+filter with `pr_changes_only_filter` before drafting comments. Use
+`pr_comment_dedup` and `pr_comment_footer` only for local comment text.
+
+## BDM And bds-service-proxy CLI Changes
+
+When reviewing or debugging BDM and bds-service-proxy base-image changes, do not
+assume `hdfs` is present. New images may remove HDFS-specific CLI packages. For
+non-HDFS object store paths such as S3A, prefer `hadoop fs`.
+
+For BDM tenant-health questions, read the source repo before drawing data
+conclusions:
+
+- Repo: `https://bitbucket.workday.com/projects/BDS/repos/sirius/browse`
+- Prefer `bitbucket_api_read`; use a human-reviewed local cache clone only when
+  grep/indexing is needed.
+- Search for tenant health, `TenantHealth`, `tenant_health`, `unhealthy`,
+  `brokenDsInfo`, `consistency_check`, and `operationsLog`.
+- Capture exact file path, function/class, and predicate, then map each
+  predicate to S3A evidence.
+
+For Scylla tenant bucket checks, call
+`mcp__devops_tool__bds_scylla_tenant_bucket_command_pack` first. The expected
+read-only path is:
+
+1. Confirm pods with `kcsb get pod`.
+2. Read `bucketdetails-config` to get `bds-tenant.bucket.name`.
+3. Use a BDM pod such as `bdm-0`.
+4. Run `hadoop fs -ls s3a://<bds-tenant-bucket>/data/` and then tenant-specific
+   `hadoop fs -ls` or `hadoop fs -cat` reads when needed.
+
+Tenant folders follow `data/tenants/<ENV_NAME>.<tenantName>`, for example
+`data/tenants/WD504-IMPL.schellman1`. A listed tenant folder can contain
+`operationsLog.csv`, `ted`, and `tmp`.
+
+Avoid these false starts:
+
+- `aws s3 ls` inside BDM or bds-service-proxy pods; observed behavior is
+  `NoCredentials`.
+- `hdfs dfs -ls` in new BDM or bds-service-proxy images; `hdfs` may not exist.
+- KDC pods as the default S3A inspection point; they may have `hdfs` but lack
+  the right S3A classpath.
+- `hadoop fs -get` for listing; it downloads locally and can fail on local
+  directory creation.

package/skills/bds-machine-inventory/SKILL.md

@@ -0,0 +1,48 @@
+---
+name: bds-machine-inventory
+description: Use when the user asks for BDS machine lists, hostnames, CNAMEs, node types, or cluster inventory across dev, production, or non-production environments. Query the authoritative BDS config repositories through the bds_machine_inventory tool before answering concrete host or type questions.
+tools:
+  - mcp__devops_tool__bds_environment_host_lookup
+  - mcp__devops_tool__bds_machine_inventory
+  - mcp__devops_tool__bds_log_path_knowledge
+  - mcp__devops_tool__workday_boundary_login
+---
+
+# BDS Machine Inventory
+
+Use this skill for concrete BDS environment inventory questions such as:
+
+- "list all KMS machines in nprd"
+- "what types of machines exist in prod?"
+- "show node managers for cluster X"
+- "any BDS env, any machine type"
+
+## Workflow
+
+1. Normalize the environment to `dev`, `production`, or `non-production`.
+2. If the request asks for concrete private-cloud/DC hosts for a BDS cluster,
+   call `bds_environment_host_lookup` first. Use roles such as
+   `kube_control_planes`, `kube_workers`, `etcd`, `namenodes`, `datanodes`,
+   `resourcemanagers`, or `nodemanagers`.
+3. If the request asks what machine types exist, call `bds_machine_inventory` with `action: "list_types"`.
+4. If the request asks for hostnames by a broad inventory type, call `bds_machine_inventory` with `action: "list_hosts"` and either:
+   - `machineType` for semantic names such as `kms`, `resource_manager`, or `node_manager`
+   - `hostField` for an exact config path such as `bds.kms_nodes`
+5. Use `cluster` only as a filter. If the user did not name a cluster, leave it empty and report that the answer spans all matching cluster JSON files.
+6. Use the existing local cache or a user-provided `repoPath` by default. The
+   tool always runs `git pull --ff-only` before reading inventory so lookups use
+   the latest config code.
+7. Do not set `allowCacheUpdate: true` unless the user explicitly permits
+   creating the missing local read-only config cache and the tool call includes
+   a complete `hitlApproval` object with `approved`, `approvedBy`, `action`,
+   `target`, `reason`, `expectedImpact`, and `risk`.
+
+## Guardrails
+
+- Treat BDS config repos as the source of truth for host inventory.
+- Do not infer hostnames from memory or prior terminal output when a live config query is possible.
+- Do not print secrets, LDAP tokens, or Boundary credentials.
+- This skill lists inventory only. Use `workday-infra-access` or
+  `remote-readonly-investigation` before reading logs on a returned host.
+- Use `bds_log_path_knowledge` for static server-type log path planning, then
+  verify paths on the host with read-only commands.

package/skills/bds-oncall-followup/SKILL.md

@@ -0,0 +1,71 @@
+---
+name: bds-oncall-followup
+description: Maintain BDS on-call Confluence handoff rows and BDS follow-up Jira drafts for BDS-owned on-call issues.
+allowed-tools:
+  - Read
+  - Glob
+  - Grep
+  - mcp__devops_tool__bds_oncall_followup_payload_draft
+  - mcp__devops_tool__confluence_search_read
+  - mcp__devops_tool__confluence_content_read
+  - mcp__devops_tool__confluence_handoff_page_draft
+  - mcp__devops_tool__jira_api_read
+  - mcp__devops_tool__jira_ticket_draft
+---
+
+# BDS On-call Follow-up
+
+Use this skill when a BDS on-call issue must be recorded on the current
+Confluence handoff page or when a BDS-owned follow-up Jira needs to be drafted.
+
+Load `references/oncall-followup-runbook.md` only for incident-specific row,
+Jira, and verification details.
+
+## Boundaries
+
+- Keep content BDS-only. Exclude DB, MTP, TAR, rsync, or customer workflow
+  detail unless it explains a concrete BDS code or workflow gap.
+- Use `bds_oncall_followup_payload_draft` first. It owns on-call range calculation,
+  report title shape, row normalization, and BDS follow-up Jira draft shape.
+- For Slack-sourced incidents, use the Slack thread as Issue source and
+  narrative anchor. Do not force the Issue title to be the main Jira key.
+- Confluence tools find/read current handoff pages and draft page bodies; they
+  do not create or update pages directly.
+- Jira tools read existing tickets or produce Jira-ready drafts; they do not
+  create tickets directly.
+- Do not mention Codex, assistant provenance, or internal thread/source names in
+  Confluence or Jira content. Use neutral labels such as `Discussion`.
+- Do not over-specify Jira implementation. Identify the current code or workflow
+  problem, then leave the fix design to the assignee.
+
+## Evidence Order
+
+1. Determine the current two-week range with `bds_oncall_followup_payload_draft`.
+2. Locate/read the current handoff page by exact title under parent page
+   `4259305001`; draft a child page if it does not exist.
+3. Read the current handoff table and classify rows as exact match,
+   partial/mixed issue row, or unrelated row.
+4. Read main Jira status/summary/`issuelinks` when present and prefer a linked
+   monitoring/product follow-up ticket over the main incident Jira.
+5. Search Jira for existing follow-ups by tenant, cluster, alert, and issue
+   keywords before drafting a new ticket.
+6. Draft paragraph-formatted Confluence row and BDS Jira text.
+7. If writes happen through an approved external path, read back Jira and
+   Confluence before reporting completion.
+
+## Defaults
+
+- Confluence space: `BD`.
+- Parent page id: `4259305001`.
+- BDS on-call report title format:
+  `BDS On Call Handoff YYYY/MM/DD-YYYY/MM/DD`.
+- Default two-week anchor: `2026/05/26`.
+- Follow-up Jira project: `BDS`.
+- Follow-up Jira type: `Story`.
+- Follow-up Jira component: `Devops`.
+
+## Output
+
+Return either a concise confirmation with Confluence page URL, Jira URL, and
+read-back verification, or a atomic-helper result containing the exact Jira draft and
+paragraph-formatted Confluence row draft when write access is unavailable.

package/skills/bds-oncall-followup/references/oncall-followup-runbook.md

@@ -0,0 +1,58 @@
+# BDS On-call Follow-up Runbook
+
+This reference holds row/Jira details for `bds-oncall-followup/SKILL.md`.
+
+## Confluence Row Handling
+
+- Before drafting an edit for an existing Confluence table, read the current
+  storage body and identify the exact target row. Preserve unrelated rows.
+- If one row contains multiple issues, split issues into separate rows or append
+  the new row; do not replace the whole mixed row.
+- Format every Confluence issue row as paragraphs. Split description,
+  RCA/evidence, impact, Slack/Jira links, and follow-up links into separate
+  `<p>` blocks instead of one dense paragraph per cell.
+- If an operator has already created the Jira or Confluence page outside this
+  read-only runtime, read it back and prepare link text for the systems.
+
+## Jira Handling
+
+- For a main Jira that tracks the immediate fix, read `issuelinks` before
+  deciding the report Follow-up Jira. Use the linked monitoring/product
+  follow-up ticket when present.
+- Before drafting a follow-up Jira for an approved external write path, search
+  for existing tickets with tenant, cluster, alert, and issue keywords. Reuse an
+  existing matching BDS Jira in the draft instead of proposing a duplicate.
+- If follow-up is needed and no Jira exists, draft a BDS Jira that names the
+  BDS-owned code/workflow gap and includes a short code pointer when useful.
+- The Jira should include what BDS behavior failed, why this is BDS-owned, the
+  current code/workflow block, why the bad state was left behind when known, and
+  acceptance criteria requiring tests and better diagnostics.
+- Avoid long incident timelines, non-BDS repair details, Mermaid diagrams,
+  unsupported formatting, or telling the assignee exactly how to implement.
+
+## DR Copy SLA / BAS Lag Incidents
+
+- For `bas_dr_sla_miss` or DR copy SLA incidents, include affected tenants,
+  alert threshold/lag, stale mirror snapshot time, Airflow pod status, bad
+  worker node, zombie/node runtime evidence, remediation actions, verification
+  snapshot id, BAS lag recovery below `3600s`, and DC follow-up Jira when
+  present.
+- Do not report the issue as resolved only because Slack/PagerDuty stopped
+  firing.
+- When a bad Kubernetes worker caused stale Airflow task pods, distinguish
+  read-only evidence from approved operator actions. State whether the operator
+  cordoned the node, force-deleted stale completed pods, allowed replacement
+  pods to schedule on healthy nodes, and left the bad node cordoned pending DC
+  investigation or reboot.
+- For DR copy SLA incidents, include both the immediate approval/remediation
+  Jira and the DC/node follow-up Jira when they are different tickets.
+
+## Verification
+
+- When an operator performs writes through an approved external path, read back
+  Jira and Confluence before reporting completion.
+- For Confluence, verify the page version advanced, the target row contains the
+  Slack thread, main Jira, and follow-up Jira, paragraph formatting is preserved,
+  and unrelated rows are still present.
+- Remove internal provenance wording such as `Codex` from requester-facing
+  report and Jira text.