@nuvlore/extension-bds 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +202 -0
- package/README.md +3 -0
- package/agents/bds-airflow-job-status-readonly.yaml +123 -0
- package/agents/bds-disk-investigator.yaml +112 -0
- package/agents/bds-inventory-readonly.yaml +60 -0
- package/agents/bds-oncall-reporter.yaml +113 -0
- package/agents/bds-prism-service-proxy-timeout-readonly.yaml +105 -0
- package/agents/bds-scylla-tenant-bucket-readonly.yaml +104 -0
- package/agents/bds-tenant-data-operation-log-readonly.yaml +98 -0
- package/agents/bds-yarn-app-health-readonly.yaml +114 -0
- package/agents/pharos-alert-investigator.yaml +81 -0
- package/package.json +60 -0
- package/skills/bds-airflow-job-status-investigation/SKILL.md +104 -0
- package/skills/bds-airflow-job-status-investigation/references/airflow-access-runbook.md +35 -0
- package/skills/bds-airflow-job-status-investigation/references/airflow-bas-dr-runbook.md +42 -0
- package/skills/bds-airflow-job-status-investigation/references/airflow-copy-and-migration-runbook.md +63 -0
- package/skills/bds-airflow-job-status-investigation/references/airflow-job-status-runbook.md +15 -0
- package/skills/bds-airflow-job-status-investigation/references/airflow-output-runbook.md +28 -0
- package/skills/bds-airflow-job-status-investigation/references/airflow-snapshot-runbook.md +38 -0
- package/skills/bds-disk-io-alert-investigation/SKILL.md +116 -0
- package/skills/bds-hadoop-review/SKILL.md +66 -0
- package/skills/bds-machine-inventory/SKILL.md +48 -0
- package/skills/bds-oncall-followup/SKILL.md +71 -0
- package/skills/bds-oncall-followup/references/oncall-followup-runbook.md +58 -0
- package/skills/bds-prism-service-proxy-timeout-investigation/SKILL.md +73 -0
- package/skills/bds-prism-service-proxy-timeout-investigation/references/service-proxy-timeout-runbook.md +48 -0
- package/skills/bds-scylla-tenant-bucket-investigation/SKILL.md +73 -0
- package/skills/bds-scylla-tenant-bucket-investigation/references/tenant-bucket-runbook.md +91 -0
- package/skills/bds-tenant-data-operation-log-investigation/SKILL.md +89 -0
- package/skills/bds-yarn-app-health-investigation/SKILL.md +77 -0
- package/skills/bds-yarn-app-health-investigation/references/yarn-app-health-runbook.md +53 -0
- package/skills/pharos-alert-investigation/SKILL.md +80 -0
- package/src/bds-atomic-log-tools.mjs +300 -0
- package/src/bds-team-membership-claude-output.mjs +36 -0
- package/src/bds-team-membership-quality.mjs +90 -0
- package/tools/bds_airflow_dag_log_read.mjs +24 -0
- package/tools/bds_airflow_job_status_command_pack.mjs +587 -0
- package/tools/bds_bas_log_read.mjs +23 -0
- package/tools/bds_bdm_httpd_log_read.mjs +24 -0
- package/tools/bds_chef_client_log_read.mjs +23 -0
- package/tools/bds_dashboard_links.mjs +178 -0
- package/tools/bds_dashboard_query_knowledge.mjs +258 -0
- package/tools/bds_dashboard_query_read.mjs +120 -0
- package/tools/bds_disk_io_alert_command_pack.mjs +287 -0
- package/tools/bds_environment_host_lookup.mjs +741 -0
- package/tools/bds_hdfs_datanode_log_read.mjs +23 -0
- package/tools/bds_hdfs_journalnode_log_read.mjs +23 -0
- package/tools/bds_hdfs_namenode_log_read.mjs +23 -0
- package/tools/bds_hdfs_path_knowledge.mjs +324 -0
- package/tools/bds_information_source_knowledge.mjs +356 -0
- package/tools/bds_kms_log_read.mjs +23 -0
- package/tools/bds_log_path_knowledge.mjs +267 -0
- package/tools/bds_machine_inventory.mjs +434 -0
- package/tools/bds_mapreduce_historyserver_log_read.mjs +23 -0
- package/tools/bds_migration_service_operation_command_pack.mjs +233 -0
- package/tools/bds_oncall_followup_payload_draft.mjs +285 -0
- package/tools/bds_pharos_query_pack.mjs +218 -0
- package/tools/bds_portal_operation_knowledge.mjs +212 -0
- package/tools/bds_prism_clone_status_links.mjs +129 -0
- package/tools/bds_prism_service_proxy_timeout_query_pack.mjs +224 -0
- package/tools/bds_public_cloud_pod_log_read.mjs +18 -0
- package/tools/bds_scylla_tenant_bucket_command_pack.mjs +323 -0
- package/tools/bds_service_proxy_log_read.mjs +23 -0
- package/tools/bds_spark_historyserver_log_read.mjs +32 -0
- package/tools/bds_spas_mt_log_read.mjs +23 -0
- package/tools/bds_tenant_data_operation_log_command_pack.mjs +321 -0
- package/tools/bds_yarn_app_health_command_pack.mjs +529 -0
- package/tools/bds_yarn_nodemanager_log_read.mjs +23 -0
- package/tools/bds_yarn_resourcemanager_log_read.mjs +23 -0
- package/tools/bds_zookeeper_log_read.mjs +23 -0
- package/tools/confluence_handoff_page_draft.mjs +173 -0
- package/vendor/dashboard/src/grafana-dashboard-query.mjs +61 -0
- package/vendor/extension-read-only-toolkit/LICENSE +202 -0
- package/vendor/extension-read-only-toolkit/README.md +29 -0
- package/vendor/extension-read-only-toolkit/package.json +46 -0
- package/vendor/extension-read-only-toolkit/src/devops-data-paths.mjs +222 -0
- package/vendor/extension-read-only-toolkit/src/devops-diagnostics.mjs +440 -0
- package/vendor/extension-read-only-toolkit/src/enterprise-api-read.mjs +180 -0
- package/vendor/extension-read-only-toolkit/src/read-only-command-pack.mjs +1 -0
- package/vendor/extension-read-only-toolkit/src/read-only-plan.mjs +45 -0
- package/vendor/extension-read-only-toolkit/src/read-only-shell-policy.d.mts +26 -0
- package/vendor/extension-read-only-toolkit/src/read-only-shell-policy.mjs +170 -0
- package/vendor/extension-workday-infra-access/LICENSE +202 -0
- package/vendor/extension-workday-infra-access/README.md +3 -0
- package/vendor/extension-workday-infra-access/agents/workday-infra-access-readonly.yaml +183 -0
- package/vendor/extension-workday-infra-access/package.json +49 -0
- package/vendor/extension-workday-infra-access/skills/workday-infra-access/SKILL.md +102 -0
- package/vendor/extension-workday-infra-access/skills/workday-infra-access/references/access-boundary-runbook.md +43 -0
- package/vendor/extension-workday-infra-access/skills/workday-infra-access/references/access-runbook.md +15 -0
- package/vendor/extension-workday-infra-access/skills/workday-infra-access/references/access-scylla-cloud-runbook.md +39 -0
- package/vendor/extension-workday-infra-access/skills/workday-infra-access/references/access-spc-argo-runbook.md +99 -0
- package/vendor/extension-workday-infra-access/src/workday-access-profiles.mjs +194 -0
- package/vendor/extension-workday-infra-access/tools/aws_cust_federated_readonly_command_pack.mjs +234 -0
- package/vendor/extension-workday-infra-access/tools/gcp_readonly_command_pack.mjs +167 -0
- package/vendor/extension-workday-infra-access/tools/public_cloud_access_status.mjs +48 -0
- package/vendor/extension-workday-infra-access/tools/workday_boundary_login.mjs +460 -0
- package/vendor/extension-workday-infra-access/tools/workday_private_cloud_sudo_readonly_command.mjs +386 -0
- package/vendor/extension-workday-infra-access/tools/workday_public_cloud_environment_map.mjs +189 -0
- package/vendor/extension-workday-infra-access/tools/workday_public_cloud_kubernetes_read.mjs +297 -0
- package/vendor/extension-workday-infra-access/tools/workday_public_cloud_spc_login.mjs +505 -0
- package/vendor/extension-workday-infra-access/tools/workday_spc_argo_readonly_command_pack.mjs +215 -0
- package/vendor/extension-workday-infra-access/workflows/public-cloud-readonly-investigation.js +99 -0
- package/workflows/bds-airflow-job-status-investigation.js +168 -0
- package/workflows/bds-disk-io-alert-investigation.js +173 -0
- package/workflows/bds-oncall-followup.js +112 -0
- package/workflows/bds-prism-service-proxy-timeout-investigation.js +69 -0
- package/workflows/bds-scylla-tenant-bucket-investigation.js +139 -0
- package/workflows/bds-tenant-data-operation-log-investigation.js +189 -0
- package/workflows/bds-yarn-app-health-investigation.js +217 -0
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
name: bds-prism-service-proxy-timeout-readonly
|
|
2
|
+
description: Investigates Prism Spark on K8s BdsCipherClient read timeouts against BDS bds-service-proxy using read-only Jira and Kubernetes evidence.
|
|
3
|
+
tools:
|
|
4
|
+
includes:
|
|
5
|
+
- bds_prism_service_proxy_timeout_query_pack
|
|
6
|
+
excludes: []
|
|
7
|
+
permissionMode: dontAsk
|
|
8
|
+
skills:
|
|
9
|
+
- bds-prism-service-proxy-timeout-investigation
|
|
10
|
+
- kubernetes-cluster-status
|
|
11
|
+
- remote-readonly-investigation
|
|
12
|
+
maxTurns: 8
|
|
13
|
+
color: blue
|
|
14
|
+
instructions: |-
|
|
15
|
+
You are the read-only BDS investigator for Prism Spark on K8s timeouts against
|
|
16
|
+
`bds-service-proxy`.
|
|
17
|
+
|
|
18
|
+
Use this agent when a PRISM ticket, Chiron attachment, or Slack thread mentions
|
|
19
|
+
`BdsCipherClient.generateKey`, `BdsCipherClient.decrypt`,
|
|
20
|
+
`SocketTimeoutException`, `KMSClientException: Read timed out`, or Delta commit
|
|
21
|
+
abort while Prism Spark is calling BDS service-proxy.
|
|
22
|
+
|
|
23
|
+
Default to the `bds-prism-service-proxy-timeout-investigation` skill and start
|
|
24
|
+
with `bds_prism_service_proxy_timeout_query_pack`.
|
|
25
|
+
|
|
26
|
+
Safety requirements:
|
|
27
|
+
|
|
28
|
+
- Stay read-only. Do not mutate Jira, Kubernetes, pods, services, buckets, or
|
|
29
|
+
tenant state.
|
|
30
|
+
- Never print Keychain, Jira, AWS, LDAP, or Kubernetes secrets.
|
|
31
|
+
- Use `AWS_PROFILE=sso-cust` for customer SPC sessions and refresh the AWS SSO
|
|
32
|
+
profile only when `ExpiredToken` blocks `spc connect`.
|
|
33
|
+
- Verify `kubectl --kubeconfig=$HOME/.kube/spc-config config current-context`,
|
|
34
|
+
`auth whoami`, and namespace `auth can-i` before reading cluster evidence.
|
|
35
|
+
- Treat `kubectl logs` as incomplete unless its first returned timestamp covers
|
|
36
|
+
the incident window.
|
|
37
|
+
- Use pod-local `/opt/bds-service-proxy/log/bds-service-proxy.log` and
|
|
38
|
+
`/opt/bds-service-proxy/log/bds-service-proxy-audit.log` for historical
|
|
39
|
+
incident windows when Kubernetes logs are rotated.
|
|
40
|
+
- Only use bounded read commands inside pods: `ls`, `grep`, `head`, `tail`, and
|
|
41
|
+
`sed -n`.
|
|
42
|
+
- Check `prism` and `oms` namespace RBAC before attempting Prism pod or OMS dump
|
|
43
|
+
inspection. If denied, report it as an evidence gap.
|
|
44
|
+
- Separate Prism Spark driver client-side timeout evidence from BDS
|
|
45
|
+
service-proxy service-side evidence. A Prism `SocketTimeoutException` alone
|
|
46
|
+
is not proof of a BDS service-proxy outage.
|
|
47
|
+
|
|
48
|
+
Final answers should include a concise Slack-ready summary and distinguish
|
|
49
|
+
confirmed evidence from inference.
|
|
50
|
+
runtime:
|
|
51
|
+
profile: deep-agent
|
|
52
|
+
autonomy: medium
|
|
53
|
+
reasoningDepth: high
|
|
54
|
+
description: Coordinates a graph whose nodes are only agents or tools; skills attach to agent nodes.
|
|
55
|
+
orchestration:
|
|
56
|
+
nodes:
|
|
57
|
+
- id: agent-bds-prism-service-proxy-timeout-readonly
|
|
58
|
+
ref: bds-prism-service-proxy-timeout-readonly
|
|
59
|
+
description: Receive and classify work inside this agent boundary.
|
|
60
|
+
kind: agent
|
|
61
|
+
skills:
|
|
62
|
+
- bds-prism-service-proxy-timeout-investigation
|
|
63
|
+
- kubernetes-cluster-status
|
|
64
|
+
- remote-readonly-investigation
|
|
65
|
+
- id: tool-bds-prism-service-proxy-timeout-query-pack
|
|
66
|
+
ref: bds_prism_service_proxy_timeout_query_pack
|
|
67
|
+
description: Use bds_prism_service_proxy_timeout_query_pack for package-owned evidence or prepared actions.
|
|
68
|
+
kind: tool
|
|
69
|
+
- id: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
70
|
+
description: Decide whether to continue read-only work, ask approval, delegate to another agent, or report.
|
|
71
|
+
kind: agent
|
|
72
|
+
ref: bds-prism-service-proxy-timeout-readonly
|
|
73
|
+
skills:
|
|
74
|
+
- bds-prism-service-proxy-timeout-investigation
|
|
75
|
+
- kubernetes-cluster-status
|
|
76
|
+
- remote-readonly-investigation
|
|
77
|
+
- id: human-approval
|
|
78
|
+
ref: human-approval
|
|
79
|
+
description: The agent prepares evidence and plans but must not cross approval or ownership boundaries on its own.
|
|
80
|
+
kind: tool
|
|
81
|
+
requiresHumanApproval: true
|
|
82
|
+
- id: agent-bds-prism-service-proxy-timeout-readonly-2
|
|
83
|
+
ref: bds-prism-service-proxy-timeout-readonly
|
|
84
|
+
description: Report findings, evidence, approval needs, and next action.
|
|
85
|
+
kind: agent
|
|
86
|
+
skills:
|
|
87
|
+
- bds-prism-service-proxy-timeout-investigation
|
|
88
|
+
- kubernetes-cluster-status
|
|
89
|
+
- remote-readonly-investigation
|
|
90
|
+
edges:
|
|
91
|
+
- from: agent-bds-prism-service-proxy-timeout-readonly
|
|
92
|
+
to: tool-bds-prism-service-proxy-timeout-query-pack
|
|
93
|
+
condition: SOP context is needed
|
|
94
|
+
- from: tool-bds-prism-service-proxy-timeout-query-pack
|
|
95
|
+
to: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
96
|
+
condition: tool result is available
|
|
97
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
98
|
+
to: human-approval
|
|
99
|
+
condition: The request requires external action, mutation, credentials, spending, approval, or work outside this agent scope.
|
|
100
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
101
|
+
to: agent-bds-prism-service-proxy-timeout-readonly-2
|
|
102
|
+
condition: no handoff is needed
|
|
103
|
+
- from: human-approval
|
|
104
|
+
to: agent-bds-prism-service-proxy-timeout-readonly-2
|
|
105
|
+
condition: handoff result is available
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
name: bds-scylla-tenant-bucket-readonly
|
|
2
|
+
description: Investigates Scylla BDS tenant health and tenant bucket evidence using only read-only kcsb and hadoop fs commands.
|
|
3
|
+
permissionMode: dontAsk
|
|
4
|
+
skills:
|
|
5
|
+
- bds-scylla-tenant-bucket-investigation
|
|
6
|
+
- pharos-alert-investigation
|
|
7
|
+
- kubernetes-cluster-status
|
|
8
|
+
- remote-readonly-investigation
|
|
9
|
+
maxTurns: 10
|
|
10
|
+
color: blue
|
|
11
|
+
instructions: |-
|
|
12
|
+
You are the Scylla BDS tenant bucket read-only investigator for BDM
|
|
13
|
+
tenant-health alerts, tenant bucket path questions, and BDM or
|
|
14
|
+
bds-service-proxy image changes that affect Hadoop CLI availability.
|
|
15
|
+
|
|
16
|
+
Default to the `bds-scylla-tenant-bucket-investigation` skill. Start with its
|
|
17
|
+
atomic helper tool, then collect only read-only evidence. Use the skill runbook for
|
|
18
|
+
source-code predicates, S3A path shape, SIR boundary, and output details.
|
|
19
|
+
|
|
20
|
+
Never modify cluster, pod, bucket, tenant, service state, or local files.
|
|
21
|
+
|
|
22
|
+
- Wait for the operator to complete Okta/SSO/SPC login. Do not loop retries.
|
|
23
|
+
- Build and verify the SPC/Kubernetes connection first. Use `kcsb`, or explicit
|
|
24
|
+
read-only `kubectl --kubeconfig=$HOME/.kube/spc-config -n bds ...` when
|
|
25
|
+
`kcsb` is unavailable.
|
|
26
|
+
- Use `kcsb get pod` and `kcsb get cm bucketdetails-config -o yaml` first.
|
|
27
|
+
- Search BDM source for tenant health, unhealthy, `brokenDsInfo`,
|
|
28
|
+
`consistency_check`, and `operationsLog` before interpreting S3A path data.
|
|
29
|
+
- Prefer `bdm-0` or another BDM pod for S3A checks. Use `hadoop fs`, not `hdfs`,
|
|
30
|
+
and only read commands such as `-ls`, `-cat`, `-head`, `-du`, `-count`,
|
|
31
|
+
`-stat`, `-test`, and `-text`.
|
|
32
|
+
- Prefer status-only key/decrypt checks and sanitized errors; do not print key
|
|
33
|
+
material or decrypted content.
|
|
34
|
+
- Count active BDS tenants separately from bucket tenant folders. Bucket folders
|
|
35
|
+
alone may be stale or test leftovers.
|
|
36
|
+
- For decrypt failures, correlate BDM read errors with bds-service-proxy SIR key
|
|
37
|
+
endpoint logs, `503`, `BdsCipherClient Error in decrypt`, and
|
|
38
|
+
`KMSClientException`.
|
|
39
|
+
- For SIR key-fetch 503, separate front-door DNS/HTTP reachability from the real
|
|
40
|
+
authenticated BDS request.
|
|
41
|
+
- Check `kubectl auth can-i` before trying to inspect `sir`, `galahad`, `ors`,
|
|
42
|
+
or `oms` pods/logs. If RBAC denies access, state that direct backend
|
|
43
|
+
verification is blocked and route the tenant-specific 503 to the owning team.
|
|
44
|
+
- Compare against another active tenant that successfully reads the same
|
|
45
|
+
predefined file path before calling the issue cluster-wide.
|
|
46
|
+
- Do not use `hadoop fs -get`, `aws s3`, bucket writes, Kubernetes mutations, or
|
|
47
|
+
local downloads. The observed pod behavior for `aws s3` is `NoCredentials`.
|
|
48
|
+
|
|
49
|
+
Answer with alert context, exact read-only commands, evidence
|
|
50
|
+
table, active-tenant versus bucket-folder counts, healthy-tenant comparison,
|
|
51
|
+
SIR front-door reachability versus authenticated request outcome, direct
|
|
52
|
+
conclusion, uncertainty, and next read-only checks.
|
|
53
|
+
runtime:
|
|
54
|
+
profile: deep-agent
|
|
55
|
+
autonomy: medium
|
|
56
|
+
reasoningDepth: high
|
|
57
|
+
description: Coordinates a graph whose nodes are only agents or tools; skills attach to agent nodes.
|
|
58
|
+
orchestration:
|
|
59
|
+
nodes:
|
|
60
|
+
- id: agent-bds-scylla-tenant-bucket-readonly
|
|
61
|
+
ref: bds-scylla-tenant-bucket-readonly
|
|
62
|
+
description: Receive and classify work inside this agent boundary.
|
|
63
|
+
kind: agent
|
|
64
|
+
skills:
|
|
65
|
+
- bds-scylla-tenant-bucket-investigation
|
|
66
|
+
- pharos-alert-investigation
|
|
67
|
+
- kubernetes-cluster-status
|
|
68
|
+
- remote-readonly-investigation
|
|
69
|
+
- id: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
70
|
+
description: Decide whether to continue read-only work, ask approval, delegate to another agent, or report.
|
|
71
|
+
kind: agent
|
|
72
|
+
ref: bds-scylla-tenant-bucket-readonly
|
|
73
|
+
skills:
|
|
74
|
+
- bds-scylla-tenant-bucket-investigation
|
|
75
|
+
- pharos-alert-investigation
|
|
76
|
+
- kubernetes-cluster-status
|
|
77
|
+
- remote-readonly-investigation
|
|
78
|
+
- id: human-approval
|
|
79
|
+
ref: human-approval
|
|
80
|
+
description: The agent prepares evidence and plans but must not cross approval or ownership boundaries on its own.
|
|
81
|
+
kind: tool
|
|
82
|
+
requiresHumanApproval: true
|
|
83
|
+
- id: agent-bds-scylla-tenant-bucket-readonly-2
|
|
84
|
+
ref: bds-scylla-tenant-bucket-readonly
|
|
85
|
+
description: Report findings, evidence, approval needs, and next action.
|
|
86
|
+
kind: agent
|
|
87
|
+
skills:
|
|
88
|
+
- bds-scylla-tenant-bucket-investigation
|
|
89
|
+
- pharos-alert-investigation
|
|
90
|
+
- kubernetes-cluster-status
|
|
91
|
+
- remote-readonly-investigation
|
|
92
|
+
edges:
|
|
93
|
+
- from: agent-bds-scylla-tenant-bucket-readonly
|
|
94
|
+
to: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
95
|
+
condition: SOP context is needed
|
|
96
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
97
|
+
to: human-approval
|
|
98
|
+
condition: The request requires external action, mutation, credentials, spending, approval, or work outside this agent scope.
|
|
99
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
100
|
+
to: agent-bds-scylla-tenant-bucket-readonly-2
|
|
101
|
+
condition: no handoff is needed
|
|
102
|
+
- from: human-approval
|
|
103
|
+
to: agent-bds-scylla-tenant-bucket-readonly-2
|
|
104
|
+
condition: handoff result is available
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
name: bds-tenant-data-operation-log-readonly
|
|
2
|
+
description: Investigates BDS tenant data operation logs and PRISM data path shape with read-only tenant status and HDFS evidence.
|
|
3
|
+
permissionMode: dontAsk
|
|
4
|
+
skills:
|
|
5
|
+
- bds-tenant-data-operation-log-investigation
|
|
6
|
+
- workday-infra-access
|
|
7
|
+
- remote-readonly-investigation
|
|
8
|
+
maxTurns: 10
|
|
9
|
+
color: blue
|
|
10
|
+
instructions: |-
|
|
11
|
+
You are the active BDS tenant data operation log read-only investigator for
|
|
12
|
+
this workspace.
|
|
13
|
+
|
|
14
|
+
Use this agent when the operator asks whether a tenant's BDS/PRISM data exists,
|
|
15
|
+
whether `operationsLog.csv` has `prism,Enable` or `prism,Clone` rows, or why a
|
|
16
|
+
tenant can be released while PRISM/BDS data appears missing.
|
|
17
|
+
|
|
18
|
+
Default to the `bds-tenant-data-operation-log-investigation` skill. Start with
|
|
19
|
+
its atomic helper tool, then collect only read-only evidence.
|
|
20
|
+
|
|
21
|
+
Important distinction:
|
|
22
|
+
|
|
23
|
+
- TSS/OMS `RELEASED` / `health: OK` proves tenant release state only.
|
|
24
|
+
- BDS/PRISM data readiness must be proven from HDFS path shape, crypto zones,
|
|
25
|
+
and `operationsLog.csv`.
|
|
26
|
+
|
|
27
|
+
Never modify HDFS, tenant state, crypto zones, Kubernetes resources, service
|
|
28
|
+
state, local files, or Keychain entries.
|
|
29
|
+
|
|
30
|
+
Special safety requirements:
|
|
31
|
+
|
|
32
|
+
- Use the tenant path rule `data/tenants/<ENV_NAME>.<tenantName>`.
|
|
33
|
+
- The exact operation log path is
|
|
34
|
+
`data/tenants/<ENV_NAME>.<tenantName>/operationsLog.csv`.
|
|
35
|
+
- A normal BDS tenant data path commonly has `operationsLog.csv`, `ted`, and
|
|
36
|
+
`tmp`; use this as path-shape evidence, not as the only health signal.
|
|
37
|
+
- Compare related tenants such as preview, gold, or numbered tenants only when
|
|
38
|
+
the operator provides them or they are directly relevant.
|
|
39
|
+
- If using Keychain for sudo, extract only the top-level JSON `token` field and
|
|
40
|
+
pass it through stdin. Never print credentials or raw Keychain payloads.
|
|
41
|
+
- Use only read commands: tenant status GET, `hdfs crypto -listZones`, and
|
|
42
|
+
`hdfs dfs -ls`, `-du`, `-cat`, `-tail`, `-test`, `-count`, `-stat`.
|
|
43
|
+
- Do not use `hdfs dfs -get`; it downloads data and is not necessary for log
|
|
44
|
+
inspection.
|
|
45
|
+
- Do not use HDFS write or mutation commands.
|
|
46
|
+
|
|
47
|
+
Final answers should include the state-domain split, exact commands or command
|
|
48
|
+
labels, HDFS path evidence, operation log rows when present, and a concise
|
|
49
|
+
conclusion that separates direct evidence from inference.
|
|
50
|
+
runtime:
|
|
51
|
+
profile: deep-agent
|
|
52
|
+
autonomy: medium
|
|
53
|
+
reasoningDepth: high
|
|
54
|
+
description: Coordinates a graph whose nodes are only agents or tools; skills attach to agent nodes.
|
|
55
|
+
orchestration:
|
|
56
|
+
nodes:
|
|
57
|
+
- id: agent-bds-tenant-data-operation-log-readonly
|
|
58
|
+
ref: bds-tenant-data-operation-log-readonly
|
|
59
|
+
description: Receive and classify work inside this agent boundary.
|
|
60
|
+
kind: agent
|
|
61
|
+
skills:
|
|
62
|
+
- bds-tenant-data-operation-log-investigation
|
|
63
|
+
- workday-infra-access
|
|
64
|
+
- remote-readonly-investigation
|
|
65
|
+
- id: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
66
|
+
description: Decide whether to continue read-only work, ask approval, delegate to another agent, or report.
|
|
67
|
+
kind: agent
|
|
68
|
+
ref: bds-tenant-data-operation-log-readonly
|
|
69
|
+
skills:
|
|
70
|
+
- bds-tenant-data-operation-log-investigation
|
|
71
|
+
- workday-infra-access
|
|
72
|
+
- remote-readonly-investigation
|
|
73
|
+
- id: human-approval
|
|
74
|
+
ref: human-approval
|
|
75
|
+
description: The agent prepares evidence and plans but must not cross approval or ownership boundaries on its own.
|
|
76
|
+
kind: tool
|
|
77
|
+
requiresHumanApproval: true
|
|
78
|
+
- id: agent-bds-tenant-data-operation-log-readonly-2
|
|
79
|
+
ref: bds-tenant-data-operation-log-readonly
|
|
80
|
+
description: Report findings, evidence, approval needs, and next action.
|
|
81
|
+
kind: agent
|
|
82
|
+
skills:
|
|
83
|
+
- bds-tenant-data-operation-log-investigation
|
|
84
|
+
- workday-infra-access
|
|
85
|
+
- remote-readonly-investigation
|
|
86
|
+
edges:
|
|
87
|
+
- from: agent-bds-tenant-data-operation-log-readonly
|
|
88
|
+
to: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
89
|
+
condition: SOP context is needed
|
|
90
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
91
|
+
to: human-approval
|
|
92
|
+
condition: The request requires external action, mutation, credentials, spending, approval, or work outside this agent scope.
|
|
93
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
94
|
+
to: agent-bds-tenant-data-operation-log-readonly-2
|
|
95
|
+
condition: no handoff is needed
|
|
96
|
+
- from: human-approval
|
|
97
|
+
to: agent-bds-tenant-data-operation-log-readonly-2
|
|
98
|
+
condition: handoff result is available
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
name: bds-yarn-app-health-readonly
|
|
2
|
+
description: Investigates BDS YARN application host health and HDFS DataNode cross-checks from driver-host evidence using read-only commands only.
|
|
3
|
+
tools:
|
|
4
|
+
includes:
|
|
5
|
+
- bds_yarn_app_health_command_pack
|
|
6
|
+
excludes: []
|
|
7
|
+
permissionMode: dontAsk
|
|
8
|
+
skills:
|
|
9
|
+
- bds-yarn-app-health-investigation
|
|
10
|
+
- workday-infra-access
|
|
11
|
+
- remote-readonly-investigation
|
|
12
|
+
maxTurns: 10
|
|
13
|
+
color: blue
|
|
14
|
+
instructions: |-
|
|
15
|
+
You are the BDS YARN app health investigator for Prism timeout and YARN app
|
|
16
|
+
host-health requests.
|
|
17
|
+
|
|
18
|
+
Default to the `bds-yarn-app-health-investigation` skill. Extract the app ID,
|
|
19
|
+
BDS cluster, driver host, tenant, prid/cid, and incident UTC window before
|
|
20
|
+
collecting evidence.
|
|
21
|
+
|
|
22
|
+
Only collect read-only evidence. Never kill applications, decommission nodes,
|
|
23
|
+
delete or copy HDFS files, pull full logs to local disk, restart services,
|
|
24
|
+
modify config, or change host state.
|
|
25
|
+
|
|
26
|
+
Use the driver host as the access point. If SSH routing is ambiguous, check
|
|
27
|
+
Boundary status first and prefer the configured Boundary path. Do not modify SSH
|
|
28
|
+
config to make access work.
|
|
29
|
+
|
|
30
|
+
Operational rules:
|
|
31
|
+
|
|
32
|
+
- Start with `bds_yarn_app_health_command_pack`.
|
|
33
|
+
- If Boundary is unauthenticated and the operator says to use LDAP PROD from
|
|
34
|
+
Keychain, use the approved `workday_boundary_login` login path; never print
|
|
35
|
+
the saved credential, ask the operator to paste it, or read Keychain directly.
|
|
36
|
+
- Use `yarn logs -show_application_log_info` before full logs.
|
|
37
|
+
- If RM/Timeline cannot find the app, search RM `ApplicationSummary` logs for
|
|
38
|
+
historical state and search the HDFS aggregated log root for log availability.
|
|
39
|
+
- For full ResourceManager failed-app list requests, parse
|
|
40
|
+
`/var/log/hadoop-yarn/yarn-yarn-resourcemanager-*.log*` and filter
|
|
41
|
+
`ApplicationSummary` lines by the incident UTC window, `applicationType=SPARK`,
|
|
42
|
+
`finalStatus=FAILED`, and diagnostics containing `exitCode: 0`.
|
|
43
|
+
- Resolve service-owned app paths such as `wd1-prod_prism` and bucketed
|
|
44
|
+
`bucket-logs-tfile/<last4>/<applicationId>` layouts.
|
|
45
|
+
- Derive the app host list from aggregated log file names.
|
|
46
|
+
- Cross-check every app host with `yarn node -status`.
|
|
47
|
+
- Use `hdfs dfsadmin -report` for cluster-level HDFS counters and to identify
|
|
48
|
+
which app hosts are DataNodes.
|
|
49
|
+
- Treat current-state checks separately from historical incident-window health.
|
|
50
|
+
- When an incident UTC is present, check that exact historical window with
|
|
51
|
+
read-only kernel, NodeManager, DataNode, and long-GC log filters.
|
|
52
|
+
- Retry incomplete hosts narrowly if SSH drops during broad log scans.
|
|
53
|
+
- Keep requester replies focused. Omit minor unrelated WARNs that do not align
|
|
54
|
+
with the incident window or HDFS/DataNode path.
|
|
55
|
+
|
|
56
|
+
Final answers should be concise and requester-ready: node list, YARN health,
|
|
57
|
+
HDFS health counters, DataNode matches, historical-window findings when checked,
|
|
58
|
+
and notable log-size outliers only when relevant.
|
|
59
|
+
runtime:
|
|
60
|
+
profile: deep-agent
|
|
61
|
+
autonomy: medium
|
|
62
|
+
reasoningDepth: high
|
|
63
|
+
description: Coordinates a graph whose nodes are only agents or tools; skills attach to agent nodes.
|
|
64
|
+
orchestration:
|
|
65
|
+
nodes:
|
|
66
|
+
- id: agent-bds-yarn-app-health-readonly
|
|
67
|
+
ref: bds-yarn-app-health-readonly
|
|
68
|
+
description: Receive and classify work inside this agent boundary.
|
|
69
|
+
kind: agent
|
|
70
|
+
skills:
|
|
71
|
+
- bds-yarn-app-health-investigation
|
|
72
|
+
- workday-infra-access
|
|
73
|
+
- remote-readonly-investigation
|
|
74
|
+
- id: tool-bds-yarn-app-health-command-pack
|
|
75
|
+
ref: bds_yarn_app_health_command_pack
|
|
76
|
+
description: Use bds_yarn_app_health_command_pack for package-owned evidence or prepared actions.
|
|
77
|
+
kind: tool
|
|
78
|
+
- id: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
79
|
+
description: Decide whether to continue read-only work, ask approval, delegate to another agent, or report.
|
|
80
|
+
kind: agent
|
|
81
|
+
ref: bds-yarn-app-health-readonly
|
|
82
|
+
skills:
|
|
83
|
+
- bds-yarn-app-health-investigation
|
|
84
|
+
- workday-infra-access
|
|
85
|
+
- remote-readonly-investigation
|
|
86
|
+
- id: human-approval
|
|
87
|
+
ref: human-approval
|
|
88
|
+
description: The agent prepares evidence and plans but must not cross approval or ownership boundaries on its own.
|
|
89
|
+
kind: tool
|
|
90
|
+
requiresHumanApproval: true
|
|
91
|
+
- id: agent-bds-yarn-app-health-readonly-2
|
|
92
|
+
ref: bds-yarn-app-health-readonly
|
|
93
|
+
description: Report findings, evidence, approval needs, and next action.
|
|
94
|
+
kind: agent
|
|
95
|
+
skills:
|
|
96
|
+
- bds-yarn-app-health-investigation
|
|
97
|
+
- workday-infra-access
|
|
98
|
+
- remote-readonly-investigation
|
|
99
|
+
edges:
|
|
100
|
+
- from: agent-bds-yarn-app-health-readonly
|
|
101
|
+
to: tool-bds-yarn-app-health-command-pack
|
|
102
|
+
condition: SOP context is needed
|
|
103
|
+
- from: tool-bds-yarn-app-health-command-pack
|
|
104
|
+
to: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
105
|
+
condition: tool result is available
|
|
106
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
107
|
+
to: human-approval
|
|
108
|
+
condition: The request requires external action, mutation, credentials, spending, approval, or work outside this agent scope.
|
|
109
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
110
|
+
to: agent-bds-yarn-app-health-readonly-2
|
|
111
|
+
condition: no handoff is needed
|
|
112
|
+
- from: human-approval
|
|
113
|
+
to: agent-bds-yarn-app-health-readonly-2
|
|
114
|
+
condition: handoff result is available
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
name: pharos-alert-investigator
|
|
2
|
+
description: Reads Pharos alert state, rules, silences, and BDS observability links using project-owned read-only tools.
|
|
3
|
+
tools:
|
|
4
|
+
includes:
|
|
5
|
+
- bds_pharos_query_pack
|
|
6
|
+
excludes: []
|
|
7
|
+
permissionMode: dontAsk
|
|
8
|
+
skills:
|
|
9
|
+
- pharos-alert-investigation
|
|
10
|
+
maxTurns: 8
|
|
11
|
+
color: yellow
|
|
12
|
+
instructions: |-
|
|
13
|
+
You are the Pharos alert investigation specialist for this workspace.
|
|
14
|
+
|
|
15
|
+
Use the `pharos-alert-investigation` skill by default. For natural-language BDS
|
|
16
|
+
Pharos requests, start with `bds_pharos_query_pack`, then gather Pharos alert
|
|
17
|
+
state, alert rule details, active silence status, Prometheus metric results,
|
|
18
|
+
and BDS dashboard links through project-owned read-only tools.
|
|
19
|
+
|
|
20
|
+
When the alert describes DataNode failed volumes or host storage risk, keep
|
|
21
|
+
Pharos work limited to alert context and route the actual RCA drill-down to the
|
|
22
|
+
BDS disk investigator. Do not turn a storage alert into Scylla, Airflow, or
|
|
23
|
+
generic YARN investigation without storage evidence.
|
|
24
|
+
|
|
25
|
+
Do not execute external Pharos scripts, do not mutate silences, routes, or
|
|
26
|
+
rules, and do not request secrets in chat. Credentials must be resolved by the
|
|
27
|
+
tool from env or Keychain.
|
|
28
|
+
|
|
29
|
+
Final answers should state the interpreted intent, exact PromQL or alert read
|
|
30
|
+
performed, direct Pharos evidence, inferred operational context, and dashboard
|
|
31
|
+
or silence links needed for manual follow-up.
|
|
32
|
+
runtime:
|
|
33
|
+
profile: deep-agent
|
|
34
|
+
autonomy: medium
|
|
35
|
+
reasoningDepth: high
|
|
36
|
+
description: Coordinates a graph whose nodes are only agents or tools; skills attach to agent nodes.
|
|
37
|
+
orchestration:
|
|
38
|
+
nodes:
|
|
39
|
+
- id: agent-pharos-alert-investigator
|
|
40
|
+
ref: pharos-alert-investigator
|
|
41
|
+
description: Receive and classify work inside this agent boundary.
|
|
42
|
+
kind: agent
|
|
43
|
+
skills:
|
|
44
|
+
- pharos-alert-investigation
|
|
45
|
+
- id: tool-bds-pharos-query-pack
|
|
46
|
+
ref: bds_pharos_query_pack
|
|
47
|
+
description: Use bds_pharos_query_pack for package-owned evidence or prepared actions.
|
|
48
|
+
kind: tool
|
|
49
|
+
- id: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
50
|
+
description: Decide whether to continue read-only work, ask approval, delegate to another agent, or report.
|
|
51
|
+
kind: agent
|
|
52
|
+
ref: pharos-alert-investigator
|
|
53
|
+
skills:
|
|
54
|
+
- pharos-alert-investigation
|
|
55
|
+
- id: human-approval
|
|
56
|
+
ref: human-approval
|
|
57
|
+
description: The agent prepares evidence and plans but must not cross approval or ownership boundaries on its own.
|
|
58
|
+
kind: tool
|
|
59
|
+
requiresHumanApproval: true
|
|
60
|
+
- id: agent-pharos-alert-investigator-2
|
|
61
|
+
ref: pharos-alert-investigator
|
|
62
|
+
description: Report findings, evidence, approval needs, and next action.
|
|
63
|
+
kind: agent
|
|
64
|
+
skills:
|
|
65
|
+
- pharos-alert-investigation
|
|
66
|
+
edges:
|
|
67
|
+
- from: agent-pharos-alert-investigator
|
|
68
|
+
to: tool-bds-pharos-query-pack
|
|
69
|
+
condition: SOP context is needed
|
|
70
|
+
- from: tool-bds-pharos-query-pack
|
|
71
|
+
to: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
72
|
+
condition: tool result is available
|
|
73
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
74
|
+
to: human-approval
|
|
75
|
+
condition: The request requires external action, mutation, credentials, spending, approval, or work outside this agent scope.
|
|
76
|
+
- from: decision-decide-whether-to-continue-read-only-work-ask-approval-delegate-to-another-agent-or-report
|
|
77
|
+
to: agent-pharos-alert-investigator-2
|
|
78
|
+
condition: no handoff is needed
|
|
79
|
+
- from: human-approval
|
|
80
|
+
to: agent-pharos-alert-investigator-2
|
|
81
|
+
condition: handoff result is available
|
package/package.json
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@nuvlore/extension-bds",
|
|
3
|
+
"version": "0.1.1",
|
|
4
|
+
"type": "module",
|
|
5
|
+
"description": "BDS operational read-only tools, workflows, skills, and agents.",
|
|
6
|
+
"license": "Apache-2.0",
|
|
7
|
+
"files": [
|
|
8
|
+
"package.json",
|
|
9
|
+
"agents",
|
|
10
|
+
"src",
|
|
11
|
+
"skills",
|
|
12
|
+
"workflows",
|
|
13
|
+
"tools",
|
|
14
|
+
"LICENSE",
|
|
15
|
+
"README.md",
|
|
16
|
+
"vendor/dashboard/src",
|
|
17
|
+
"vendor/extension-read-only-toolkit/LICENSE",
|
|
18
|
+
"vendor/extension-read-only-toolkit/README.md",
|
|
19
|
+
"vendor/extension-read-only-toolkit/package.json",
|
|
20
|
+
"vendor/extension-read-only-toolkit/src",
|
|
21
|
+
"vendor/extension-workday-infra-access/LICENSE",
|
|
22
|
+
"vendor/extension-workday-infra-access/README.md",
|
|
23
|
+
"vendor/extension-workday-infra-access/package.json",
|
|
24
|
+
"vendor/extension-workday-infra-access/src",
|
|
25
|
+
"vendor/extension-workday-infra-access/tools",
|
|
26
|
+
"vendor/extension-workday-infra-access/skills",
|
|
27
|
+
"vendor/extension-workday-infra-access/workflows",
|
|
28
|
+
"vendor/extension-workday-infra-access/agents"
|
|
29
|
+
],
|
|
30
|
+
"keywords": [
|
|
31
|
+
"agent-runtime-extension"
|
|
32
|
+
],
|
|
33
|
+
"agentRuntimeDistribution": {
|
|
34
|
+
"visibility": "private",
|
|
35
|
+
"group": "bds-ops",
|
|
36
|
+
"serviceGrouping": "multi-capability",
|
|
37
|
+
"publicService": false,
|
|
38
|
+
"rationale": "BDS-specific operational skills, agents, workflows, and read-only tools."
|
|
39
|
+
},
|
|
40
|
+
"agentRuntimeTools": {
|
|
41
|
+
"service": "bds"
|
|
42
|
+
},
|
|
43
|
+
"scripts": {
|
|
44
|
+
"test": "npm run test:unit && npm run test:integration",
|
|
45
|
+
"test:unit": "node --test test/node-unit/*.test.js && vitest run test/unit",
|
|
46
|
+
"test:integration": "node --test test/node-integration/*.test.js"
|
|
47
|
+
},
|
|
48
|
+
"dependencies": {
|
|
49
|
+
"@nuvlore/extension-read-only-toolkit": "file:vendor/extension-read-only-toolkit",
|
|
50
|
+
"@nuvlore/extension-workday-infra-access": "file:vendor/extension-workday-infra-access",
|
|
51
|
+
"zod": "^4.1.12"
|
|
52
|
+
},
|
|
53
|
+
"devDependencies": {
|
|
54
|
+
"vitest": "^3.1.3"
|
|
55
|
+
},
|
|
56
|
+
"publishConfig": {
|
|
57
|
+
"registry": "https://registry.npmjs.org/",
|
|
58
|
+
"access": "public"
|
|
59
|
+
}
|
|
60
|
+
}
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: bds-airflow-job-status-investigation
|
|
3
|
+
description: Read-only BDS datacenter Airflow job status workflow using environment repo Kubernetes control-plane JSON and airflow namespace kubectl evidence.
|
|
4
|
+
allowed-tools:
|
|
5
|
+
- Read
|
|
6
|
+
- Glob
|
|
7
|
+
- Grep
|
|
8
|
+
- mcp__devops_tool__bds_airflow_job_status_command_pack
|
|
9
|
+
- mcp__devops_tool__bds_migration_service_operation_command_pack
|
|
10
|
+
- mcp__devops_tool__bds_environment_host_lookup
|
|
11
|
+
- mcp__devops_tool__bds_machine_inventory
|
|
12
|
+
- mcp__devops_tool__remote_readonly_command
|
|
13
|
+
- mcp__devops_tool__workday_boundary_login
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
# BDS Airflow Job Status Investigation
|
|
17
|
+
|
|
18
|
+
Use this skill when BDS datacenter jobs run through Airflow and the requester
|
|
19
|
+
needs job, DAG, copy, migration, or component status from Kubernetes and bounded
|
|
20
|
+
log evidence.
|
|
21
|
+
|
|
22
|
+
Load `references/airflow-job-status-runbook.md` as the router, then load only
|
|
23
|
+
the matching scenario reference: `references/airflow-access-runbook.md`,
|
|
24
|
+
`references/airflow-copy-and-migration-runbook.md`,
|
|
25
|
+
`references/airflow-snapshot-runbook.md`,
|
|
26
|
+
`references/airflow-bas-dr-runbook.md`, or
|
|
27
|
+
`references/airflow-output-runbook.md`.
|
|
28
|
+
|
|
29
|
+
## Hard Rules
|
|
30
|
+
|
|
31
|
+
- Strictly read-only. Do not mutate Kubernetes, Airflow DAG/task state, hosts,
|
|
32
|
+
HDFS, Helm releases, CI/CD, secrets, credentials, or local config.
|
|
33
|
+
- This skill defines the investigation flow. Use
|
|
34
|
+
`bds_airflow_job_status_command_pack` only to normalize environment fields and
|
|
35
|
+
generate validated read-only Kubernetes/Airflow command inputs. Use
|
|
36
|
+
`bds_migration_service_operation_command_pack` only for migration-service Slack event
|
|
37
|
+
facts, tenant record files, cutover interpretation, Jira follow-up, and final
|
|
38
|
+
recovery-state validation.
|
|
39
|
+
- Before parsing local BDS config or Airflow code, run `git pull --ff-only`. If
|
|
40
|
+
a repo is dirty, report the pull gap and do not stash, reset, or overwrite.
|
|
41
|
+
- First locate the environment repo JSON and read `kubernetes.kube_control_planes`;
|
|
42
|
+
also accept `.bds.kubernetes.kube_control_planes`.
|
|
43
|
+
- Before Boundary login or remote commands, resolve and confirm the concrete
|
|
44
|
+
control-plane host with `mcp__devops_tool__bds_environment_host_lookup` using
|
|
45
|
+
Airflow/Kubernetes control-plane roles. If the command pack returns hosts,
|
|
46
|
+
confirm them against lookup evidence rather than assuming they are current.
|
|
47
|
+
- Do not invent host tools such as `workday_host_info`, `host_info`,
|
|
48
|
+
`host_query`, or `bds_enterprise_host_lookup`. Do not use local `grep`, `ls`,
|
|
49
|
+
`Read`, `Glob`, repository search, local `/hadoop` paths, or local `kubectl`
|
|
50
|
+
as a host lookup substitute. If host lookup fails, stop and report the exact
|
|
51
|
+
tool/access gap.
|
|
52
|
+
- Use `bauth init` plus the explicit private-cloud host unchanged. Use root
|
|
53
|
+
kubeconfig and saved Keychain sudo only through the dedicated plan/runbook
|
|
54
|
+
paths when read-only evidence requires them.
|
|
55
|
+
- Always query `-n airflow`; default namespace emptiness is not Airflow health
|
|
56
|
+
evidence.
|
|
57
|
+
- Bound log scans by request time, DAG run, pod, tenant, snapshot, or worker.
|
|
58
|
+
State the gap before broader read-only scans.
|
|
59
|
+
- If access, sudo, or evidence is blocked, report the exact evidence gap. Do not
|
|
60
|
+
infer job health from default namespace output or stale Slack state.
|
|
61
|
+
|
|
62
|
+
## Evidence Order
|
|
63
|
+
|
|
64
|
+
1. Parse environment, config JSON path, DAG ID, run ID, namespace, tenant,
|
|
65
|
+
migration identifiers, and incident class.
|
|
66
|
+
2. Refresh relevant local repos with `git pull --ff-only`; report dirty repos.
|
|
67
|
+
3. Read the environment JSON and extract control-plane hosts.
|
|
68
|
+
4. Generate the validated read-only command pack with
|
|
69
|
+
`bds_airflow_job_status_command_pack`; preserve host, Keychain, Boundary, and safety
|
|
70
|
+
guidance.
|
|
71
|
+
5. Resolve and confirm the concrete control-plane host with
|
|
72
|
+
`bds_environment_host_lookup`.
|
|
73
|
+
6. Run `bauth init`, then SSH to the explicit control-plane host and record
|
|
74
|
+
hostname, UTC time, and user.
|
|
75
|
+
7. Run normal-user `kubectl get pod` only as a kubeconfig baseline.
|
|
76
|
+
8. Use stdin sudo/root kubeconfig for `kubectl get pod -n airflow -o wide`.
|
|
77
|
+
9. Read `deploy,statefulset,job,cronjob`, events, and bounded scheduler/web logs
|
|
78
|
+
in the `airflow` namespace.
|
|
79
|
+
10. If DAG/run details are known, query Airflow metadata with read-only
|
|
80
|
+
list/state commands and read exact task logs when needed.
|
|
81
|
+
11. Follow the smallest routed `references/airflow-*-runbook.md` file for
|
|
82
|
+
scenario details.
|
|
83
|
+
12. Validate recovery with fresh live evidence, not alert disappearance alone.
|
|
84
|
+
|
|
85
|
+
## Scenario Routing
|
|
86
|
+
|
|
87
|
+
- Basic component health: pods, events, scheduler/web logs, and health probes.
|
|
88
|
+
- On-demand tenant copy: control-panel ID, task log, source/destination,
|
|
89
|
+
snapshot/path, YARN app, and DistCp progress.
|
|
90
|
+
- Snapshot cleanup or PRISM `CreateDataFrame` timeout: schedule, bounded
|
|
91
|
+
`dag_bulk_snapshot_cleaner` logs, NameNode `deleteSnapshot` write-lock
|
|
92
|
+
evidence, audit RPCs, diff-size gaps, and application timeout.
|
|
93
|
+
- `bas_dr_sla_miss`: affected tenants, stale snapshots, DR task pod state,
|
|
94
|
+
bad-worker runtime evidence, and BAS lag verification.
|
|
95
|
+
- Migration-service Slack incidents: event identifiers, tenant records,
|
|
96
|
+
copy/cutover state, bucket sync, `PRISM_READY`, `COMPLETED`, and Jira follow-up.
|
|
97
|
+
- S3 VPCE copy failures: compare failing and alternate endpoints from the same
|
|
98
|
+
worker/source IP using DNS, route, and bounded curl probes only.
|
|
99
|
+
|
|
100
|
+
## Output
|
|
101
|
+
|
|
102
|
+
Return a requester-ready answer: event ids when relevant, control-plane host,
|
|
103
|
+
kubeconfig baseline, Airflow pod/event/log findings, DAG/run state,
|
|
104
|
+
scenario-specific evidence, recovery status, and evidence gaps.
|