siclaw 0.1.0 → 0.1.2

package/skills/core/statefulset-debug/SKILL.md

@@ -0,0 +1,280 @@
+---
+name: statefulset-debug
+description: >-
+  Diagnose StatefulSet rollout and scaling failures (ordered update stuck, OnDelete not updating, partition misconfiguration, PVC binding deadlocks).
+  Checks update strategy, pod ordinal progression, PVC bindings, and ordered startup to identify why a StatefulSet is not progressing.
+---
+
+# StatefulSet Rollout & Scaling Failure Diagnosis
+
+When a StatefulSet rollout is stuck, pods are not updating, or scaling is not progressing, follow this flow to identify the root cause.
+
+**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify the StatefulSet, delete pods, or change PVCs — that should be left to the user.
+
+**When to use:** A StatefulSet is not progressing — pods are not updating to the new version, scaling up/down is stuck, or specific ordinal pods are not becoming ready.
+
+**Not for Deployments:** Deployment rollouts have different semantics (parallel, unordered). Use `deployment-rollout-debug` for Deployments.
+
+## Key Concepts
+
+StatefulSets differ fundamentally from Deployments:
+- **Fixed pod identity** — pods have stable names with ordinal suffixes (pod-0, pod-1, ...)
+- **Ordered operations** — updates go in reverse order (N-1 → 0), scaling up goes in forward order (0 → N-1)
+- **Per-pod PVCs** — each pod gets its own PersistentVolumeClaim via `volumeClaimTemplates`
+- **Blocking progression** — in OrderedReady mode (default), if pod at ordinal K is not Ready, all pods with ordinal < K will NOT be updated
+
+## Diagnostic Flow
+
+### 1. Get StatefulSet overview
+
+```bash
+kubectl get statefulset <name> -n <ns> -o wide
+```
+
+Compare the columns:
+- **READY** — pods that are running and ready
+- **REPLICAS** — desired replica count (from `spec.replicas`)
+- **UP-TO-DATE** — pods running the current version (matching `currentRevision` == `updateRevision`)
+
+If `READY < REPLICAS` or there is no `UP-TO-DATE` column showing full count, the rollout or scaling is incomplete.
+
+### 2. Describe the StatefulSet
+
+```bash
+kubectl describe statefulset <name> -n <ns>
+```
+
+Focus on:
+- **Update Strategy** — `RollingUpdate` or `OnDelete`
+- **Partition** — if set, only pods with ordinal ≥ partition are updated
+- **maxUnavailable** — if set (Kubernetes 1.24+), allows multiple pods to be updated simultaneously instead of one-at-a-time
+- **Current Revision / Update Revision** — if different, an update is in progress
+- **Events** — look for errors or warnings
+
+### 3. Check pod status by ordinal
+
+First get the StatefulSet's pod selector to reliably find its pods:
+
+```bash
+kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.selector.matchLabels}'
+```
+
+Then use the returned labels to list pods:
+
+```bash
+kubectl get pods -n <ns> -l <key>=<value> --sort-by='.metadata.name'
+```
+
+Identify which ordinal pod is stuck. In a StatefulSet with OrderedReady policy, **the stuck pod blocks all subsequent operations**.
+
+### 4. Match the failure pattern
+
+---
+
+#### OnDelete strategy — Pods not updating after StatefulSet change
+
+The StatefulSet uses `updateStrategy.type: OnDelete`. In this mode, Kubernetes does **not** automatically update pods — the user must manually delete each pod for it to be recreated with the new spec.
+
+```bash
+kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.updateStrategy}'
+```
+
+If the output shows `{"type":"OnDelete"}` or no `rollingUpdate` field:
+
+Check if the current and update revisions differ:
+
+```bash
+kubectl get statefulset <name> -n <ns> -o jsonpath='current={.status.currentRevision} update={.status.updateRevision}'
+```
+
+If they differ, the StatefulSet spec has been updated but pods are still running the old version. This is **expected behavior** for OnDelete — pods must be manually deleted to pick up the new version.
+
+Check which pods are still on the old revision (use the selector from step 3):
+
+```bash
+kubectl get pods -n <ns> -l <key>=<value> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.controller-revision-hash}{"\n"}{end}'
+```
+
+Pods whose `controller-revision-hash` matches `currentRevision` (not `updateRevision`) are still on the old version.
+
+---
+
+#### RollingUpdate stuck at a specific ordinal — Ordered update blocked
+
+In RollingUpdate mode, StatefulSet updates pods in **reverse ordinal order** (N-1 → N-2 → ... → 0). By default (one-at-a-time), if pod at ordinal K is not Ready, the update stops — pods K-1, K-2, ..., 0 will not be updated.
+
+**Check maxUnavailable** (Kubernetes 1.24+, GA in 1.27):
+
+```bash
+kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'
+```
+
+If `maxUnavailable` is set (e.g., `3`), multiple pods can be updated simultaneously instead of strict one-at-a-time. In this case, seeing 2-3 pods updating at once is normal — not a sign of being stuck. Only investigate if the number of updating pods is below `maxUnavailable` for an extended period, or if specific pods are stuck in a non-Ready state.
+
+Find pods that are not Ready:
+
+```bash
+kubectl get pods -n <ns> -l <key>=<value> --sort-by='.metadata.name'
+```
+
+Check the stuck pod's status:
+- **Pending** → Use `pod-pending-debug`
+- **CrashLoopBackOff / Error** → Use `pod-crash-debug`
+- **ImagePullBackOff** → Use `image-pull-debug`
+- **Running but not Ready** → Check readiness probe (see below)
+
+If the pod is Running but not Ready:
+
+```bash
+kubectl describe pod <stuck-pod> -n <ns>
+```
+
+Look for `Readiness probe failed` events. Common causes:
+- Application not listening on the expected port after config change
+- New version has a bug that prevents health check from passing
+- Readiness probe configuration too aggressive for the new version's startup time
+
+---
+
+#### Partition update — Only some pods updated
+
+The StatefulSet has `spec.updateStrategy.rollingUpdate.partition` set. Only pods with ordinal **≥ partition** are updated; pods with ordinal < partition remain on the old version.
+
+```bash
+kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.updateStrategy.rollingUpdate.partition}'
+```
+
+If this returns a number (e.g., `3`), then pods 0, 1, 2 will NOT be updated. This is often used intentionally for **canary rollouts** — update a subset first, verify, then lower the partition to 0 to roll out fully.
+
+If the user expects all pods to be updated, the partition value needs to be set to `0` or removed.
+
+---
+
+#### Scaling up stuck — Ordered creation blocked
+
+When scaling up, StatefulSet creates pods in **forward ordinal order** (0 → 1 → 2 → ...). Pod at ordinal K+1 is not created until pod K is Running and Ready.
+
+```bash
+kubectl get pods -n <ns> | grep <statefulset-name>
+```
+
+Find the highest ordinal pod that exists — the next ordinal is waiting for this pod to become Ready.
+
+Check why the current highest pod is not Ready (same diagnosis as the "stuck at specific ordinal" pattern above).
+
+For the `podManagementPolicy` field:
+
+```bash
+kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.podManagementPolicy}'
+```
+
+- **OrderedReady** (default) — strict ordered creation, one at a time
+- **Parallel** — all pods are created simultaneously (no ordering guarantee)
+
+If the policy is `Parallel` and pods are still stuck, the issue is not ordering — check individual pod status.
+
+---
+
+#### PVC binding deadlock — Pod stuck in Pending due to volume topology
+
+StatefulSet pods use `volumeClaimTemplates` to create per-pod PVCs. If the PVC is bound to a PV in a specific availability zone (AZ) or node, but that node/AZ has no resources, the pod cannot be scheduled.
+
+Check PVC status for the stuck pod:
+
+```bash
+kubectl get pvc -n <ns> | grep <statefulset-name>
+```
+
+```bash
+kubectl describe pvc <pvc-name> -n <ns>
+```
+
+Check the StorageClass's `volumeBindingMode`:
+
+```bash
+kubectl get storageclass $(kubectl get pvc <pvc-name> -n <ns> -o jsonpath='{.spec.storageClassName}') -o jsonpath='{.volumeBindingMode}'
+```
+
+- **Immediate** — PVC is bound to a PV as soon as created, regardless of pod scheduling. If the PV is in a different zone than the only available nodes, the pod cannot be scheduled.
+- **WaitForFirstConsumer** — PVC binding is delayed until the pod is scheduled. If no node can satisfy both the pod's scheduling constraints and the storage topology, the PVC stays `Pending` and the pod stays `Pending` — a deadlock.
+
+Check if the PV has a node affinity constraint:
+
+```bash
+kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity}'
+```
+
+If the PV is locked to a specific node/zone:
+- Check if that node has available resources: `kubectl describe node <node>`
+- Check if that node is healthy: `kubectl get node <node>`
+
+**Common scenario:** A node was replaced or drained, but the PV is still bound to the old node's zone. The new pod can only be scheduled to nodes that can access this PV, but those nodes may be full or tainted.
+
+For further PVC diagnosis, use the `pvc-debug` skill.
+
+---
+
+#### Scaling down — PVCs left behind
+
+When a StatefulSet is scaled down, pods are deleted in **reverse ordinal order** (N-1 → N-2 → ...). However, Kubernetes does **not** automatically delete the associated PVCs.
+
+```bash
+kubectl get pvc -n <ns> | grep <statefulset-name>
+```
+
+If there are PVCs for ordinals that no longer exist (e.g., `data-myapp-3` when replicas is 2), these are orphaned PVCs from a previous scale-down.
+
+This is by design to prevent data loss. But when scaling back up, the new pod will reattach to the old PVC with stale data, which may cause application issues.
+
+Check the StatefulSet's `persistentVolumeClaimRetentionPolicy` (Kubernetes 1.27+):
+
+```bash
+kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}'
+```
+
+- **whenDeleted: Retain** (default) — PVCs are kept when StatefulSet is deleted
+- **whenScaled: Retain** (default) — PVCs are kept when scaling down
+- **whenScaled: Delete** — PVCs are automatically deleted on scale-down
+
+---
+
+#### Pod stuck in Terminating during update or scale-down
+
+During an update or scale-down, if a pod is stuck in `Terminating`, the next operation cannot proceed.
+
+```bash
+kubectl describe pod <terminating-pod> -n <ns>
+```
+
+First check if a PodDisruptionBudget (PDB) is preventing the deletion:
+
+```bash
+kubectl get pdb -n <ns>
+```
+
+```bash
+kubectl describe pdb <pdb-name> -n <ns>
+```
+
+If the PDB's `minAvailable` or `maxUnavailable` limit has been reached, the StatefulSet controller cannot delete the pod. Check `status.disruptionsAllowed` — if it is `0`, no more pods can be disrupted until other pods become Ready.
+
+If PDB is not the issue, check other common causes:
+- **Finalizer blocking deletion** — check `metadata.finalizers`
+- **PreStop hook hanging** — a long-running preStop hook delays termination
+- **Process not responding to SIGTERM** — the container process ignores shutdown signals and must wait for `terminationGracePeriodSeconds` to expire
+- **Volume unmount stuck** — the volume cannot be detached from the node
+
+Check the grace period:
+
+```bash
+kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.terminationGracePeriodSeconds}'
+```
+
+## Notes
+
+- StatefulSet updates go in **reverse** ordinal order (N-1 → 0), but scaling up goes in **forward** order (0 → N-1). This is a common source of confusion.
+- `OnDelete` is frequently used in database StatefulSets (MySQL, PostgreSQL, etc.) where the operator wants manual control over when each replica is restarted. If a user complains that pods are not updating, check the strategy before assuming there is a bug.
+- The `partition` field is for canary rollouts. A common workflow: set partition=N-1 to update only the last pod, verify, then set partition=0 to roll out to all pods. If a user sees partial updates, check partition before investigating further.
+- PVCs created by `volumeClaimTemplates` follow the naming convention `<volumeClaimTemplate-name>-<statefulset-name>-<ordinal>`. Use this pattern to find PVCs for specific ordinals.
+- Unlike Deployments, StatefulSets do NOT create new ReplicaSets for updates. They update pods in-place (delete old pod, create new pod with same name and PVC).
+- For cross-reference: if the stuck pod's issue is at the scheduling level, use `pod-pending-debug`. If it is crashing, use `pod-crash-debug`. If PVCs are not binding, use `pvc-debug`.

package/skills/core/volcano-diagnose-pod/SKILL.md

@@ -0,0 +1,196 @@
+---
+name: volcano-diagnose-pod
+description: >-
+  Diagnose Volcano-managed Pod scheduling issues.
+  Checks Pod status, PodGroup, events, and Queue to identify scheduling failures.
+---
+
+# Volcano Pod Diagnosis
+
+Diagnose Volcano-managed Pod scheduling issues. This skill checks Pod status, associated PodGroup, scheduling events, and Queue configuration to identify why a Pod cannot be scheduled.
+
+**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify pod specs, PodGroups, or Queues — that should be left to the user.
+
+## Usage
+
+```bash
+bash skills/core/volcano-diagnose-pod/scripts/diagnose-pod.sh --pod <pod-name> --namespace <namespace>
+```
+
+## Parameters
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `--pod POD` | yes | Pod name to diagnose |
+| `--namespace NS` | no | Namespace (default: `default`) |
+| `--verbose` | no | Show detailed output including node resources |
+
+## Examples
+
+Diagnose a pending pod in default namespace:
+```bash
+bash skills/core/volcano-diagnose-pod/scripts/diagnose-pod.sh --pod my-job-0
+```
+
+Diagnose a pod in specific namespace:
+```bash
+bash skills/core/volcano-diagnose-pod/scripts/diagnose-pod.sh --pod my-job-0 --namespace training
+```
+
+Verbose mode with node resource information:
+```bash
+bash skills/core/volcano-diagnose-pod/scripts/diagnose-pod.sh --pod my-job-0 --namespace training --verbose
+```
+
+## Diagnostic Flow
+
+The script performs the following checks in order:
+
+### 1. Pod Status
+Check the Pod's current phase and conditions.
+
+```bash
+kubectl get pod <pod> -n <ns> -o wide
+kubectl describe pod <pod> -n <ns>
+```
+
+### 2. PodGroup Status
+Check if the Pod is associated with a PodGroup and its scheduling status.
+
+```bash
+kubectl get pod <pod> -n <ns> -o jsonpath='{.metadata.annotations.scheduling.volcano.sh/pod-group}'
+kubectl get podgroup <podgroup> -n <ns>
+```
+
+Key fields to check:
+- `spec.minMember`: Minimum members required for Gang scheduling
+- `status.phase`: Pending, Inqueue, Running, Unknown
+- `status.running`: Number of running pods
+- `status.pending`: Number of pending pods
+
+### 3. Events Analysis
+Check scheduling events for failure reasons.
+
+```bash
+kubectl get events -n <ns> --field-selector involvedObject.name=<pod> --sort-by='.lastTimestamp'
+```
+
+Look for these event patterns:
+
+#### `FailedScheduling` - General scheduling failure
+The scheduler attempted but failed to schedule the pod. Check the message for specific reasons.
+
+**Volcano-specific sub-patterns:**
+
+| Event Message | Meaning | Next Step |
+|---------------|---------|-----------|
+| `0/N nodes are available` + `minMember` | Gang constraint not satisfied | Use `volcano-gang-scheduling` |
+| `exceeded quota` / `queue resource exceeded` | Queue deserved resources exhausted | Use `volcano-queue-diagnose` |
+| `Insufficient cpu/memory` + Gang mention | Resource shortage blocking Gang | Use `volcano-resource-insufficient` |
+| `pod group is not ready` | PodGroup not in Inqueue phase | Check PodGroup status |
+| `task <name> is not ready` | Task dependencies not met | Check dependent tasks |
+
+> **Quick Reference vs Detailed Analysis:** The table above provides a quick lookup for common patterns. The sections below provide detailed analysis, additional context, and more diagnostic commands for each pattern.
+
+#### `Insufficient cpu` / `Insufficient memory` - Resource shortage
+No node has enough allocatable resources. Check:
+- Node resources: `kubectl top nodes`
+- Pod resource requests: `kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].resources.requests}'`
+
+**Volcano context:** If this is a Gang-scheduled pod, even if total cluster resources are sufficient, you need enough resources **simultaneously** on enough nodes. Use `volcano-resource-insufficient` to check fragmentation.
+
+#### `minMember` not satisfied - Gang constraint
+The PodGroup requires `minMember` pods to be scheduled simultaneously, but the cluster cannot satisfy this. Use `volcano-gang-scheduling` skill for detailed diagnosis.
+
+**Key insight:** Even if `kubectl top nodes` shows enough total resources, Gang requires **simultaneous** availability on **different nodes**.
+
+#### `queue resource exceeded` - Queue quota limit
+The Queue associated with this Pod has exceeded its deserved resources. Check Queue status with `volcano-queue-diagnose` skill.
+
+**Volcano-specific terms you might see:**
+- `overused` - Queue has exceeded its fair share
+- `deserved resources` - Calculated from queue weight proportion
+- `allocated resources` - Currently used by jobs in this queue
+
+#### `reclaim` events - Resource reclamation triggered
+If you see events mentioning `reclaim`:
+- Another queue is trying to reclaim resources from your pod's queue
+- Your queue may be `over-allocated` (allocated > deserved)
+- Check queue status: `volcano-queue-diagnose --queue <queue>`
+
+#### `preempt` events - Priority preemption
+Higher priority workload is evicting this pod. Check:
+- Pod priority class: `kubectl get pod <pod> -o jsonpath='{.spec.priorityClassName}'`
+- Preemptor details in scheduler logs: `volcano-scheduler-logs --keyword preempt`
+
+#### `enqueue` related events
+- `PodGroup is enqueued` - PodGroup admitted to queue, ready for scheduling
+- `PodGroup is pending` - Waiting for queue admission (capacity or resource check)
+- `enqueue failed` - Failed admission check (overcommit, queue closed, etc.)
+
+### 4. Queue Status
+Check the Queue configuration and resource allocation.
+
+```bash
+kubectl get podgroup <podgroup> -n <ns> -o jsonpath='{.spec.queue}'
+kubectl get queue <queue>
+kubectl describe queue <queue>
+```
+
+Key fields:
+- `spec.weight`: Queue weight for resource sharing
+- `spec.capability`: Maximum resources the queue can use
+- `status.state`: Open, Closed, or Closing
+- `status.deserved`: Resources deserved by this queue
+- `status.allocated`: Resources currently allocated
+
+### 5. Node Resources (verbose mode)
+When `--verbose` is specified, also check node allocatable resources.
+
+```bash
+kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory'
+```
+
+## Common Issues and Solutions
+
+### Pod stuck in Pending, no events
+- Check if Volcano scheduler is running: `kubectl get pods -n volcano-system -l app=volcano-scheduler`
+- Check if Volcano controller-manager is running: `kubectl get pods -n volcano-system -l app=volcano-controller-manager`
+  - The controller-manager is responsible for Job lifecycle, PodGroup creation, and queue management — if it's down, jobs won't transition states even if the scheduler is healthy
+- Check scheduler logs: `volcano-scheduler-logs` skill
+
+### PodGroup phase is Pending
+- The PodGroup is waiting for enqueue action to admit it
+- **Verify the queue actually exists** — a typo in queue name causes the PodGroup to stay Pending silently:
+  ```bash
+  kubectl get podgroup <pg> -n <ns> -o jsonpath='{.spec.queue}'
+  kubectl get queue <queue-name>
+  ```
+  If the queue name is empty, the job uses the `default` queue — verify it exists and is Open
+- Check Queue capacity and deserved resources
+- Check if cluster has sufficient resources
+
+### PodGroup phase is Inqueue but Pod is Pending
+- Check if `minMember` constraint is not satisfied
+- Check if there are affinity/anti-affinity conflicts
+- Check if taints prevent scheduling
+
+### Queue status shows insufficient deserved resources
+- The queue may have insufficient weight or capability configured
+- Other queues may be reclaiming resources
+- Use `volcano-queue-diagnose` for detailed analysis
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `VOLCANO_NAMESPACE` | `default` | Default namespace for Pod lookup |
+| `VOLCANO_SCHEDULER_NS` | `volcano-system` | Namespace where volcano scheduler runs |
+
+## See Also
+
+- `volcano-gang-scheduling` - Detailed Gang scheduling diagnosis
+- `volcano-queue-diagnose` - Queue status and quota analysis
+- `volcano-scheduler-logs` - Scheduler log analysis
+- `volcano-resource-insufficient` - Resource shortage diagnosis
+- `quota-debug` - Native Kubernetes ResourceQuota/LimitRange diagnosis (non-Volcano)

package/skills/core/volcano-diagnose-pod/scripts/diagnose-pod.sh

@@ -0,0 +1,175 @@
+#!/bin/bash
+# Diagnose Volcano-managed Pod scheduling issues.
+# This script performs read-only operations using kubectl.
+set -euo pipefail
+
+show_help() {
+  cat <<EOF
+Usage: $0 --pod <pod> [options]
+
+Diagnose Volcano-managed Pod scheduling issues.
+Checks Pod status, PodGroup, events, and Queue configuration.
+
+Options:
+  --pod POD         Pod name to diagnose (required)
+  --namespace NS    Namespace (default: default)
+  --verbose         Show detailed output including node resources
+  -h, --help        Show this help message
+
+Environment:
+  VOLCANO_NAMESPACE     Override default namespace
+  VOLCANO_SCHEDULER_NS  Scheduler namespace (default: volcano-system)
+
+Examples:
+  $0 --pod my-job-0
+  $0 --pod my-job-0 --namespace training
+  $0 --pod my-job-0 --namespace training --verbose
+EOF
+  exit 0
+}
+
+# Parse arguments
+POD=""
+NS="${VOLCANO_NAMESPACE:-default}"
+SCHEDULER_NS="${VOLCANO_SCHEDULER_NS:-volcano-system}"
+VERBOSE=false
+
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    -h|--help) show_help ;;
+    --pod) POD="$2"; shift 2 ;;
+    --namespace) NS="$2"; shift 2 ;;
+    --verbose) VERBOSE=true; shift ;;
+    *) echo "Unknown option: $1. Use --help for usage." >&2; exit 1 ;;
+  esac
+done
+
+[[ -z "$POD" ]] && { echo "Error: --pod is required. Use --help for usage." >&2; exit 1; }
+
+echo "=== Volcano Pod Diagnosis: $NS/$POD ==="
+echo
+
+# 1. Pod Status
+echo "[1/5] Pod Status"
+echo "----------------"
+if ! kubectl get pod "$POD" -n "$NS" -o wide 2>/dev/null; then
+  echo "Error: Pod '$POD' not found in namespace '$NS'" >&2
+  exit 1
+fi
+echo
+
+# Get Pod phase
+POD_PHASE=$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null || echo "Unknown")
+echo "Pod Phase: $POD_PHASE"
+echo
+
+# 2. PodGroup Information
+echo "[2/5] PodGroup Information"
+echo "--------------------------"
+PG=$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.metadata.annotations.scheduling\.volcano\.sh/pod-group}' 2>/dev/null || true)
+
+if [[ -n "$PG" ]]; then
+  echo "PodGroup: $PG"
+  echo
+  if kubectl get podgroup "$PG" -n "$NS" 2>/dev/null; then
+    echo
+    PG_PHASE=$(kubectl get podgroup "$PG" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null || echo "Unknown")
+    PG_MINMEMBER=$(kubectl get podgroup "$PG" -n "$NS" -o jsonpath='{.spec.minMember}' 2>/dev/null || echo "0")
+    PG_RUNNING=$(kubectl get podgroup "$PG" -n "$NS" -o jsonpath='{.status.running}' 2>/dev/null || echo "0")
+    PG_PENDING=$(kubectl get podgroup "$PG" -n "$NS" -o jsonpath='{.status.pending}' 2>/dev/null || echo "0")
+    PG_QUEUE=$(kubectl get podgroup "$PG" -n "$NS" -o jsonpath='{.spec.queue}' 2>/dev/null || echo "default")
+
+    echo "PodGroup Phase: $PG_PHASE"
+    echo "MinMember: $PG_MINMEMBER"
+    echo "Running: $PG_RUNNING"
+    echo "Pending: $PG_PENDING"
+    echo "Queue: $PG_QUEUE"
+  else
+    echo "Warning: PodGroup '$PG' not found"
+  fi
+else
+  echo "⚠️  No PodGroup annotation found — this Pod is NOT managed by Volcano scheduler."
+  echo "   Recommended: Use 'pod-pending-debug' skill for standard kube-scheduler issues."
+  echo ""
+  echo "   Continuing with basic event analysis..."
+fi
+echo
+
+# 3. Events Analysis
+echo "[3/5] Recent Events"
+echo "-------------------"
+kubectl get events -n "$NS" --field-selector "involvedObject.name=$POD" --sort-by='.lastTimestamp' 2>/dev/null | tail -15 || echo "No events found"
+echo
+
+# 4. Queue Status (if PodGroup exists and has a queue)
+if [[ -n "$PG" ]]; then
+  PG_QUEUE=$(kubectl get podgroup "$PG" -n "$NS" -o jsonpath='{.spec.queue}' 2>/dev/null || echo "")
+  if [[ -n "$PG_QUEUE" ]]; then
+    echo "[4/5] Queue Status: $PG_QUEUE"
+    echo "------------------------------"
+    if kubectl get queue "$PG_QUEUE" 2>/dev/null; then
+      echo
+      QUEUE_STATE=$(kubectl get queue "$PG_QUEUE" -o jsonpath='{.status.state}' 2>/dev/null || echo "Unknown")
+      QUEUE_WEIGHT=$(kubectl get queue "$PG_QUEUE" -o jsonpath='{.spec.weight}' 2>/dev/null || echo "N/A")
+      echo "Queue State: $QUEUE_STATE"
+      echo "Queue Weight: $QUEUE_WEIGHT"
+      echo
+      echo "Deserved Resources:"
+      kubectl get queue "$PG_QUEUE" -o jsonpath='{.status.deserved}' 2>/dev/null || echo "  N/A"
+      echo
+      echo "Allocated Resources:"
+      kubectl get queue "$PG_QUEUE" -o jsonpath='{.status.allocated}' 2>/dev/null || echo "  N/A"
+    else
+      echo "Warning: Queue '$PG_QUEUE' not found"
+    fi
+    echo
+  else
+    echo "[4/5] Queue Status"
+    echo "------------------"
+    echo "No queue specified in PodGroup"
+    echo
+  fi
+else
+  echo "[4/5] Queue Status"
+  echo "------------------"
+  echo "Skipping (no PodGroup found)"
+  echo
+fi
+
+# 5. Node Resources (verbose mode)
+if [[ "$VERBOSE" == "true" ]]; then
+  echo "[5/5] Node Resources"
+  echo "--------------------"
+  echo "Node Allocatable Resources:"
+  kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,GPU:.status.allocatable.nvidia\.com/gpu' 2>/dev/null | head -10
+  echo
+
+  echo "Node Resource Usage (if metrics available):"
+  kubectl top nodes 2>/dev/null | head -10 || echo "Metrics not available (requires metrics-server)"
+  echo
+fi
+
+# Summary
+echo "=== Diagnosis Summary ==="
+echo "Pod: $NS/$POD"
+echo "Phase: $POD_PHASE"
+if [[ -n "$PG" ]]; then
+  echo "PodGroup: $PG (Phase: ${PG_PHASE:-Unknown})"
+  if [[ -n "${PG_QUEUE:-}" ]]; then
+    echo "Queue: $PG_QUEUE (State: ${QUEUE_STATE:-Unknown})"
+  fi
+else
+  echo "PodGroup: Not found"
+fi
+
+if [[ "$POD_PHASE" == "Pending" ]]; then
+  echo
+  echo "Recommendations:"
+  echo "1. Check events above for 'FailedScheduling' reasons"
+  echo "2. If PodGroup phase is 'Pending', check Queue capacity"
+  echo "3. If minMember is not satisfied, use volcano-gang-scheduling skill"
+  echo "4. Check scheduler logs with volcano-scheduler-logs skill"
+fi
+
+echo
+echo "=== Diagnosis Complete ==="