siclaw 0.1.1 → 0.1.2

package/skills/core/volcano-gang-scheduling/SKILL.md

@@ -0,0 +1,299 @@
+---
+name: volcano-gang-scheduling
+description: >-
+  Gang Scheduling diagnostic guide for Volcano.
+  Use when PodGroup cannot schedule completely, member Pods remain Pending,
+  or minAvailable/minMember constraints are not satisfied.
+---
+
+# Gang Scheduling Diagnosis
+
+This is a diagnostic guide for Gang scheduling issues in Volcano. Gang scheduling requires that all members of a PodGroup be scheduled simultaneously. If the cluster cannot satisfy the `minMember` requirement, none of the pods will be scheduled.
+
+**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify PodGroups or resource configurations.
+
+## When to Use This Guide
+
+Use this skill when:
+- PodGroup status is `Inqueue` but member Pods remain `Pending`
+- Events contain `minMember` related errors
+- Volcano Job has `minAvailable` or `minMember` that cannot be satisfied
+- Some member Pods are running, others are Pending, and the entire group won't start
+- You see `FailedScheduling` events mentioning Gang constraints
+
+## Understanding Gang Scheduling
+
+Gang scheduling in Volcano ensures that either all members of a workload are scheduled, or none are. This is crucial for distributed workloads like MPI, TensorFlow, PyTorch where partial scheduling is wasteful.
+
+**Key Concepts:**
+- `minMember` (in PodGroup spec): Minimum number of pods that must be scheduled simultaneously
+- `minResources` (in PodGroup spec): Aggregate resource floor (e.g., total GPUs) that must be available — **both** `minMember` and `minResources` must be satisfied if set
+- `minAvailable` (in Job spec): Similar concept at Job level
+- The scheduler checks if there are **simultaneous** resources for all minMember pods before allocating
+
+## Diagnostic Steps
+
+### Step 1: Identify the PodGroup
+
+Find the PodGroup associated with the pending pods:
+
+```bash
+kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.metadata.annotations.scheduling.volcano.sh/pod-group}'
+```
+
+### Step 2: Check PodGroup Status
+
+Get detailed PodGroup information:
+
+```bash
+kubectl get podgroup <podgroup-name> -n <namespace> -o yaml
+```
+
+**Key fields to examine:**
+
+| Field | Meaning | What to Look For |
+|-------|---------|------------------|
+| `spec.minMember` | Minimum pods required | Is this number achievable? |
+| `spec.minResources` | Aggregate resource floor | Is total cluster capacity sufficient? |
+| `status.phase` | Current scheduling phase | Should be `Inqueue` for ready-to-schedule |
+| `status.running` | Currently running pods | Compare to minMember |
+| `status.pending` | Pending pods | These are waiting for Gang constraint |
+| `spec.queue` | Queue name | Check if queue has sufficient resources |
+
+**Common scenarios:**
+
+- `status.phase: Pending` - PodGroup is waiting to be enqueued
+- `status.phase: Inqueue` - Ready for scheduling but constraint not met
+- `status.running < spec.minMember` - Gang constraint not satisfied
+
+### Step 3: Calculate Resource Requirements
+
+Calculate the total resources needed for the Gang:
+
+```
+Total CPU = minMember × single Pod CPU request
+Total Memory = minMember × single Pod Memory request
+Total GPU = minMember × single Pod GPU request (if applicable)
+```
+
+Get a pod's resource requests:
+
+```bash
+kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources.requests}'
+```
+
+### Step 4: Check Cluster Resources
+
+#### Option A: Check Node Resources
+
+View available resources across nodes:
+
+```bash
+kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,GPU:.status.allocatable.nvidia\.com/gpu'
+```
+
+Check current resource usage:
+
+```bash
+kubectl top nodes
+```
+
+#### Option B: Check by Node Labels (if pods have node affinity)
+
+If pods target specific nodes:
+
+```bash
+kubectl get nodes -l <label-key>=<label-value> -o wide
+```
+
+### Step 5: Check Events for Gang Errors
+
+Look for Gang-specific scheduling errors:
+
+```bash
+kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'
+```
+
+**Common Gang-related event messages:**
+
+| Message | Meaning | Investigation |
+|---------|---------|---------------|
+| `minMember not satisfied` | Gang constraint preventing scheduling | Check if total resources >= minMember requirements |
+| `gang member not ready` | Some pods in the gang are not ready | Check individual pod status |
+| `resource insufficient` | Not enough resources for all members | Use `volcano-resource-insufficient` skill |
+
+### Step 6: Verify Queue Resources
+
+If the PodGroup is in a Queue, check if the queue has sufficient deserved resources:
+
+```bash
+kubectl get queue <queue-name>
+kubectl describe queue <queue-name>
+```
+
+Look for:
+- `status.deserved` vs `status.allocated`
+- If allocated >= deserved, the queue is at capacity
+- Check `status.state` is `Open` (not `Closing` or `Closed`)
+
+## Common Causes and Solutions
+
+### Cause 1: minMember Too Large
+
+**Symptom:** `minMember` is larger than the number of available nodes, or requires more resources than any single node can provide.
+
+**Example:**
+- minMember = 10
+- Each pod requests 8 GPUs
+- Only 5 nodes have 8 GPUs each
+- **Result:** Gang can never be satisfied
+
+**Solution:**
+- Reduce `minMember` in PodGroup spec
+- Increase cluster capacity (add nodes)
+- Reduce per-pod resource requests
+
+### Cause 2: Resource Fragmentation
+
+**Symptom:** Total cluster resources are sufficient, but not concentrated on enough nodes to satisfy simultaneous scheduling.
+
+**Example:**
+- minMember = 4, each needs 4 CPUs
+- Total cluster: 20 CPUs available
+- But distributed across 10 nodes with 2 CPUs each
+- **Result:** Cannot find 4 nodes with 4 CPUs simultaneously
+
+**Solution:**
+- Configure `binpack` plugin to concentrate pods on fewer nodes
+- Defragment cluster by rescheduling or draining nodes
+- Adjust resource requests to fit node sizes
+
+### Cause 3: Priority Preemption
+
+**Symptom:** Resources exist but are being used by lower-priority workloads that should be preempted.
+
+**Check:**
+- Compare PodGroup priority vs running PodGroups
+- Check if higher priority exists in the same queue
+
+**Solution:**
+- Ensure correct PriorityClass is assigned
+- Check `priority` plugin is enabled in scheduler config
+
+### Cause 4: Queue Resource Exhaustion
+
+**Symptom:** The PodGroup's queue has used all its deserved resources.
+
+**Check:**
+```bash
+kubectl get queue <queue-name> -o jsonpath='{.status.allocated}'
+kubectl get queue <queue-name> -o jsonpath='{.status.deserved}'
+```
+
+**Solution:**
+- Increase queue weight or capability
+- Wait for other jobs to complete
+- Use `volcano-queue-diagnose` for detailed analysis
+
+### Cause 5: Affinity/Anti-Affinity Conflicts (Effective Node Pool Narrowing)
+
+**Symptom:** Queue shows available capacity, but Gang still blocks. Pod scheduling constraints narrow the effective node pool below what Gang requires.
+
+**Diagnosis — compute the effective node pool:**
+```bash
+# 1. Check pod's nodeSelector
+kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeSelector}'
+
+# 2. Check matching nodes
+kubectl get nodes -l <selector-key>=<selector-value> -o custom-columns="NAME:.metadata.name,CPU:.status.allocatable.cpu,GPU:.status.allocatable['nvidia.com/gpu']"
+
+# 3. Check tolerations (tainted nodes require matching tolerations)
+kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.tolerations}'
+kubectl get nodes -o custom-columns="NAME:.metadata.name,TAINTS:.spec.taints[*].key"
+```
+
+Volcano scheduling is **two-phase**: first queue-level admission (capacity check), then node-level placement. A job can pass the queue check but fail node placement if all matching nodes are occupied.
+
+**Solution:**
+- Relax affinity constraints if possible
+- Ensure sufficient nodes match the constraints
+- Verify toleration matches for tainted nodes
+
+### Cause 6: Queue Has Capacity but Gang Still Blocks
+
+**Symptom:** Queue `allocated < deserved`, PodGroup is `Inqueue`, but pods remain Pending.
+
+**Check — verify remaining capacity vs Gang requirement:**
+```bash
+# Queue remaining capacity
+kubectl get queue <queue> -o jsonpath='{"deserved: "}{.status.deserved}{"\nallocated: "}{.status.allocated}'
+
+# PodGroup minMember and minResources
+kubectl get podgroup <pg> -n <ns> -o jsonpath='{"minMember: "}{.spec.minMember}{"\nminResources: "}{.spec.minResources}'
+```
+
+Calculate: `remaining = deserved - allocated`. If `remaining < minMember × per-pod-resources`, the Gang cannot be satisfied even though the queue is not fully used.
+
+If `minResources` is set, also verify: `remaining >= minResources` for each resource dimension.
+
+**Solution:**
+- Wait for enough resources to free up in the queue
+- Reduce `minMember` or `minResources` if the job can tolerate partial scheduling
+
+### Cause 7: Post-Scheduling Gang Breakage
+
+**Symptom:** Job was Running, then moves to Aborted. Running pod count dropped below `minMember`.
+
+This happens when pods are evicted (preemption, node failure, OOM) and the remaining count falls below the Gang constraint, causing the entire group to be torn down.
+
+**Check:**
+```bash
+# Current running vs required
+kubectl get podgroup <pg> -n <ns> -o jsonpath='{"running: "}{.status.running}{"\nminMember: "}{.spec.minMember}'
+
+# Check for eviction/preemption events
+kubectl get events -n <ns> --field-selector reason=Preempted
+kubectl get events -n <ns> --field-selector reason=Evicted
+```
+
+**Solution:**
+- Investigate why pods were evicted (resource pressure, preemption, node failure)
+- Consider setting `reclaimable: false` on the queue to prevent preemption
+- Increase cluster capacity to reduce eviction pressure
+
+## Verification Steps
+
+After identifying the issue, verify your analysis:
+
+1. **Check if issue is Gang-specific:**
+   - Try scheduling a single pod with same resources
+   - If single pod schedules, it's a Gang constraint issue
+   - If single pod doesn't schedule, it's a resource/affinity issue
+
+2. **Calculate minimum requirements:**
+   - Confirm minMember × per-pod-resources ≤ available resources
+   - Confirm enough nodes can accommodate the pods
+
+3. **Check scheduler logs:**
+   ```bash
+   # Use volcano-scheduler-logs skill
+   bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --keyword gang
+   ```
+
+## Key Insight
+
+Gang Scheduling constraint: **Must have enough resources to schedule minMember Pods simultaneously on different nodes.**
+
+Even if total cluster resources are sufficient, if resources are released gradually over time (as other pods complete), the "simultaneous" requirement may not be met.
+
+**Distinguish between:**
+1. **Total shortage** - Entire cluster lacks resources
+2. **Cannot satisfy simultaneously** - Resources exist but not on enough nodes at the same time
+3. **Queue limit** - Queue deserved resources are exhausted
+
+## See Also
+
+- `volcano-diagnose-pod` - General Pod scheduling diagnosis
+- `volcano-queue-diagnose` - Queue status and resource analysis
+- `volcano-resource-insufficient` - Resource shortage diagnosis
+- `volcano-scheduler-logs` - Scheduler log analysis

package/skills/core/volcano-job-diagnose/SKILL.md

@@ -0,0 +1,319 @@
+---
+name: volcano-job-diagnose
+description: >-
+  Diagnose Volcano Job status and issues.
+  Check Job phases, task statuses, PodGroup associations, and overall job health.
+---
+
+# Volcano Job Diagnosis
+
+Diagnose Volcano Job (batch.volcano.sh/v1beta1) status and issues. This skill checks Job phases, task statuses, PodGroup associations, and overall job health.
+
+**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify job specs or restart jobs — that should be left to the user.
+
+## Usage
+
+```bash
+bash skills/core/volcano-job-diagnose/scripts/diagnose-job.sh --job <job-name> --namespace <namespace>
+```
+
+## Parameters
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `--job JOB` | yes | Job name to diagnose |
+| `--namespace NS` | no | Namespace (default: `default`) |
+| `--verbose` | no | Show detailed task and pod information |
+
+## Examples
+
+Diagnose a Volcano Job:
+```bash
+bash skills/core/volcano-job-diagnose/scripts/diagnose-job.sh --job my-training-job --namespace training
+```
+
+Verbose mode with task details:
+```bash
+bash skills/core/volcano-job-diagnose/scripts/diagnose-job.sh --job my-training-job --namespace training --verbose
+```
+
+## Understanding Volcano Jobs
+
+### Job Structure
+
+```yaml
+apiVersion: batch.volcano.sh/v1beta1
+kind: Job
+spec:
+  schedulerName: volcano
+  tasks:
+    - name: worker
+      replicas: 4
+      template:
+        spec:
+          containers:
+            - name: worker
+              resources:
+                requests:
+                  cpu: "4"
+                  memory: "8Gi"
+  maxRetry: 3              # Max retries before job is Aborted
+  policies:
+    - event: PodFailed
+      action: RestartJob
+```
+
+> **Note:** Volcano Jobs can also be queried using the short name `vcjob` (e.g., `kubectl get vcjob`). This is an alias for `job.batch.volcano.sh`. Be careful not to confuse with native Kubernetes `batch/v1 Job` — always use `job.batch.volcano.sh` or `vcjob` for Volcano Jobs.
+
+### Job Phases
+
+| Phase | Meaning |
+|-------|---------|
+| `Pending` | Job is waiting for resources or admission |
+| `Running` | Job is executing |
+| `Completing` | Job tasks are completing |
+| `Completed` | Job finished successfully |
+| `Failed` | Job failed |
+| `Restarting` | Job is being restarted due to policy |
+| `Terminating` | Job is being terminated |
+| `Aborted` | Job was aborted |
+
+### Task Statuses
+
+Each task within a job has its own status:
+- `Pending` - Task pods not yet scheduled
+- `Running` - Task pods are running
+- `Completed` - Task finished
+- `Failed` - Task failed
+
+## Diagnostic Flow
+
+### Step 1: Job Overview
+
+Get the Job status:
+
+```bash
+kubectl get job.batch.volcano.sh <job-name> -n <namespace> -o yaml
+```
+
+**Key fields to check:**
+- `status.state.phase` - Current job phase
+- `status.failed` - Number of failed tasks
+- `status.succeeded` - Number of succeeded tasks
+- `status.running` - Number of running tasks
+- `status.pending` - Number of pending tasks
+
+### Step 2: Check Tasks
+
+List all tasks and their statuses:
+
+```bash
+kubectl get pods -n <namespace> -l volcano.sh/job-name=<job-name> -o wide
+```
+
+**What to look for:**
+- Pod phases (Pending, Running, Completed, Failed)
+- Pod restart counts
+- Node assignments
+
+### Step 3: Check PodGroup Association
+
+Find the PodGroup created for this Job:
+
+```bash
+kubectl get podgroups -n <namespace> -l volcano.sh/job-name=<job-name>
+```
+
+Or check the Job's tasks for PodGroup annotations:
+
+```bash
+kubectl get pods -n <namespace> -l volcano.sh/job-name=<job-name> \
+  -o jsonpath='{.items[0].metadata.annotations.scheduling\.volcano\.sh/pod-group}'
+```
+
+**Next step:** If PodGroup status is problematic, use `volcano-diagnose-pod` for detailed PodGroup analysis.
+
+### Step 4: Check Policies
+
+Review job policies that may affect behavior:
+
+```bash
+kubectl get job.batch.volcano.sh <job-name> -n <namespace> -o jsonpath='{.spec.policies}'
+```
+
+**Common policies:**
+- `PodFailed` → `RestartJob` - Restart entire job on any pod failure
+- `PodFailed` → `RestartTask` - Restart only the failed task
+- `PodEvicted` → `RestartTask` - Restart evicted tasks
+- `PodEvicted` → `AbortJob` - Abort entire job when a pod is evicted (can cause unexpected aborts during preemption)
+- `TaskCompleted` → `CompleteJob` - Complete job when task finishes
+
+Also check `maxRetry` — when retries are exhausted the job moves to `Aborted`:
+```bash
+kubectl get job.batch.volcano.sh <job-name> -n <namespace> -o jsonpath='{.spec.maxRetry}'
+```
+
+### Step 5: Events Analysis
+
+Check job-related events:
+
+```bash
+kubectl get events -n <namespace> --field-selector involvedObject.name=<job-name>
+```
+
+**Common event patterns:**
+
+#### `JobFailed` - Job has failed
+Check the reason and message for failure details.
+
+#### `JobRestarting` - Job is being restarted
+Check the restart policy and previous failure reason.
+
+#### `TaskFailed` - Individual task failed
+May or may not cause entire job to fail depending on policy.
+
+## Common Issues
+
+### Issue 1: Job Stuck in Pending
+
+**Symptom:** Job phase is `Pending`, no pods created.
+
+**Check:**
+1. PodGroup status: `kubectl get podgroups -n <ns>`
+2. Queue state: `kubectl get queue <queue>`
+3. Events: `kubectl get events -n <ns> | grep <job-name>`
+
+**Likely causes:**
+- Queue is Closed
+- PodGroup cannot be enqueued (resource shortage)
+- Admission webhook rejection
+
+### Issue 2: Some Tasks Running, Others Pending
+
+**Symptom:** Partial task scheduling (e.g., 2/4 tasks running).
+
+**Check:**
+1. PodGroup minMember vs actual pod count
+2. Gang scheduling constraints
+3. Resource availability
+
+**Likely causes:**
+- Gang constraint not satisfied (use `volcano-gang-scheduling`)
+- Resource fragmentation
+- Queue quota exhausted
+
+### Issue 3: Job Restarting Repeatedly
+
+**Symptom:** Job keeps restarting, never completes.
+
+**Check:**
+1. Restart policy: `kubectl get job.batch.volcano.sh -o jsonpath='{.spec.policies}'`
+2. Pod failure reasons: `kubectl describe pod <pod>`
+3. Container logs: `kubectl logs <pod>`
+
+**Likely causes:**
+- Application crashing (check container logs)
+- Resource pressure causing evictions
+- Misconfigured restart policy
+
+### Issue 4: Job Failed After Some Tasks Completed
+
+**Symptom:** Some tasks succeeded, but job marked as Failed.
+
+**Check:**
+1. Failed task details
+2. Job completion policy
+3. Task lifecycle policies
+
+**Likely causes:**
+- One critical task failed
+- Completion policy is strict (all tasks must succeed)
+- Lifecycle policy triggered premature job failure
+
+### Issue 5: Job Aborted Unexpectedly
+
+**Symptom:** Job was Running, then moved to `Aborted`.
+
+**Check:**
+```bash
+# Check maxRetry
+kubectl get job.batch.volcano.sh <job> -n <ns> -o jsonpath='{.spec.maxRetry}'
+
+# Check for preemption/eviction events
+kubectl get events -n <ns> --field-selector reason=Preempted
+kubectl get events -n <ns> --field-selector reason=Evicted
+
+# Check if running pod count dropped below minMember (Gang breakage)
+kubectl get podgroup -n <ns> -l volcano.sh/job-name=<job> -o jsonpath='{"running: "}{.items[0].status.running}{"\nminMember: "}{.items[0].spec.minMember}'
+```
+
+**Likely causes:**
+- `maxRetry` exhausted — job restarted too many times
+- Preemption by higher-priority job — pods evicted, triggering `PodEvicted → AbortJob` policy
+- Gang breakage — pod eviction caused running count to drop below `minMember`, tearing down the entire group
+- Lifecycle policy mismatch — e.g., `PodEvicted → AbortJob` when `RestartTask` would be more appropriate
+
+## Task Lifecycle Policies
+
+Volcano controls task coordination through lifecycle policies, not explicit task dependencies.
+
+```yaml
+spec:
+  tasks:
+    - name: master
+      replicas: 1
+      policies:
+        - event: TaskCompleted
+          action: CompleteJob
+    - name: worker
+      replicas: 4
+      policies:
+        - event: PodFailed
+          action: RestartTask
+```
+
+**Diagnosis:**
+```bash
+# Check per-task status counts
+kubectl get job.batch.volcano.sh <job> -o jsonpath='{.status.taskStatusCount}'
+
+# Check configured policies
+kubectl get job.batch.volcano.sh <job> -o jsonpath='{.spec.tasks[*].policies}'
+```
+
+Look for mismatched events/actions that could cause unexpected restarts or premature completion.
+
+## Integration with Other Skills
+
+Use this skill in combination with others:
+
+```bash
+# 1. Job-level diagnosis
+bash skills/core/volcano-job-diagnose/scripts/diagnose-job.sh --job my-job --namespace training
+
+# 2. If PodGroup issues found → Pod-level diagnosis
+bash skills/core/volcano-diagnose-pod/scripts/diagnose-pod.sh --pod my-job-worker-0 --namespace training
+
+# 3. If Gang issues → Gang scheduling analysis
+# (refer to volcano-gang-scheduling skill)
+
+# 4. If Queue issues → Queue diagnosis
+bash skills/core/volcano-queue-diagnose/scripts/diagnose-queue.sh --queue training-queue
+
+# 5. Check scheduler logs for decisions
+bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --pod my-job-worker-0 --since 1h
+```
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `VOLCANO_NAMESPACE` | `default` | Default namespace for job lookup |
+
+## See Also
+
+- `volcano-diagnose-pod` - Pod-level scheduling diagnosis
+- `volcano-gang-scheduling` - Gang scheduling constraint analysis
+- `volcano-queue-diagnose` - Queue resource analysis
+- `volcano-scheduler-logs` - Scheduler decision logs
+- `deployment-rollout-debug` - (Similar concept for Deployments)