@techwavedev/agi-agent-kit 1.1.7 → 1.2.1

package/templates/skills/ec/karpenter/references/troubleshooting.md

@@ -0,0 +1,359 @@
+# Karpenter Troubleshooting Reference
+
+Detailed troubleshooting guide for common Karpenter issues.
+
+---
+
+## Controller Issues
+
+### Karpenter Not Running
+
+**Symptoms:** No pods in karpenter namespace
+
+```bash
+kubectl get pods -n karpenter
+# No resources found
+```
+
+**Diagnosis:**
+
+```bash
+# Check Helm release
+helm list -n karpenter
+
+# Check events
+kubectl get events -n karpenter --sort-by='.lastTimestamp'
+```
+
+**Solutions:**
+
+1. Verify Helm installation completed
+2. Check IRSA configuration (ServiceAccount annotation)
+3. Verify node exists for controller to run on
+
+### Controller CrashLoopBackOff
+
+**Symptoms:**
+
+```bash
+kubectl get pods -n karpenter
+# NAME                         READY   STATUS             RESTARTS
+# karpenter-xyz                0/1     CrashLoopBackOff   5
+```
+
+**Diagnosis:**
+
+```bash
+kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -c controller --previous
+```
+
+**Common Causes:**
+
+| Error                                            | Cause                   | Fix                               |
+| ------------------------------------------------ | ----------------------- | --------------------------------- |
+| `WebIdentityErr: failed to retrieve credentials` | IRSA not configured     | Verify OIDC provider and IAM role |
+| `sts.amazonaws.com: i/o timeout`                 | DNS not resolving       | Set `dnsPolicy: Default` in Helm  |
+| `unauthorized`                                   | IAM permissions missing | Check IAM role policies           |
+
+### Enable Debug Logging
+
+```bash
+# Via Helm
+helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
+  --namespace karpenter \
+  --reuse-values \
+  --set logLevel=debug
+
+# Via env var
+kubectl set env deployment/karpenter -n karpenter LOG_LEVEL=debug
+
+# View debug logs
+kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -c controller -f
+```
+
+---
+
+## Provisioning Issues
+
+### Pods Stuck Pending
+
+**Symptoms:** Pods remain in `Pending` state despite unschedulability
+
+**Diagnosis:**
+
+```bash
+# Check pod events
+kubectl describe pod <pod-name> -n <namespace>
+
+# Look for Karpenter logs
+kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -c controller | grep <pod-name>
+```
+
+**Common Causes:**
+
+1. **No matching NodePool:**
+
+   ```bash
+   # Verify NodePools exist
+   kubectl get nodepools
+
+   # Check requirements match pod spec
+   kubectl get nodepool <name> -o yaml
+   ```
+
+2. **NodePool limits exceeded:**
+
+   ```bash
+   kubectl describe nodepool <name>
+   # Look for: Status.Resources showing at or near limits
+   ```
+
+3. **No valid instance types:**
+   - Requirements too restrictive
+   - Instance types unavailable in region/AZ
+   - Check: `kubectl logs -n karpenter ... | grep "no instance type"`
+
+### Node Launched But Not Ready
+
+**Symptoms:** Node appears in cluster but shows `NotReady`
+
+**Diagnosis:**
+
+```bash
+kubectl describe node <node-name>
+# Check Conditions section
+
+kubectl get events --field-selector involvedObject.name=<node-name>
+```
+
+**Common Causes:**
+
+| Issue                 | Diagnosis            | Solution                               |
+| --------------------- | -------------------- | -------------------------------------- |
+| CNI not running       | Check aws-node pods  | Verify VPC CNI DaemonSet               |
+| kubelet failing       | Check kubelet logs   | SSH to node, check `/var/log/messages` |
+| ENI attachment failed | EC2 console          | Check subnet capacity, security groups |
+| IP exhaustion         | Check pod-eni errors | Expand CIDR or use prefix delegation   |
+
+### Insufficient Capacity
+
+**Log Message:**
+
+```
+could not schedule pod, incompatible with nodepool "<name>", no instance type satisfied resources
+```
+
+**Solutions:**
+
+1. Expand allowed instance types in NodePool
+2. Add more instance categories/generations
+3. Check pod resource requests aren't too large
+4. Verify instances available in all target AZs
+
+---
+
+## Disruption Issues
+
+### Nodes Not Consolidating
+
+**Symptoms:** Underutilized nodes persist
+
+**Diagnosis:**
+
+```bash
+# Check consolidation policy
+kubectl get nodepool <name> -o jsonpath='{.spec.disruption}'
+
+# Check node conditions
+kubectl get nodeclaim <name> -o jsonpath='{.status.conditions}'
+```
+
+**Common Blockers:**
+
+1. **PodDisruptionBudgets:**
+
+   ```bash
+   kubectl get pdb --all-namespaces
+   # PDBs blocking eviction prevent consolidation
+   ```
+
+2. **Pod annotations:**
+
+   ```yaml
+   # Pods with this annotation block node deletion
+   karpenter.sh/do-not-disrupt: "true"
+   ```
+
+3. **NodePool annotation:**
+
+   ```yaml
+   # NodePool-level disruption blocking
+   karpenter.sh/do-not-disrupt: "true"
+   ```
+
+4. **System pods:**
+   - kube-system pods without PDBs
+   - Check DaemonSets tolerating Karpenter nodes
+
+### Drift Not Detected
+
+**Symptoms:** AMI updated but nodes not replaced
+
+**Diagnosis:**
+
+```bash
+# Check drift conditions
+kubectl get nodeclaim -o custom-columns=NAME:.metadata.name,DRIFTED:.status.conditions[?(@.type=="Drifted")].status
+
+# Verify EC2NodeClass AMI
+kubectl get ec2nodeclass <name> -o jsonpath='{.status.amis}'
+```
+
+**Force Drift:**
+
+```bash
+kubectl annotate ec2nodeclass <name> karpenter.k8s.aws/forced-drift=$(date +%s) --overwrite
+```
+
+### Nodes Expiring Unexpectedly
+
+**Symptoms:** Nodes terminated before expected lifetime
+
+**Diagnosis:**
+
+```bash
+kubectl get nodeclaim <name> -o jsonpath='{.status.conditions[?(@.type=="Expired")]}'
+```
+
+**Configuration:**
+
+```yaml
+spec:
+  disruption:
+    expireAfter: 720h # 30 days
+```
+
+---
+
+## Installation Issues
+
+### Missing Service Linked Role
+
+**Error:**
+
+```
+ServiceLinkedRoleCreationNotPermitted: The provided credentials do not have permission to create the service-linked role for EC2 Spot Instances
+```
+
+**Solution:**
+
+```bash
+aws iam create-service-linked-role --aws-service-name spot.amazonaws.com
+```
+
+### STS Timeout
+
+**Error:**
+
+```
+Post "https://sts.eu-west-1.amazonaws.com/": dial tcp: lookup sts.eu-west-1.amazonaws.com: i/o timeout
+```
+
+**Solutions:**
+
+1. Set `dnsPolicy: Default` in Karpenter deployment
+2. Ensure CoreDNS is running before Karpenter
+3. Use Fargate profile or MNG for system pods
+
+### Role Name Too Long
+
+**Error:** IAM role name exceeds 64 characters
+
+**Solution:** Use shorter role name:
+
+```bash
+# Instead of
+KarpenterNodeRole-my-very-long-cluster-name-with-many-characters
+
+# Use
+KarpenterNodeRole-prod-cluster
+```
+
+---
+
+## Spot Instance Issues
+
+### Spot Interruption Handling
+
+Karpenter automatically:
+
+1. Receives interruption notice (2 min warning)
+2. Cordons the node
+3. Drains pods gracefully
+4. Terminates the instance
+
+**Verify SQS queue configured:**
+
+```bash
+kubectl get deployment -n karpenter karpenter -o yaml | grep -A5 interruptionQueue
+```
+
+### Spot Capacity Unavailable
+
+**Log Message:**
+
+```
+no offering matched the requirements, none of the instance types appear to be available
+```
+
+**Solutions:**
+
+1. Add more instance types to requirements
+2. Enable multiple availability zones
+3. Mix Spot with On-Demand via capacity-type weights
+
+---
+
+## Useful Commands
+
+### View All Karpenter Resources
+
+```bash
+# All Karpenter CRDs
+kubectl api-resources | grep karpenter
+
+# NodePools with status
+kubectl get nodepools -o wide
+
+# EC2NodeClasses with status
+kubectl get ec2nodeclasses -o wide
+
+# NodeClaims (current nodes)
+kubectl get nodeclaims -o wide
+
+# Karpenter-managed nodes
+kubectl get nodes -l karpenter.sh/nodepool -o wide
+```
+
+### Export Current Configuration
+
+```bash
+# Export all NodePools
+kubectl get nodepools -o yaml > nodepools-backup.yaml
+
+# Export all EC2NodeClasses
+kubectl get ec2nodeclasses -o yaml > ec2nodeclasses-backup.yaml
+```
+
+### Monitor Karpenter Events
+
+```bash
+# Watch Karpenter controller logs
+kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -c controller -f
+
+# Watch node creation/deletion
+kubectl get nodes -l karpenter.sh/nodepool -w
+
+# Watch NodeClaims
+kubectl get nodeclaims -w
+```

package/templates/skills/ec/karpenter/scripts/generate_ec2nodeclass.py

@@ -0,0 +1,187 @@
+#!/usr/bin/env python3
+"""
+Script: generate_ec2nodeclass.py
+Purpose: Generate Karpenter EC2NodeClass YAML configuration
+
+Usage:
+    python generate_ec2nodeclass.py --name <name> --cluster <cluster-name> [options]
+
+Arguments:
+    --name              EC2NodeClass name (required)
+    --cluster           EKS cluster name (required, used for discovery tags)
+    --role              IAM role name (default: KarpenterNodeRole-<cluster>)
+    --ami-family        AMI family: AL2, AL2023, Bottlerocket, Ubuntu, Windows2019, Windows2022
+    --ami-selector      AMI selector type: alias, tag, id (default: alias)
+    --volume-size       Root volume size in Gi (default: 100)
+    --volume-type       Volume type: gp2, gp3, io1, io2 (default: gp3)
+    --encrypted         Encrypt volumes (default: true)
+    --imdsv2            Require IMDSv2 (default: true)
+    --user-data         Path to user data script file
+    --tags              Comma-separated tags: key=value
+    --output            Output file (default: stdout)
+
+Exit Codes:
+    0 - Success
+    1 - Invalid arguments
+"""
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+
+def generate_ec2nodeclass(args: argparse.Namespace) -> dict:
+    """Generate EC2NodeClass configuration."""
+    # Determine role name
+    role = args.role or f"KarpenterNodeRole-{args.cluster}"
+
+    # Build AMI selector
+    ami_selector_terms = []
+    if args.ami_selector == "alias":
+        ami_family_aliases = {
+            "AL2": "al2@latest",
+            "AL2023": "al2023@latest",
+            "Bottlerocket": "bottlerocket@latest"
+        }
+        alias = ami_family_aliases.get(args.ami_family, "al2023@latest")
+        ami_selector_terms.append({"alias": alias})
+    elif args.ami_selector == "tag":
+        ami_selector_terms.append({
+            "tags": {
+                "karpenter.sh/discovery": args.cluster
+            }
+        })
+
+    # Build subnet and security group selectors
+    subnet_selector_terms = [{
+        "tags": {
+            "karpenter.sh/discovery": args.cluster
+        }
+    }]
+
+    security_group_selector_terms = [{
+        "tags": {
+            "karpenter.sh/discovery": args.cluster
+        }
+    }]
+
+    # Build block device mappings
+    device_name = "/dev/xvda"
+    if args.ami_family in ["Windows2019", "Windows2022"]:
+        device_name = "/dev/sda1"
+
+    block_device_mappings = [{
+        "deviceName": device_name,
+        "ebs": {
+            "volumeSize": f"{args.volume_size}Gi",
+            "volumeType": args.volume_type,
+            "encrypted": args.encrypted,
+            "deleteOnTermination": True
+        }
+    }]
+
+    # Add IOPS and throughput for gp3
+    if args.volume_type == "gp3":
+        block_device_mappings[0]["ebs"]["iops"] = 3000
+        block_device_mappings[0]["ebs"]["throughput"] = 125
+
+    # Build metadata options
+    metadata_options = {
+        "httpEndpoint": "enabled",
+        "httpProtocolIPv6": "disabled",
+        "httpPutResponseHopLimit": 1,
+        "httpTokens": "required" if args.imdsv2 else "optional"
+    }
+
+    # Build tags
+    tags = {
+        "Environment": "production",
+        "ManagedBy": "karpenter",
+        f"kubernetes.io/cluster/{args.cluster}": "owned"
+    }
+
+    if args.tags:
+        for tag in args.tags.split(","):
+            tag = tag.strip()
+            if "=" in tag:
+                key, value = tag.split("=", 1)
+                tags[key] = value
+
+    # Assemble EC2NodeClass
+    ec2nodeclass: dict[str, Any] = {
+        "apiVersion": "karpenter.k8s.aws/v1",
+        "kind": "EC2NodeClass",
+        "metadata": {
+            "name": args.name
+        },
+        "spec": {
+            "role": role,
+            "amiSelectorTerms": ami_selector_terms,
+            "subnetSelectorTerms": subnet_selector_terms,
+            "securityGroupSelectorTerms": security_group_selector_terms,
+            "blockDeviceMappings": block_device_mappings,
+            "metadataOptions": metadata_options,
+            "tags": tags
+        }
+    }
+
+    # Add AMI family if not using alias
+    if args.ami_selector != "alias" and args.ami_family:
+        ec2nodeclass["spec"]["amiFamily"] = args.ami_family
+
+    # Add user data if specified
+    if args.user_data:
+        user_data_path = Path(args.user_data)
+        if user_data_path.exists():
+            ec2nodeclass["spec"]["userData"] = user_data_path.read_text()
+
+    return ec2nodeclass
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Generate Karpenter EC2NodeClass YAML")
+    parser.add_argument("--name", required=True, help="EC2NodeClass name")
+    parser.add_argument("--cluster", required=True, help="EKS cluster name")
+    parser.add_argument("--role", help="IAM role name")
+    parser.add_argument("--ami-family", default="AL2023",
+                        choices=["AL2", "AL2023", "Bottlerocket", "Ubuntu", "Windows2019", "Windows2022"])
+    parser.add_argument("--ami-selector", default="alias", choices=["alias", "tag", "id"])
+    parser.add_argument("--volume-size", type=int, default=100, help="Volume size in Gi")
+    parser.add_argument("--volume-type", default="gp3", choices=["gp2", "gp3", "io1", "io2"])
+    parser.add_argument("--encrypted", type=bool, default=True, help="Encrypt volumes")
+    parser.add_argument("--imdsv2", type=bool, default=True, help="Require IMDSv2")
+    parser.add_argument("--user-data", help="Path to user data script")
+    parser.add_argument("--tags", help="Comma-separated tags: key=value")
+    parser.add_argument("--output", help="Output file (default: stdout)")
+    parser.add_argument("--format", choices=["yaml", "json"], default="yaml", help="Output format")
+
+    args = parser.parse_args()
+
+    try:
+        ec2nodeclass = generate_ec2nodeclass(args)
+
+        if args.format == "json":
+            output = json.dumps(ec2nodeclass, indent=2)
+        else:
+            output = yaml.dump(ec2nodeclass, default_flow_style=False, sort_keys=False)
+
+        if args.output:
+            with open(args.output, "w") as f:
+                f.write(output)
+            print(f"EC2NodeClass configuration written to: {args.output}")
+        else:
+            print(output)
+
+        sys.exit(0)
+
+    except Exception as e:
+        print(f"Error: {e}", file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()