sdtk-ops-kit 0.2.0

package/assets/toolkit/toolkit/skills/ops-debug/SKILL.md

@@ -0,0 +1,311 @@
+<!-- Based on superpowers by Jesse Vincent (MIT License, 2025). Adapted for SDTK-OPS. -->
+---
+name: ops-debug
+description: Systematic infrastructure debugging. Use when encountering deployment failure, service outage, network issue, or unexpected infrastructure behavior -- binary search to root cause, evidence-first, no guessing fixes.
+---
+
+# Ops Debug
+
+## Overview
+
+Random fixes waste time and create new bugs. Quick patches mask underlying issues.
+
+**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.
+
+**Violating the letter of this process is violating the spirit of debugging.**
+
+## The Iron Law
+
+```
+NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
+```
+
+If you have not completed Phase 1, you cannot propose fixes.
+
+## When to Use
+
+Use for ANY infrastructure or operations issue:
+- Deployment failures
+- Service outages
+- Unexpected behavior
+- Performance problems
+- Build or release failures
+- Integration issues
+- Network or DNS problems
+
+**Use this ESPECIALLY when:**
+- Under time pressure (emergencies make guessing tempting)
+- "Just one quick fix" seems obvious
+- You have already tried multiple fixes
+- Previous fix did not work
+- You do not fully understand the issue
+
+**Do not skip when:**
+- Issue seems simple (simple bugs have root causes too)
+- You are in a hurry (rushing guarantees rework)
+- Someone wants it fixed NOW (systematic is faster than thrashing)
+
+## The Four Phases
+
+You MUST complete each phase before proceeding to the next.
+
+### Phase 1: Root Cause Investigation
+
+**BEFORE attempting ANY fix:**
+
+1. **Read Error Messages Carefully**
+   - Do not skip past errors or warnings
+   - They often contain the exact clue you need
+   - Read stack traces completely
+   - Note line numbers, file paths, error codes, probe failures, and status codes
+
+2. **Reproduce Consistently**
+   - Can you trigger it reliably?
+   - What are the exact steps?
+   - Does it happen every time?
+   - If not reproducible -> gather more data, do not guess
+
+3. **Check Recent Changes**
+   - What changed that could cause this?
+   - Git diff, recent commits, manifest changes
+   - New dependencies, config changes, secret rotations
+   - Environmental differences between healthy and broken systems
+
+4. **Gather Evidence in Multi-Component Systems**
+
+   **WHEN system has multiple components (load balancer -> service -> database, CI -> artifact -> deploy target):**
+
+   **BEFORE proposing fixes, add diagnostic instrumentation:**
+   ```
+   For EACH component boundary:
+     - Log what data enters component
+     - Log what data exits component
+     - Verify environment and config propagation
+     - Check state at each layer
+
+   Run once to gather evidence showing WHERE it breaks
+   THEN analyze evidence to identify failing component
+   THEN investigate that specific component
+   ```
+
+   **Infrastructure example (LB -> App -> DB -> DNS -> Network):**
+   ```bash
+   # Layer 1: Load balancer or ingress
+   curl -Ik https://api.example.com/health
+
+   # Layer 2: Application workload
+   kubectl get pods -n production
+   kubectl logs deploy/api -n production --tail=100
+
+   # Layer 3: Database connectivity
+   kubectl exec deploy/api -n production -- sh -c 'nc -zv db.internal 5432'
+
+   # Layer 4: DNS
+   dig +short api.example.com
+
+   # Layer 5: Network controls
+   kubectl get networkpolicy -n production
+   ```
+
+   **This reveals:** Whether the break happens at traffic entry, application runtime, database reachability, DNS resolution, or network controls.
+
+5. **Trace Data Flow**
+
+   **WHEN error is deep in the stack or spread across layers:**
+
+   See `./references/root-cause-tracing.md` in this directory for the complete backward tracing technique.
+
+   **Quick version:**
+   - Where does the bad value or bad state originate?
+   - What called this with bad input?
+   - Keep tracing up until you find the source
+   - Fix at source, not at symptom
+
+### Phase 2: Pattern Analysis
+
+**Find the pattern before fixing:**
+
+1. **Find Working Examples**
+   - Locate similar working infrastructure or runbooks in the same codebase
+   - What works that is similar to what is broken?
+
+2. **Compare Against References**
+   - If implementing a pattern, read the reference implementation COMPLETELY
+   - Do not skim
+   - Understand the pattern fully before applying it
+
+3. **Identify Differences**
+   - What is different between working and broken?
+   - List every difference, however small
+   - Do not assume "that cannot matter"
+
+4. **Understand Dependencies**
+   - What other components does this need?
+   - What settings, config, secrets, or environment?
+   - What assumptions does it make?
+
+### Phase 3: Hypothesis and Testing
+
+**Scientific method:**
+
+1. **Form Single Hypothesis**
+   - State clearly: "I think X is the root cause because Y"
+   - Write it down
+   - Be specific, not vague
+
+2. **Test Minimally**
+   - Make the SMALLEST possible change to test the hypothesis
+   - One variable at a time
+   - Do not fix multiple things at once
+
+3. **Verify Before Continuing**
+   - Did it work? Yes -> Phase 4
+   - Did it fail? Form a NEW hypothesis
+   - DO NOT add more fixes on top
+
+4. **When You Do Not Know**
+   - Say "I do not understand X"
+   - Do not pretend to know
+   - Ask for help
+   - Research more
+
+### Phase 4: Implementation
+
+**Fix the root cause, not the symptom:**
+
+1. **Create a Repeatable Failing Check**
+   - Simplest possible reproduction
+   - Automated test if possible
+   - Diagnostic command or small script if no framework exists
+   - MUST have before fixing
+
+2. **Implement Single Fix**
+   - Address the root cause identified
+   - ONE change at a time
+   - No "while I'm here" improvements
+   - No bundled refactoring
+
+3. **Verify Fix**
+   - Check passes now?
+   - No other health checks broken?
+   - Issue actually resolved?
+
+4. **If Fix Does Not Work**
+   - STOP
+   - Count: How many fixes have you tried?
+   - If < 3: Return to Phase 1 and re-analyze with new information
+   - **If >= 3: STOP and question the architecture (step 5 below)**
+   - DO NOT attempt Fix #4 without architectural discussion
+
+5. **If 3+ Fixes Failed: Question Architecture**
+
+   **Pattern indicating architectural problem:**
+   - Each fix reveals new shared state, coupling, or drift in a different place
+   - Fixes require broad refactoring to implement
+   - Each fix creates new symptoms elsewhere
+
+   **STOP and question fundamentals:**
+   - Is this pattern fundamentally sound?
+   - Are we "sticking with it through sheer inertia"?
+   - Should we refactor architecture vs. continue fixing symptoms?
+
+   **Discuss with the user before attempting more fixes**
+
+   This is NOT a failed hypothesis. This is a wrong architecture.
+
+## Infrastructure Debugging Patterns
+
+| Symptom | Investigation Path |
+|---------|--------------------|
+| 5xx errors spike | deployment timing, pod health, resource limits, DB connections |
+| High latency | DNS resolution, network hops, DB query time, container CPU throttling |
+| Connection refused | security groups, network policies, service mesh, port bindings |
+| Intermittent failures | DNS TTL, load balancer health checks, resource saturation, race conditions |
+| Container restart loop | OOM kills, liveness probe config, startup dependencies |
+| Certificate errors | expiry dates, chain completeness, SANs, renewal automation |
+
+## Red Flags - STOP and Follow Process
+
+If you catch yourself thinking:
+- "Quick fix for now, investigate later"
+- "Just try changing X and see if it works"
+- "Add multiple changes, run checks"
+- "Skip the failing check, I will verify manually"
+- "It is probably X, let me fix that"
+- "I do not fully understand but this might work"
+- "Pattern says X but I will adapt it differently"
+- "Here are the main problems:" followed by fixes without investigation
+- Proposing solutions before tracing data flow
+- **"One more fix attempt" (when already tried 2+)**
+- **Each fix reveals new problem in different place**
+
+**ALL of these mean: STOP. Return to Phase 1.**
+
+**If 3+ fixes failed:** Question the architecture (see Phase 4.5)
+
+## Signals You Are Doing It Wrong
+
+**Watch for these redirections:**
+- "Is that not happening?" - You assumed without verifying
+- "Will it show us...?" - You should have added evidence gathering
+- "Stop guessing" - You are proposing fixes without understanding
+- "Think harder about the system" - Question fundamentals, not just symptoms
+- "We are stuck?" - Your approach is not working
+
+**When you see these:** STOP. Return to Phase 1.
+
+## Common Rationalizations
+
+| Excuse | Reality |
+|--------|---------|
+| "Issue is simple, do not need process" | Simple issues have root causes too. Process is fast for simple bugs. |
+| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |
+| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |
+| "I will write the check after confirming the fix works" | Unverified fixes do not stick. Prove the failure first. |
+| "Multiple fixes at once saves time" | You cannot isolate what worked. It creates new bugs. |
+| "Reference is too long, I will adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |
+| "I see the problem, let me fix it" | Seeing symptoms is not the same as understanding root cause. |
+| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, do not fix again. |
+
+## Common Mistakes
+
+| Mistake | Why it fails |
+|---------|--------------|
+| Skip reproduction and jump to fixes | You never prove what actually changed |
+| Bundle multiple changes into one test | You lose causality and create new uncertainty |
+| Stop tracing at the first visible failure | The real trigger stays in place |
+| Keep patching after repeated failed fixes | The architecture problem remains unchallenged |
+
+## Quick Reference
+
+| Phase | Key Activities | Success Criteria |
+|-------|---------------|------------------|
+| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |
+| **2. Pattern** | Find working examples, compare | Identify differences |
+| **3. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |
+| **4. Implementation** | Create failing check, fix, verify | Issue resolved, checks pass |
+
+## When Process Reveals "No Root Cause"
+
+If systematic investigation reveals the issue is truly environmental, timing-dependent, or external:
+
+1. You completed the process
+2. Document what you investigated
+3. Implement appropriate handling (retry, timeout, circuit breaker, error message)
+4. Add monitoring and logging for future investigation
+
+**But:** most "no root cause" cases are incomplete investigation.
+
+## Supporting Techniques
+
+- **`./references/root-cause-tracing.md`** - Trace failures backward through layers to find the original trigger
+
+**Related skills:**
+- **`ops-verify`** - Verify the fix worked before claiming success
+
+## Real-World Impact
+
+From debugging sessions:
+- Systematic investigation minimizes thrash
+- First-time fix rate improves when evidence replaces guessing
+- New bugs introduced by speculative patches drop sharply

package/assets/toolkit/toolkit/skills/ops-debug/references/root-cause-tracing.md

@@ -0,0 +1,138 @@
+<!-- Based on superpowers by Jesse Vincent (MIT License, 2025). Adapted for SDTK-OPS. -->
+
+# Root Cause Tracing
+
+## Overview
+
+Failures often appear deep in an operational stack: a load balancer returns 502, a pod restarts, or a deployment step times out. The instinct is to fix where the error appears, but that only treats the symptom.
+
+**Core principle:** Trace backward through the call chain until you find the original trigger, then fix at the source.
+
+## When to Use
+
+Use this technique when:
+- the error appears deep in execution rather than at the entry point
+- multiple layers are involved
+- it is unclear where invalid data or state originated
+- you need to find which system, pipeline step, or configuration change triggered the failure
+
+## The Tracing Process
+
+### 1. Observe the Symptom
+
+```
+Error: https://api.example.com returns 502 through the load balancer
+```
+
+### 2. Find the Immediate Cause
+
+Ask: what directly causes this symptom?
+
+```
+Load balancer target health checks are failing
+```
+
+### 3. Ask: What Called This?
+
+Trace one level back at a time:
+
+```
+Load balancer target marked unhealthy
+-> readiness probe failing on application pods
+-> application cannot open a database connection
+-> connection string secret missing in deployment environment
+```
+
+### 4. Keep Tracing Up
+
+Ask what introduced the bad value or broken state:
+
+- Was the secret renamed?
+- Was the manifest updated only in one environment?
+- Did a pipeline step skip secret injection?
+- Did an operator apply the wrong values file?
+
+### 5. Find the Original Trigger
+
+Example original trigger:
+
+```yaml
+env:
+  - name: DATABASE_URL
+    valueFrom:
+      secretKeyRef:
+        name: api-secrets-prod
+        key: database_url
+```
+
+But the deployment applied to staging uses `api-secrets-staging`, and the values file still points to the prod secret name. The readiness failure is a symptom. The wrong values reference is the source.
+
+## Adding Stack Traces And Instrumentation
+
+When you cannot trace manually, add instrumentation at the boundary you suspect:
+
+```bash
+echo "=== ingress ==="
+curl -Ik https://api.example.com/health
+
+echo "=== workload ==="
+kubectl get pods -n production
+kubectl logs deploy/api -n production --tail=100
+
+echo "=== runtime env ==="
+kubectl exec deploy/api -n production -- env | grep DATABASE_URL || true
+
+echo "=== dependency reachability ==="
+kubectl exec deploy/api -n production -- sh -c 'nc -zv db.internal 5432'
+```
+
+Analyze the output in order:
+- traffic entry
+- workload health
+- environment propagation
+- downstream dependency reachability
+
+## Real Example: Missing Secret Reference
+
+**Symptom:** public endpoint returns 502
+
+**Trace chain:**
+1. ingress reports targets unhealthy
+2. readiness probe on the app fails
+3. app logs show database connection failure
+4. `DATABASE_URL` is unset inside the running pod
+5. Helm values file points to the wrong secret name
+
+**Root cause:** wrong configuration at deployment source
+
+**Fix:** correct the values file and redeploy
+
+**Also add defense in depth:**
+- validate required secrets before rollout
+- fail the pipeline if mandatory env vars are absent
+- alert on repeated readiness failures
+- log explicit startup errors for missing config
+
+## Key Principle
+
+```
+Found immediate cause
+-> Can trace one level up?
+-> Keep tracing
+-> Is this the source?
+-> Fix at source
+-> Add validation at each layer
+```
+
+**NEVER fix only where the error appears.** Trace back to the original trigger.
+
+## Tracing Tips
+
+- Log before the dangerous operation, not after it fails
+- Capture enough context to compare healthy vs broken paths
+- Include environment, target, region, namespace, revision, or release identifiers
+- Save the exact command output you used so `ops-verify` can confirm the fix later
+
+## Real-World Impact
+
+Root-cause tracing turns "the load balancer is broken" into a precise failure chain. That reduces guesswork, shortens incidents, and prevents the same class of outage from recurring.

package/assets/toolkit/toolkit/skills/ops-deploy/SKILL.md

@@ -0,0 +1,102 @@
+<!-- Based on agency-agents by AgentLand Contributors (MIT License, 2025) and gstack by Garry Tan (MIT License, 2026). Adapted for SDTK-OPS. -->
+---
+name: ops-deploy
+description: Deployment execution. Use when deploying infrastructure changes or application releases to any environment -- covers deployment strategies, health check gates, traffic management, and rollback procedures.
+---
+
+# Ops Deploy
+
+## Overview
+
+Deployment is an operational change, not a hopeful push. Every rollout needs a pre-flight baseline, a traffic strategy, explicit health gates, and a rollback procedure that is ready before the first change lands.
+
+## The Iron Law
+
+```
+NO DEPLOYMENT WITHOUT HEALTH CHECK GATES AND ROLLBACK PROCEDURE
+```
+
+## When to Use
+
+Use for:
+- application releases
+- infrastructure rollouts that can affect service health
+- database-linked releases
+- traffic routing changes
+- staged promotions between environments
+
+## Pre-Flight Checks
+
+Before starting deployment, confirm:
+- target revision or artifact is fixed and identifiable
+- target environment is healthy before the change
+- rollback access, previous revision, and credentials are available
+- deployment window avoids known peak traffic or freeze periods
+- required backup or snapshot is complete if stateful systems are involved
+
+## Deployment Strategies
+
+| Strategy | When To Use | Rollback Method | Time To Roll Back |
+|----------|-------------|-----------------|-------------------|
+| Rolling | low-risk stateless changes with healthy autoscaling | roll back deployment revision or image tag | minutes |
+| Blue-Green | high-confidence traffic cutover with fast reversal | switch traffic back to blue | seconds to minutes |
+| Canary | risky changes that need live traffic sampling | stop promotion and route all traffic to stable version | seconds to minutes |
+
+## Deployment Execution Flow
+
+Execute in this order:
+1. pre-flight baseline
+2. deploy to target slice or environment
+3. health check gate
+4. smoke test for critical user path
+5. stability window of 15-30 minutes
+6. declare complete or roll back
+
+Do not compress these stages into one command sequence with no pause for evidence.
+
+## <HARD-GATE>
+
+Do not advance to the next stage unless all are true:
+- health endpoint returns 200 or equivalent healthy state
+- error rate is not materially above baseline
+- p95 latency is not materially above baseline
+- no crash loops, failed probes, or repeated restarts are present
+
+If any gate fails, stop promotion and execute rollback.
+
+## Database Migration Safety
+
+For releases that touch data shape:
+- ship backward-compatible schema first
+- deploy code that can run against both old and new shape
+- migrate data in a controlled step
+- remove old schema only after rollback is no longer needed
+
+Never combine irreversible schema change with first traffic cutover unless a specific rollback path exists and has been reviewed.
+
+## Rollback Procedures
+
+| Strategy | Rollback Trigger | Rollback Action | Verification |
+|----------|------------------|-----------------|--------------|
+| Rolling | error spike or unhealthy pods | roll back workload revision | old revision healthy and stable |
+| Blue-Green | failed smoke test after cutover | repoint traffic to blue | blue health and traffic recovery confirmed |
+| Canary | canary metrics outside threshold | stop promotion and restore stable routing | stable slice absorbs full traffic cleanly |
+
+## Common Mistakes
+
+| Mistake | Why it fails |
+|---------|--------------|
+| No health checks between stages | Bad release reaches more users before anyone notices |
+| Skip staging because production is "simple" | Hidden config drift appears at the worst time |
+| Deploy during peak traffic | Blast radius increases and rollback gets noisier |
+| Combine application deploy and risky schema change | Root cause and rollback path become unclear |
+| Declare success immediately after deploy command returns | Runtime failure often appears during warmup and live traffic |
+
+## Completion Discipline
+
+A deployment is complete only after:
+- the stability window passes
+- rollback remains possible or has been intentionally closed
+- the exact evidence is recorded
+- `ops-verify` confirms the post-deploy state
+

package/assets/toolkit/toolkit/skills/ops-discover/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: ops-discover
+description: Discover SDTK-OPS skills. Use when starting a conversation or unsure which skill to use -- lists all operations skills with trigger conditions and priority order.
+---
+
+# Ops Discover
+
+## Overview
+
+Use this index skill when the request is operational but the correct SDTK-OPS entry point is unclear. Start with the highest-priority foundational skill that matches the situation, then add domain-specific skills only as needed.
+
+## When to Use
+
+Use for:
+- the first turn of an operations task
+- unclear routing between planning, debugging, deployment, monitoring, or security
+- deciding which skill chain should run first
+
+## Priority Order
+
+Choose in this order:
+1. `ops-verify`
+2. `ops-debug`
+3. `ops-plan` or `ops-infra-plan`
+4. the domain-specific skill
+5. `ops-parallel`
+
+Use `ops-parallel` only when the work can safely split after the primary skill choice is clear.
+
+## Core Process
+
+| Skill | Trigger Condition |
+|-------|-------------------|
+| `ops-verify` | you need evidence before claiming an operational task is complete |
+| `ops-debug` | a deployment, service, network, DNS, or runtime issue needs root-cause analysis |
+| `ops-plan` | the team needs a step-by-step operational change plan |
+| `ops-parallel` | two or more independent operations tasks can proceed without shared state |
+
+## Deployment
+
+| Skill | Trigger Condition |
+|-------|-------------------|
+| `ops-infra-plan` | you are designing infrastructure, networking, IAM, or resource topology before provisioning |
+| `ops-deploy` | you are executing a rollout, promotion, cutover, or rollback |
+| `ops-container` | you are building images, Dockerfiles, manifests, or container runtime patterns |
+| `ops-ci-cd` | you are setting up or modifying pipelines, artifact flow, or deployment gates |
+
+## Operations
+
+| Skill | Trigger Condition |
+|-------|-------------------|
+| `ops-monitor` | you need SLOs, alerts, dashboards, logs, traces, or observability coverage |
+| `ops-incident` | a production incident is active or incident procedures need to be established |
+| `ops-backup` | backup, restore, disaster recovery, RTO, or RPO design is in scope |
+| `ops-cost` | you are reviewing spend, right-sizing, or budget controls |
+
+## Security And Compliance
+
+| Skill | Trigger Condition |
+|-------|-------------------|
+| `ops-security-infra` | infrastructure hardening, secrets, network policy, or security scanning is the main concern |
+| `ops-compliance` | audit readiness, evidence collection, retention, or policy enforcement is the main concern |
+
+## Discovery
+
+| Skill | Trigger Condition |
+|-------|-------------------|
+| `ops-discover` | you need help choosing among SDTK-OPS skills or sequencing them |
+
+## Common Workflow Chains
+
+| Situation | Suggested Chain |
+|-----------|-----------------|
+| New service rollout | `ops-infra-plan` -> `ops-container` -> `ops-ci-cd` -> `ops-deploy` -> `ops-monitor` -> `ops-verify` |
+| Production incident | `ops-incident` -> `ops-debug` -> `ops-deploy` -> `ops-verify` |
+| Backup hardening | `ops-backup` -> `ops-infra-plan` -> `ops-monitor` -> `ops-verify` |
+| Cost review | `ops-cost` -> `ops-infra-plan` -> `ops-deploy` -> `ops-verify` |
+| Security hardening | `ops-security-infra` -> `ops-infra-plan` -> `ops-ci-cd` -> `ops-verify` |
+
+## Red Flags
+
+| Shortcut Or Smell | Correct Skill |
+|-------------------|---------------|
+| "Looks healthy enough" | `ops-verify` |
+| "Try one more quick fix" | `ops-debug` |
+| "Provision first and document later" | `ops-infra-plan` |
+| "Just push it to production" | `ops-deploy` |
+| "We can add the pipeline checks later" | `ops-ci-cd` |
+| "No time for alerts right now" | `ops-monitor` |
+| "We will write the post-mortem only if leadership asks" | `ops-incident` |
+| "Backups exist, so restore testing can wait" | `ops-backup` |
+| "We do not know who owns this spend" | `ops-cost` |
+| "Secrets in env files are good enough" | `ops-security-infra` |
+| "We will gather audit evidence manually at the end of the year" | `ops-compliance` |
+
+## Common Mistakes
+
+| Mistake | Why it fails |
+|---------|--------------|
+| Start with a domain skill before choosing the primary control skill | Planning, debugging, or verification gaps get missed |
+| Use `ops-parallel` before clarifying the main workstream | Parallelism amplifies confusion instead of speeding delivery |
+| Treat `ops-discover` as the workflow itself | It is the routing layer, not the execution skill |