aigent-team 0.1.0

package/templates/teams/devops/references/kubernetes.md

@@ -0,0 +1,397 @@
+# Kubernetes Reference
+
+## Namespace Strategy
+
+```
+Namespaces:
+├── kube-system          # Cluster components (do not deploy here)
+├── monitoring           # Prometheus, Grafana, Loki
+├── ingress              # Ingress controllers
+├── cert-manager         # TLS certificate management
+├── external-secrets     # Secrets operator
+├── app-dev              # Application workloads — dev
+├── app-staging          # Application workloads — staging
+└── app-production       # Application workloads — production
+```
+
+### Rules
+
+- One namespace per environment per application domain.
+- Apply `ResourceQuota` and `LimitRange` to every namespace.
+- Apply `NetworkPolicy` default-deny to every namespace.
+- Label namespaces consistently: `team`, `environment`, `managed-by`.
+
+### ResourceQuota Example
+
+```yaml
+apiVersion: v1
+kind: ResourceQuota
+metadata:
+  name: default-quota
+  namespace: app-production
+spec:
+  hard:
+    requests.cpu: "20"
+    requests.memory: 40Gi
+    limits.cpu: "40"
+    limits.memory: 80Gi
+    pods: "100"
+    services: "20"
+    persistentvolumeclaims: "30"
+```
+
+---
+
+## Resource Management
+
+### Requests and Limits
+
+```yaml
+resources:
+  requests:
+    cpu: 250m
+    memory: 256Mi
+  limits:
+    cpu: 1000m
+    memory: 512Mi
+```
+
+### Sizing Guidelines
+
+| Workload Type | CPU Request | CPU Limit | Memory Request | Memory Limit |
+|---|---|---|---|---|
+| API server | 250m | 1000m | 256Mi | 512Mi |
+| Worker/consumer | 500m | 2000m | 512Mi | 1Gi |
+| Batch job | 1000m | 4000m | 1Gi | 4Gi |
+| Sidecar (proxy) | 50m | 200m | 64Mi | 128Mi |
+
+### Rules
+
+- **Always set requests.** The scheduler uses requests for placement.
+- **Always set limits for memory.** OOM-killed pods are better than node
+  exhaustion.
+- **CPU limits are debated.** Set them to prevent runaway, but be aware of
+  throttling. A 4:1 limit-to-request ratio is a reasonable starting point.
+- Use VPA (Vertical Pod Autoscaler) recommendations to right-size after
+  running for 7+ days.
+- Start generous, then tighten based on metrics.
+
+### LimitRange (Namespace Defaults)
+
+```yaml
+apiVersion: v1
+kind: LimitRange
+metadata:
+  name: default-limits
+spec:
+  limits:
+    - default:
+        cpu: 500m
+        memory: 256Mi
+      defaultRequest:
+        cpu: 100m
+        memory: 128Mi
+      type: Container
+```
+
+---
+
+## Probes
+
+### Liveness Probe
+
+Answers: "Is the process hung?" — restarts the container if it fails.
+
+```yaml
+livenessProbe:
+  httpGet:
+    path: /healthz
+    port: 8080
+  initialDelaySeconds: 15
+  periodSeconds: 20
+  timeoutSeconds: 5
+  failureThreshold: 3
+```
+
+### Readiness Probe
+
+Answers: "Can the pod serve traffic?" — removes from Service endpoints if it fails.
+
+```yaml
+readinessProbe:
+  httpGet:
+    path: /readyz
+    port: 8080
+  initialDelaySeconds: 5
+  periodSeconds: 10
+  timeoutSeconds: 3
+  failureThreshold: 3
+```
+
+### Startup Probe
+
+Answers: "Has the app finished starting?" — disables liveness/readiness until it succeeds.
+
+```yaml
+startupProbe:
+  httpGet:
+    path: /healthz
+    port: 8080
+  periodSeconds: 5
+  failureThreshold: 30      # 30 x 5s = 150s max startup time
+```
+
+### Probe Rules
+
+- **Every container** must have readiness and liveness probes.
+- Use startup probe for slow-starting apps (JVM, ML model loading).
+- Liveness should check internal health only (not downstream deps).
+- Readiness should check ability to serve (including critical deps like DB).
+- Never make liveness depend on external services — cascading restarts.
+- Separate endpoints: `/healthz` (liveness), `/readyz` (readiness).
+
+---
+
+## Pod Disruption Budgets
+
+```yaml
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: user-api-pdb
+spec:
+  minAvailable: 1            # OR maxUnavailable: 1
+  selector:
+    matchLabels:
+      app: user-api
+```
+
+### Rules
+
+- Every production Deployment with 2+ replicas must have a PDB.
+- Use `minAvailable` for critical services.
+- Use `maxUnavailable: 1` for batch workers.
+- PDB prevents node drains from killing all pods simultaneously.
+
+---
+
+## Security Context
+
+### Pod Level
+
+```yaml
+securityContext:
+  runAsNonRoot: true
+  runAsUser: 65534             # nobody
+  runAsGroup: 65534
+  fsGroup: 65534
+  seccompProfile:
+    type: RuntimeDefault
+```
+
+### Container Level
+
+```yaml
+securityContext:
+  allowPrivilegeEscalation: false
+  readOnlyRootFilesystem: true
+  capabilities:
+    drop: ["ALL"]
+```
+
+### Rules
+
+- `runAsNonRoot: true` — always. No exceptions in production.
+- `readOnlyRootFilesystem: true` — mount `emptyDir` for temp writes.
+- `allowPrivilegeEscalation: false` — always.
+- Drop ALL capabilities, add back only what is strictly needed.
+- Use `seccompProfile: RuntimeDefault` at minimum.
+
+---
+
+## Network Policies
+
+### Default Deny All
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny-all
+  namespace: app-production
+spec:
+  podSelector: {}
+  policyTypes:
+    - Ingress
+    - Egress
+```
+
+### Allow Specific Traffic
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-user-api
+spec:
+  podSelector:
+    matchLabels:
+      app: user-api
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app: api-gateway
+      ports:
+        - port: 8080
+  egress:
+    - to:
+        - podSelector:
+            matchLabels:
+              app: postgres
+      ports:
+        - port: 5432
+    - to:                      # Allow DNS
+        - namespaceSelector: {}
+      ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+```
+
+### Rules
+
+- Start with default-deny in every namespace.
+- Explicitly allow only required traffic paths.
+- Always allow DNS egress (port 53) or pods cannot resolve services.
+- Document network flows in architecture diagrams.
+
+---
+
+## Secrets Management
+
+### External Secrets Operator (Preferred)
+
+```yaml
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: user-api-secrets
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: user-api-secrets
+  data:
+    - secretKey: database-url
+      remoteRef:
+        key: secret/data/user-api
+        property: database_url
+```
+
+### Rules
+
+- Never store secrets in plain K8s Secrets manifests in git.
+- Use ExternalSecrets Operator, Sealed Secrets, or CSI Secrets Store.
+- Rotate secrets on a schedule (90 days max for credentials).
+- Audit secret access via cloud provider audit logs.
+- Mount secrets as files, not env vars (env vars leak in crash dumps and
+  `kubectl describe`).
+
+---
+
+## Image Pull Policy
+
+| Tag Type | Policy | Rationale |
+|---|---|---|
+| Git SHA (`abc1234`) | `IfNotPresent` | Immutable, no need to re-pull |
+| Semver (`1.2.3`) | `IfNotPresent` | Immutable (if you follow semver) |
+| `:latest` | `Always` | Mutable — but don't use `:latest` |
+| Branch (`main`) | `Always` | Mutable — only for dev |
+
+---
+
+## Deployment Template
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: user-api
+  labels:
+    app: user-api
+    version: v1
+spec:
+  replicas: 3
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 1
+      maxUnavailable: 0
+  selector:
+    matchLabels:
+      app: user-api
+  template:
+    metadata:
+      labels:
+        app: user-api
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "8080"
+        prometheus.io/path: "/metrics"
+    spec:
+      serviceAccountName: user-api
+      securityContext:
+        runAsNonRoot: true
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: user-api
+          image: registry.company.com/user-api:abc1234
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 8080
+          resources:
+            requests:
+              cpu: 250m
+              memory: 256Mi
+            limits:
+              cpu: 1000m
+              memory: 512Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8080
+            initialDelaySeconds: 15
+            periodSeconds: 20
+          readinessProbe:
+            httpGet:
+              path: /readyz
+              port: 8080
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          volumeMounts:
+            - name: tmp
+              mountPath: /tmp
+      volumes:
+        - name: tmp
+          emptyDir: {}
+      topologySpreadConstraints:
+        - maxSkew: 1
+          topologyKey: topology.kubernetes.io/zone
+          whenUnsatisfiable: DoNotSchedule
+          labelSelector:
+            matchLabels:
+              app: user-api
+```

package/templates/teams/devops/references/monitoring.md

@@ -0,0 +1,224 @@
+# Monitoring and Observability Reference
+
+## Three Pillars
+
+### 1. Metrics (Prometheus / Grafana)
+
+Numeric measurements aggregated over time. Best for dashboards and alerting.
+
+- **Counter**: Monotonically increasing (requests_total, errors_total).
+- **Gauge**: Point-in-time value (active_connections, queue_depth).
+- **Histogram**: Distribution of values (request_duration_seconds).
+
+### 2. Logs (Loki / ELK / CloudWatch)
+
+Timestamped text records of discrete events. Best for debugging specific
+incidents.
+
+- Structured JSON only — no unstructured text logs.
+- Include: timestamp, level, service, trace_id, message, context fields.
+- Exclude: PII, secrets, full request/response bodies.
+
+### 3. Traces (Jaeger / Tempo / X-Ray)
+
+End-to-end request flow across services. Best for understanding latency
+and dependencies.
+
+- Instrument all service boundaries (HTTP, gRPC, message queues).
+- Propagate trace context headers (`traceparent` / W3C Trace Context).
+- Sample at 1-10% in production (100% in dev/staging).
+
+---
+
+## Golden Signals Dashboard
+
+Every service must have a dashboard showing the four golden signals:
+
+| Signal | Metric | Example PromQL |
+|---|---|---|
+| **Latency** | Request duration (p50, p90, p99) | `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))` |
+| **Traffic** | Requests per second | `rate(http_requests_total[5m])` |
+| **Errors** | Error rate (%) | `rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100` |
+| **Saturation** | CPU/memory/queue utilization | `container_memory_working_set_bytes / container_spec_memory_limit_bytes * 100` |
+
+### Dashboard Requirements
+
+- Every service has a dedicated Grafana dashboard.
+- Dashboard definition stored in Git (JSON model or Grafonnet).
+- Include: golden signals, resource usage, key business metrics.
+- Time range selectors: last 1h, 6h, 24h, 7d.
+- Variable selectors: environment, namespace, pod.
+
+---
+
+## Alert Severity Levels
+
+| Level | Criteria | Response Time | Notification | Example |
+|---|---|---|---|---|
+| **P1 — Critical** | Customer-facing outage, data loss risk | 5 min | Page on-call, auto-create incident | API error rate > 10%, database down |
+| **P2 — High** | Degraded performance, partial outage | 15 min | Page on-call | p99 latency > 5s, disk > 90% |
+| **P3 — Warning** | Approaching threshold, non-critical issue | 1 hour | Slack channel | Disk > 80%, certificate expiry < 14d |
+| **P4 — Info** | Anomaly, needs investigation during business hours | Next business day | Slack channel | Unusual traffic spike, dependency slow |
+
+### Alert Rules
+
+```yaml
+# Prometheus alert example
+groups:
+  - name: user-api
+    rules:
+      - alert: HighErrorRate
+        expr: |
+          rate(http_requests_total{service="user-api", status=~"5.."}[5m])
+          / rate(http_requests_total{service="user-api"}[5m]) > 0.05
+        for: 5m
+        labels:
+          severity: P1
+          team: platform
+        annotations:
+          summary: "user-api error rate > 5%"
+          runbook: "https://wiki.company.com/runbooks/user-api-errors"
+          dashboard: "https://grafana.company.com/d/user-api"
+```
+
+### Alerting Principles
+
+- **Alert on symptoms, not causes.** Alert on "error rate high," not "pod restarted."
+- **Alert on SLO burn rate.** If you are burning through your error budget
+  faster than expected, page.
+- **Every alert must have a runbook.** No runbook = not a valid alert.
+- **Every alert must be actionable.** If the response is "wait and see,"
+  it should be a dashboard, not an alert.
+- **Deduplicate.** One page per incident, not one per pod.
+- **Test alerts.** Fire test alerts monthly to verify routing.
+
+---
+
+## On-Call Runbook Requirements
+
+Every runbook must contain:
+
+```markdown
+# Runbook: <Alert Name>
+
+## Alert Description
+What this alert means and why it fires.
+
+## Impact
+What is the customer-facing impact?
+
+## Quick Mitigation
+Step-by-step actions to restore service (within 5 minutes):
+1. ...
+2. ...
+3. ...
+
+## Diagnosis
+How to determine root cause:
+- Dashboard link
+- Log query
+- Trace search
+
+## Resolution
+Longer-term fix steps.
+
+## Escalation
+Who to contact if mitigation fails.
+
+## History
+| Date | Cause | Resolution | Duration |
+|------|-------|------------|----------|
+```
+
+### Runbook Rules
+
+- Stored in Git alongside alert definitions.
+- Reviewed and updated after every incident.
+- Must be executable by anyone on the on-call rotation.
+- Include exact commands, not vague instructions.
+- Link to relevant dashboards and log queries.
+
+---
+
+## Structured Logging Standards
+
+### Log Format (JSON)
+
+```json
+{
+  "timestamp": "2025-01-15T10:30:00.123Z",
+  "level": "error",
+  "service": "user-api",
+  "version": "1.2.3",
+  "trace_id": "abc123def456",
+  "span_id": "789ghi",
+  "message": "Failed to fetch user profile",
+  "error": "connection timeout",
+  "user_id": "usr_REDACTED",
+  "duration_ms": 5000,
+  "method": "GET",
+  "path": "/api/v1/users/123",
+  "status_code": 504
+}
+```
+
+### Log Levels
+
+| Level | When to Use | Example |
+|---|---|---|
+| `error` | Operation failed, needs attention | Database query failed, external API error |
+| `warn` | Recoverable issue, may need attention | Retry succeeded, cache miss, slow query |
+| `info` | Significant business events | Request served, user created, job completed |
+| `debug` | Diagnostic detail (disabled in prod) | Query parameters, intermediate state |
+
+### Rules
+
+- JSON format only — no printf-style logs.
+- Include `trace_id` in every log line for correlation.
+- Never log: passwords, tokens, PII, credit card numbers, full request bodies.
+- Redact or hash user identifiers in logs.
+- Log at request boundaries: start, end, error.
+- Use log sampling for high-volume debug logs.
+- Set retention: 7 days hot, 30 days warm, 90 days cold (adjust per compliance).
+
+---
+
+## SLO Framework
+
+### Defining SLOs
+
+| Service | SLI | SLO | Error Budget (30d) |
+|---|---|---|---|
+| User API | Successful requests / total requests | 99.9% | 43.2 min downtime |
+| Payment API | Successful requests / total requests | 99.99% | 4.3 min downtime |
+| Background Jobs | Jobs completed / jobs submitted | 99.5% | 3.6 hours delay |
+| Dashboard | Page load < 3s | 95% | 36 hours of slow loads |
+
+### Error Budget Policy
+
+- **Budget remaining > 50%**: Ship freely, experiment.
+- **Budget remaining 20-50%**: Caution, increase review rigor.
+- **Budget remaining < 20%**: Freeze features, focus on reliability.
+- **Budget exhausted**: All engineering effort on reliability until budget recovers.
+
+---
+
+## Metrics Naming Convention
+
+```
+<namespace>_<subsystem>_<name>_<unit>
+```
+
+Examples:
+- `http_server_request_duration_seconds`
+- `http_server_requests_total`
+- `db_pool_connections_active`
+- `queue_messages_pending`
+
+### Rules
+
+- Use `_total` suffix for counters.
+- Use `_seconds`, `_bytes`, `_ratio` for units.
+- Use snake_case.
+- Prefix with service/subsystem namespace.
+- Keep cardinality low — avoid high-cardinality labels (user_id, request_id).