aigent-team 0.1.0

package/templates/teams/devops/references/disaster-recovery.md

@@ -0,0 +1,199 @@
+# Disaster Recovery Reference
+
+## RPO / RTO Tiers
+
+| Tier | RPO (Data Loss) | RTO (Downtime) | Strategy | Example Services |
+|---|---|---|---|---|
+| **Tier 1 — Critical** | < 1 min | < 15 min | Active-active multi-region, synchronous replication | Payment processing, auth |
+| **Tier 2 — High** | < 15 min | < 1 hour | Warm standby, async replication, automated failover | User API, core business logic |
+| **Tier 3 — Standard** | < 4 hours | < 4 hours | Pilot light, daily snapshots, semi-automated recovery | Internal tools, reporting |
+| **Tier 4 — Low** | < 24 hours | < 24 hours | Backup/restore from cold storage | Dev environments, archives |
+
+### Assigning Tiers
+
+- Product and engineering jointly classify each service.
+- Classification reviewed quarterly.
+- Tier drives backup frequency, replication strategy, and testing cadence.
+- Document tier assignment in the service catalog.
+
+---
+
+## Backup Verification
+
+### Backup Schedule
+
+| Data Type | Frequency | Retention | Storage | Encryption |
+|---|---|---|---|---|
+| Database (relational) | Continuous WAL + daily snapshot | 30 days | Cross-region S3/GCS | AES-256 |
+| Database (NoSQL) | Hourly snapshot | 14 days | Cross-region S3/GCS | AES-256 |
+| Object storage | Cross-region replication | Versioned, 90-day lifecycle | Secondary region | SSE-S3/KMS |
+| Configuration/secrets | Git + Vault snapshots daily | 90 days | Separate account | Vault seal |
+| Kubernetes state | etcd snapshot every 6 hours | 7 days | S3 with Object Lock | AES-256 |
+
+### Verification Rules
+
+1. **Automated restore tests.** Weekly: pick a random Tier 1/2 backup, restore
+   to isolated environment, run validation queries.
+2. **Measure actual RPO.** Compare latest backup timestamp to current time.
+   Alert if gap exceeds tier RPO.
+3. **Measure actual RTO.** Time the restore process end-to-end. Alert if it
+   exceeds tier RTO.
+4. **Verify data integrity.** Run checksums, row counts, and application-level
+   consistency checks after restore.
+5. **Test cross-region.** Monthly: restore from a backup in the secondary region.
+
+### Backup Anti-Patterns
+
+- Backups that have never been restored are not backups — they are hopes.
+- Backing up only the database but not the schema migrations.
+- Storing backups in the same account/region as the primary.
+- No monitoring on backup job success/failure.
+- Relying solely on cloud provider point-in-time recovery without testing it.
+
+---
+
+## Failover Testing
+
+### Failover Types
+
+| Type | Scope | Frequency | Duration |
+|---|---|---|---|
+| Component failover | Single service (kill pod, instance) | Weekly (automated) | Minutes |
+| Zone failover | Simulate AZ loss | Monthly | 1-2 hours |
+| Region failover | Full region switchover | Quarterly | 2-4 hours |
+| Dependency failover | External service unavailable | Monthly | 1 hour |
+
+### Failover Procedure Template
+
+```markdown
+## Failover: <Component/Zone/Region>
+
+### Pre-Flight
+- [ ] Notify stakeholders (Slack #incidents)
+- [ ] Confirm monitoring is healthy (baseline)
+- [ ] Confirm rollback procedure is ready
+- [ ] Set maintenance window in PagerDuty
+
+### Execution
+1. <Step-by-step actions to trigger failover>
+2. Observe traffic shift in dashboards
+3. Verify service health in secondary
+
+### Validation
+- [ ] All health checks passing
+- [ ] Error rate within SLO
+- [ ] Latency within SLO
+- [ ] No data loss detected
+- [ ] Customer-facing functionality verified
+
+### Rollback
+1. <Steps to revert to primary>
+
+### Post-Test
+- [ ] Document actual RTO (time from trigger to healthy)
+- [ ] Document any issues encountered
+- [ ] Create tickets for improvements
+- [ ] Update runbooks
+```
+
+---
+
+## Chaos Engineering
+
+### Progressive Approach
+
+```
+Level 1: Kill a pod (weekly, automated)
+Level 2: Inject latency to a dependency (monthly)
+Level 3: Simulate AZ failure (monthly)
+Level 4: Simulate region failure (quarterly)
+Level 5: Simulate total dependency loss (quarterly)
+```
+
+### Chaos Experiments
+
+| Experiment | Tool | What It Tests |
+|---|---|---|
+| Pod kill | Chaos Mesh, Litmus | Auto-restart, PDB, replica count |
+| Network latency | tc, Chaos Mesh | Timeout handling, circuit breakers |
+| Network partition | iptables, Chaos Mesh | Split-brain handling, retries |
+| Disk fill | Stress tools | Alerting, log rotation, eviction |
+| CPU stress | stress-ng | Autoscaling, request throttling |
+| DNS failure | CoreDNS mutation | Caching, fallback behavior |
+| External API failure | Toxiproxy | Circuit breaker, graceful degradation |
+
+### Rules
+
+- Always run chaos experiments during business hours with the team present.
+- Start in staging. Graduate to production only with confidence.
+- Define abort conditions before starting.
+- Never run chaos without functioning monitoring and alerting.
+- Document findings and create remediation tickets.
+- Chaos engineering is not "break things" — it is "verify resilience hypotheses."
+
+---
+
+## Runbook Requirements
+
+Every Tier 1 and Tier 2 service must have runbooks for:
+
+| Scenario | Content |
+|---|---|
+| Service down | Diagnosis steps, restart procedure, failover steps |
+| Database failover | Promotion steps, connection string update, data verification |
+| Dependency outage | Graceful degradation, circuit breaker verification |
+| Data corruption | Isolation, backup identification, restore procedure |
+| Security incident | Containment, investigation, communication |
+| Region failover | DNS switch, traffic routing, data sync verification |
+
+### Runbook Quality Checklist
+
+- [ ] Written for someone who has never seen this service before.
+- [ ] Includes exact commands, not "restart the service."
+- [ ] Includes expected output for each command.
+- [ ] Links to dashboards, logs, and relevant architecture docs.
+- [ ] Tested by someone other than the author.
+- [ ] Updated within the last 90 days.
+- [ ] Includes estimated time to execute.
+- [ ] Includes escalation contacts with phone numbers.
+
+---
+
+## Recovery Playbooks
+
+### Database Recovery
+
+```
+1. Identify failure (monitoring alert, health check)
+2. Assess: is it a replica or primary failure?
+   ├── Replica: remove from pool, investigate, rebuild from snapshot
+   └── Primary:
+       a. Promote replica to primary (automated or manual)
+       b. Update connection endpoints
+       c. Verify data consistency
+       d. Rebuild old primary as new replica
+3. Post-recovery: verify application health, check replication lag
+4. Postmortem within 48 hours
+```
+
+### Complete Environment Recovery
+
+```
+1. Provision infrastructure (terraform apply — from git)
+2. Restore databases (from latest verified backup)
+3. Deploy applications (from container registry — images already built)
+4. Restore configuration (from git + secrets manager)
+5. Verify: health checks, smoke tests, data integrity
+6. Update DNS / traffic routing
+7. Monitor for 1 hour before declaring recovery complete
+```
+
+### Key Principle
+
+Everything needed to rebuild from scratch must be in version control or
+automated backups. If you lose the primary region, you should be able to
+reconstruct the entire environment from:
+- Git (infrastructure code, application code, configs, dashboards, alerts)
+- Container registry (built images)
+- Backup storage (databases, state files)
+- Secrets manager (credentials)

package/templates/teams/devops/references/docker.md

@@ -0,0 +1,237 @@
+# Docker Reference
+
+## Base Image Selection
+
+| Language | Dev/Build Stage | Production Stage | Target Size |
+|---|---|---|---|
+| Go | `golang:1.22-alpine` | `gcr.io/distroless/static-debian12` | < 20 MB |
+| Node.js | `node:20-alpine` | `node:20-alpine` or `gcr.io/distroless/nodejs20-debian12` | < 150 MB |
+| Python | `python:3.12-slim` | `python:3.12-slim` | < 200 MB |
+| Java | `eclipse-temurin:21-jdk-alpine` | `eclipse-temurin:21-jre-alpine` or distroless | < 250 MB |
+| Rust | `rust:1.77-alpine` | `gcr.io/distroless/cc-debian12` | < 30 MB |
+
+### Rules
+
+- Never use `:latest`. Pin major.minor at minimum.
+- Prefer Alpine or distroless for production. Debian slim if Alpine causes
+  musl compatibility issues.
+- Distroless = no shell, no package manager. Best for compiled languages.
+- Rebuild base images weekly to pick up security patches.
+
+---
+
+## Multi-Stage Build Pattern
+
+```dockerfile
+# ---- Build stage ----
+FROM golang:1.22-alpine AS build
+WORKDIR /src
+
+COPY go.mod go.sum ./
+RUN go mod download
+
+COPY . .
+RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /app ./cmd/server
+
+# ---- Production stage ----
+FROM gcr.io/distroless/static-debian12
+COPY --from=build /app /app
+USER nonroot:nonroot
+EXPOSE 8080
+ENTRYPOINT ["/app"]
+```
+
+### Key Points
+
+- Build dependencies stay in the build stage — never leak into production.
+- Copy dependency manifests first, then source code (layer caching).
+- Strip debug symbols in compiled binaries (`-ldflags="-s -w"` for Go).
+- Use `COPY --from=build` to pull only the final artifact.
+
+---
+
+## Security Hardening
+
+### Non-Root Execution
+
+```dockerfile
+# Create user in build stage
+RUN addgroup -S appgroup && adduser -S appuser -G appgroup
+
+# In production stage
+USER appuser:appgroup
+```
+
+Or with distroless, use the built-in `nonroot` user:
+```dockerfile
+USER nonroot:nonroot
+```
+
+### Read-Only Filesystem
+
+In Kubernetes, set `readOnlyRootFilesystem: true` in the security context.
+If the app needs to write temp files, mount an `emptyDir` at the specific path.
+
+### Drop All Capabilities
+
+```yaml
+securityContext:
+  capabilities:
+    drop: ["ALL"]
+  allowPrivilegeEscalation: false
+  readOnlyRootFilesystem: true
+  runAsNonRoot: true
+```
+
+### No Secrets in Images
+
+- Never `COPY .env` or `COPY credentials`.
+- Never use `ARG` or `ENV` for secrets — they persist in image layers.
+- Pass secrets at runtime via mounted volumes or environment variables from
+  a secrets manager.
+- Use `docker history` to verify no secrets leaked into layers.
+
+---
+
+## .dockerignore
+
+Every project must have a `.dockerignore`:
+
+```
+.git
+.github
+.gitignore
+.env*
+*.md
+docs/
+node_modules/
+__pycache__/
+*.pyc
+.pytest_cache/
+coverage/
+.nyc_output/
+dist/
+build/
+*.log
+docker-compose*.yml
+Dockerfile*
+.dockerignore
+terraform/
+k8s/
+helm/
+.vscode/
+.idea/
+```
+
+### Why It Matters
+
+- Reduces build context size (faster builds).
+- Prevents secrets (`.env`) from entering the image.
+- Avoids cache-busting from irrelevant file changes.
+
+---
+
+## Layer Ordering
+
+Order instructions from least-frequently-changed to most-frequently-changed:
+
+```dockerfile
+# 1. Base image (changes rarely)
+FROM node:20-alpine
+
+# 2. System dependencies (changes rarely)
+RUN apk add --no-cache dumb-init
+
+# 3. Working directory
+WORKDIR /app
+
+# 4. Dependency manifest (changes sometimes)
+COPY package.json package-lock.json ./
+RUN npm ci --production
+
+# 5. Application code (changes often)
+COPY src/ ./src/
+
+# 6. Runtime config
+USER node
+EXPOSE 3000
+CMD ["dumb-init", "node", "src/index.js"]
+```
+
+### Layer Cache Tips
+
+- Separate `COPY` for dependency files vs source code.
+- Use `--mount=type=cache` for package manager caches (BuildKit):
+  ```dockerfile
+  RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
+  ```
+- Combine `RUN` commands that are logically related to reduce layers:
+  ```dockerfile
+  RUN apt-get update && apt-get install -y --no-install-recommends \
+      curl ca-certificates && \
+      rm -rf /var/lib/apt/lists/*
+  ```
+
+---
+
+## Health Checks
+
+```dockerfile
+HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
+  CMD ["/app", "healthcheck"]
+```
+
+Or for HTTP services:
+```dockerfile
+HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
+  CMD wget --no-verbose --tries=1 --spider http://localhost:8080/healthz || exit 1
+```
+
+### Rules
+
+- Always define a `HEALTHCHECK` in the Dockerfile or in K8s probes (not both,
+  K8s probes take precedence).
+- Use a dedicated health endpoint, not the main route.
+- Health check should verify the app can serve traffic, not just that the
+  process is alive.
+
+---
+
+## Image Size Targets
+
+| Category | Target | Action if Exceeded |
+|---|---|---|
+| Static binary (Go/Rust) | < 30 MB | Check for embedded assets, strip symbols |
+| Node.js | < 150 MB | Audit `node_modules`, use `--production` |
+| Python | < 200 MB | Remove build deps, use slim base |
+| Java | < 250 MB | Use JRE not JDK, jlink custom runtime |
+| General | < 500 MB | Investigate — likely build deps leaked |
+
+### Checking Size
+
+```bash
+# Size of final image
+docker images myapp:latest --format "{{.Size}}"
+
+# Layer-by-layer breakdown
+docker history myapp:latest
+
+# Deep analysis
+dive myapp:latest
+```
+
+---
+
+## Build Best Practices
+
+- Enable BuildKit: `DOCKER_BUILDKIT=1`
+- Tag with git SHA for traceability: `myapp:abc1234`
+- Also tag with semver if releasing: `myapp:1.2.3`
+- Scan images before push:
+  ```bash
+  trivy image myapp:abc1234
+  ```
+- Push to a private registry (ECR, GCR, ACR, Harbor). Never deploy from
+  Docker Hub in production.
+- Set `imagePullPolicy: IfNotPresent` in K8s when using immutable SHA tags.
+- Use `.dockerignore` in every project — no exceptions.

package/templates/teams/devops/references/infrastructure-as-code.md

@@ -0,0 +1,238 @@
+# Infrastructure as Code Reference
+
+## Terraform Directory Structure
+
+```
+infra/
+├── modules/                    # Reusable modules (internal registry)
+│   ├── vpc/
+│   │   ├── main.tf
+│   │   ├── variables.tf
+│   │   ├── outputs.tf
+│   │   └── README.md
+│   ├── eks-cluster/
+│   ├── rds/
+│   └── s3-bucket/
+├── environments/
+│   ├── dev/
+│   │   ├── main.tf            # Module calls with dev params
+│   │   ├── variables.tf
+│   │   ├── outputs.tf
+│   │   ├── terraform.tfvars   # Dev-specific values
+│   │   ├── backend.tf         # Remote state config for dev
+│   │   └── providers.tf
+│   ├── staging/
+│   └── production/
+├── global/                     # Shared resources (IAM, DNS zones)
+│   ├── iam/
+│   ├── dns/
+│   └── ecr/
+└── scripts/
+    ├── plan.sh
+    ├── apply.sh
+    └── import.sh
+```
+
+### Rules
+
+- One state file per environment per component. Never share state across envs.
+- `global/` resources are applied once and referenced via `terraform_remote_state`
+  or SSM parameters.
+- Modules live in a separate repo or `modules/` directory with semantic versions.
+
+---
+
+## State Management
+
+### Remote Backend (AWS Example)
+
+```hcl
+terraform {
+  backend "s3" {
+    bucket         = "company-terraform-state"
+    key            = "environments/production/network/terraform.tfstate"
+    region         = "us-east-1"
+    dynamodb_table = "terraform-locks"
+    encrypt        = true
+  }
+}
+```
+
+### State Rules
+
+1. **Always remote.** Never commit `.tfstate` to git.
+2. **Always locked.** DynamoDB (AWS), GCS (GCP), or Terraform Cloud.
+3. **Always encrypted.** Enable server-side encryption on the bucket.
+4. **State per component.** Split large configs: `network`, `compute`, `data`,
+   `monitoring`. Cross-reference with `terraform_remote_state`.
+5. **Never hand-edit state.** Use `terraform state mv`, `terraform import`,
+   `terraform state rm`.
+
+### State Key Convention
+
+```
+environments/{env}/{component}/terraform.tfstate
+```
+
+Examples: `environments/production/network/terraform.tfstate`,
+`environments/staging/eks/terraform.tfstate`.
+
+---
+
+## Plan-Review-Apply Cycle
+
+```
+Developer pushes IaC change
+  → CI runs `terraform fmt -check`
+  → CI runs `terraform validate`
+  → CI runs `tflint` / `checkov` / `tfsec`
+  → CI runs `terraform plan` and posts output to PR
+  → Reviewer approves plan output (not just code)
+  → Merge triggers `terraform apply` with saved plan file
+  → Apply output posted to PR / Slack
+```
+
+### Plan Safety
+
+- Always save plan to file: `terraform plan -out=tfplan`
+- Apply the exact plan: `terraform apply tfplan`
+- Never run `terraform apply` without a saved plan in CI.
+- Require plan output review for any change touching production.
+
+### Drift Detection
+
+- Schedule weekly `terraform plan` in CI (no apply).
+- Alert if drift detected (non-empty plan on unchanged code).
+- Investigate and reconcile — do not auto-apply drift corrections.
+
+---
+
+## Module Versioning
+
+### Semantic Versioning
+
+```hcl
+module "vpc" {
+  source  = "git::https://github.com/company/tf-modules.git//vpc?ref=v2.1.0"
+}
+```
+
+- **Major** (v2→v3): Breaking changes — variable removed, resource recreated.
+- **Minor** (v2.1→v2.2): New feature — new optional variable, new output.
+- **Patch** (v2.1.0→v2.1.1): Bug fix — no interface change.
+
+### Rules
+
+- Pin module versions in all environments. Never use `ref=main`.
+- Upgrade dev first, validate, then promote to staging, then production.
+- Module changes require a CHANGELOG entry.
+- Modules must have `variables.tf` with descriptions and types for every input.
+- Modules must have `outputs.tf` exposing values consumers need.
+
+---
+
+## Import Before Recreate
+
+When Terraform wants to destroy and recreate a critical resource:
+
+1. **Stop.** Review the plan carefully.
+2. Check if `lifecycle { prevent_destroy = true }` should be set.
+3. If migrating existing infrastructure into Terraform:
+   ```bash
+   terraform import aws_db_instance.main my-database-id
+   ```
+4. After import, run `plan` — adjust config until plan shows no changes.
+5. For state moves (renaming resources):
+   ```bash
+   terraform state mv aws_s3_bucket.old aws_s3_bucket.new
+   ```
+6. Use `moved` blocks (Terraform 1.1+) for refactoring:
+   ```hcl
+   moved {
+     from = aws_s3_bucket.old
+     to   = aws_s3_bucket.new
+   }
+   ```
+
+---
+
+## Tagging Policy
+
+Every cloud resource must carry these tags:
+
+| Tag | Example | Purpose |
+|---|---|---|
+| `Environment` | `production` | Cost allocation, access control |
+| `Team` | `platform` | Ownership |
+| `Service` | `user-api` | Dependency mapping |
+| `ManagedBy` | `terraform` | Drift tracking |
+| `CostCenter` | `eng-platform` | Finance reporting |
+| `Repository` | `github.com/co/infra` | Code traceability |
+
+### Enforcement
+
+```hcl
+# In provider or module
+default_tags {
+  tags = {
+    Environment = var.environment
+    Team        = var.team
+    ManagedBy   = "terraform"
+    Repository  = var.repository
+  }
+}
+```
+
+- Use AWS Config rules, GCP Organization Policies, or OPA/Sentinel to reject
+  untagged resources.
+- CI lint step checks every resource block for required tags.
+
+---
+
+## Resource Naming Convention
+
+```
+{company}-{environment}-{region}-{service}-{resource_type}[-{qualifier}]
+```
+
+Examples:
+- `acme-prod-use1-userapi-rds-primary`
+- `acme-stg-euw1-platform-eks`
+- `acme-dev-use1-shared-vpc`
+
+### Rules
+
+- Lowercase, hyphens only (no underscores — some cloud providers reject them).
+- Max 63 characters (DNS label limit).
+- Region abbreviated: `use1` = us-east-1, `euw1` = eu-west-1.
+- Use variables and `locals` to construct names — never hardcode.
+
+```hcl
+locals {
+  name_prefix = "${var.company}-${var.environment}-${var.region_short}-${var.service}"
+}
+
+resource "aws_s3_bucket" "assets" {
+  bucket = "${local.name_prefix}-assets"
+}
+```
+
+---
+
+## Terraform Style Guide
+
+- `terraform fmt` on every save. Enforce in CI.
+- One resource type per file for large configs, or group by logical domain.
+- Variables: always include `description`, `type`, and `default` (if optional).
+- Outputs: always include `description`.
+- Use `count` for simple toggles, `for_each` for collections.
+- Avoid `depends_on` unless absolutely necessary — implicit dependencies preferred.
+- Keep provider versions pinned with `~>` (pessimistic constraint):
+  ```hcl
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 5.0"
+    }
+  }
+  ```