coding-os 0.3.2__py3-none-any.whl

core/skills/incident-response/SKILL.md

@@ -0,0 +1,225 @@
+---
+name: incident-response
+tier: workflow
+domain: [infra]
+description: Production incident handling — severity classification, runbook execution, communication, postmortem. Use when something is on fire (CHAOTIC quadrant), when designing a runbook before launch, after an incident to drive the postmortem, or to define the SEV scale + on-call rotation. Pairs with observability (alerts → runbook) and thinking_os (CHAOTIC routing) and deployment-cicd (rollback as the most-used incident action).
+last_reviewed: "2026-05-11"
+---
+
+# Incident Response — Stop the Bleeding, Then Diagnose
+
+A practical playbook for handling production incidents and learning from them. Aligned with Google SRE practices and the PagerDuty incident-response framework, refined for 2026 SaaS reality.
+
+## When to Use This Skill
+
+- **In an incident** — pages firing, users complaining, error rate spiked. Load this skill **first**, before diagnosing.
+- **Before launch** — writing the runbook for a feature.
+- **After an incident** — driving the postmortem.
+- Designing the SEV classification.
+- Defining the on-call rotation.
+- Choosing tools (PagerDuty / Opsgenie / FireHydrant / incident.io).
+
+If the user says "the API is down" / "users can't log in" / "we have an outage" — this skill, immediately.
+
+## The Three-Phase Loop
+
+```
+   1. Stabilize        2. Diagnose        3. Resolve
+   (5 min)             (30-90 min)        (variable)
+   stop the bleeding    understand cause   actual fix
+        │                  │                  │
+        └──────────────────┴──────────────────┘
+                 ↓
+            Postmortem (within 5 business days)
+```
+
+The order is fixed. **Diagnosis before stabilization wastes the most time**, because users keep suffering while you read logs. Stabilize first — even if the fix is hacky — then diagnose.
+
+## Phase 1 — Stabilize (the first 5 minutes)
+
+When the page fires:
+
+1. **Acknowledge the page.** Don't drop it on someone else's screen.
+2. **Open the war room.** Slack thread, dedicated channel, Zoom — one place for the team.
+3. **Declare a SEV.** Stops debate over urgency. See SEV scale below.
+4. **Roll back, scale up, or feature-flag off.** In that order of preference.
+   - Rollback the deploy if the timing matches.
+   - Scale up if saturation is the signal.
+   - Flip the feature flag off if a specific feature is broken.
+5. **Communicate.** Status page, internal status, customer support team. "We're investigating an issue affecting X." Don't speculate on cause yet.
+
+**Hard rule:** in the first 5 minutes you take **safe reversible actions only**. Rollback is reversible. `kubectl delete pod` is not.
+
+## SEV Scale — Stop the Argument
+
+| SEV | Definition | Examples | Response |
+|---|---|---|---|
+| **SEV-1** | Critical: full or near-full outage of a major function. Revenue or trust at risk. | Login down, payment processing failed, data corruption | Page on-call + secondary + manager. War room within 5 min. Status page update within 10 min. CEO briefed within 30 min. |
+| **SEV-2** | Major: significant degradation. One important function broken; workaround may exist. | Search returning empty, exports failing, region-specific outage | Page on-call. War room within 15 min. Status page update within 30 min. |
+| **SEV-3** | Minor: limited-scope issue, workaround clear. | Spelling error in production, slow endpoint not on critical path | Ticket + assign. Fix within next business day. |
+| **SEV-4** | Cosmetic / non-urgent | UI alignment bug, log spam | Backlog. Fix when convenient. |
+
+**SEV is set at acknowledgement** and can be re-classified as you learn more. Better to over-classify and downgrade than under-classify and miss the escalation window.
+
+## Phase 2 — Diagnose (the next 30-90 minutes)
+
+Now read logs, traces, metrics — using [observability](../observability/SKILL.md):
+
+1. **Dashboards first.** Service dashboard with golden signals (latency / traffic / errors / saturation).
+2. **Look at the deploy timeline.** If errors started at 14:32 and a deploy went out at 14:31, the deploy is your suspect. (You already rolled it back in Phase 1.)
+3. **Trace a failing request.** `trace_id` from a recent error log → full trace shows which span broke.
+4. **Compare to a known-good window.** "What changed in the last hour" is more informative than "what does the dashboard show now".
+5. **Hypothesis-driven.** Form a hypothesis, look for confirming + disconfirming evidence. "Database connection pool exhausted because of the new endpoint" → look at pool depth metric AND the new endpoint's traffic.
+
+Document everything in the war room thread. Future-you (writing the postmortem) will thank present-you.
+
+## Phase 3 — Resolve
+
+The resolve action is one of:
+
+- **Rollback** — already done in Phase 1 if it was a deploy.
+- **Patch** — small forward fix. Reviewed and deployed via the normal pipeline (don't skip CI even in an incident — you'll cause incident #2).
+- **Scale** — add capacity, increase rate limits, raise quotas. Document the scale change so it's not forgotten.
+- **Feature flag** — disable the offending feature.
+- **Operate around** — sometimes the resolve is "shift traffic to the working region while we fix the broken one".
+
+**Hard rule:** the resolve must include "how do we prevent this from re-firing in the next hour?" Not "how do we prevent it long-term" — that's the postmortem.
+
+## Communication During an Incident
+
+### Internal
+
+- War room channel — running thread, append everything.
+- Engineering channel — periodic summary every 30 min: "Status: investigating. Hypothesis: database pool saturation. ETA: unknown."
+- Customer support — written summary they can paste to customers: "We're aware of intermittent login errors and are working on a fix."
+
+### External (status page)
+
+- Within 10 min of SEV-1: "Investigating reports of login errors."
+- Update every 30 min even if no progress: "Still investigating. We'll update by HH:MM."
+- Resolved post: include start time, end time, scope (% of users), and root cause (one sentence).
+
+### Anti-patterns
+
+- "Probably a database thing" — speculation creates rumors.
+- Going silent for an hour — silence is worse than "still investigating".
+- "Our engineers are working hard" — say what specifically.
+- Apology theater without root-cause communication.
+
+## On-Call Rotation — Sustainable Design
+
+- **Rotation length:** 1 week. Shorter is more disruptive, longer is more burnout.
+- **Pair primary + secondary.** Secondary covers if primary doesn't ack within 15 min.
+- **Follow-the-sun if global.** No one on-call at 03:00 their time more than once per quarter.
+- **Page budget:** target ≤ 2 pages per on-call week. More than that = alert hygiene work this quarter.
+- **Compensation:** time-off-in-lieu for off-hours pages, even if you "didn't really do much".
+- **No-blame culture.** The on-call who clicked the wrong rollback button isn't fired. They're given training and a better rollback button.
+
+## Postmortem — within 5 business days
+
+The postmortem is **not optional**. Every SEV-1 and SEV-2 gets one. Template:
+
+```markdown
+# Postmortem: <title>
+
+**Severity:** SEV-1
+**Date:** 2026-MM-DD
+**Authors:** @owner, @secondary
+**Status:** Draft | Reviewed | Approved
+
+## Summary
+One paragraph. What happened, who was affected, how long, root cause.
+
+## Impact
+- Users affected: ~X% / ~N users
+- Duration: HH:MM to HH:MM UTC (Y minutes)
+- Revenue impact: $X (if known)
+- SLO impact: error budget consumed = Z%
+
+## Timeline (UTC)
+- 14:31 — Deploy of commit abc123 to production
+- 14:33 — First Sentry alerts (payment.captured failures)
+- 14:35 — On-call paged
+- 14:37 — War room opened, SEV-1 declared
+- 14:39 — Rollback initiated
+- 14:42 — Error rate returns to baseline
+- 14:50 — Status page updated to "monitoring"
+- 15:10 — Status page resolved
+
+## Root cause
+Specific. "The new payment.capture endpoint did not handle Stripe's idempotency-key-conflict error. When deploy hit the production traffic mix, ~3% of captures returned a 4xx that the frontend treated as terminal."
+
+## Resolution
+What action ended the incident. (Rollback, in this case.)
+
+## Five whys (or chain-of-causes)
+1. Why did payment fail? Endpoint didn't handle idempotency conflict.
+2. Why didn't it handle that? Test fixtures didn't cover that branch.
+3. Why didn't test fixtures cover it? Stripe sandbox doesn't easily produce that error.
+4. Why didn't we record-replay? Recording tooling wasn't set up for this endpoint.
+5. Why? The pattern existed for other endpoints but wasn't generalized.
+
+## What went well
+- Rollback in 11 minutes from page to resolved.
+- Status page updated on schedule.
+
+## What went poorly
+- Pre-merge testing didn't catch the idempotency case.
+- Initial dashboard didn't show payment-specific error rate, only total 4xx rate.
+
+## Action items (with owners + due dates)
+- [ ] @alice — Add idempotency-conflict test case to payment.capture suite (due 2026-MM-DD)
+- [ ] @bob — Generalize Stripe-error-replay fixture across endpoints (due 2026-MM-DD)
+- [ ] @carol — Add payment-specific error-rate panel to service dashboard (due 2026-MM-DD)
+- [ ] @dave — Add SLO + burn-rate alert for payment.capture (due 2026-MM-DD)
+
+## Lessons learned (one-liners worth remembering)
+- ...
+```
+
+**No-blame rule:** the postmortem names actions and systems, not people-as-failures. "@alice deployed the bad code" is wrong. "The pre-merge test suite did not cover the failing case" is right.
+
+## Runbook — write before launch
+
+Every service should ship with a runbook. One markdown file per service. Sections:
+
+- **What this service does** (one paragraph)
+- **Owner team + on-call rotation pointer**
+- **Service SLO** (what we promise)
+- **Dashboards** (link to golden signals)
+- **Alerts → actions** (this alert means X, do Y)
+- **Common issues + fixes** (the top 5 things that have gone wrong)
+- **Rollback procedure** (one command, tested)
+- **Escalation contacts**
+
+Runbooks rot fast — quarterly refresh in the on-call rotation handoff.
+
+## When the Incident Doesn't End
+
+- **Hour 2:** swap primary on-call out. Fresh eyes catch things the tired ones miss.
+- **Hour 4:** if SEV-1 still active, escalate to engineering leadership for support resources.
+- **Hour 8:** if still active, that's a special incident — formally declare a "major incident" with daily executive comms.
+
+## Anti-patterns
+
+- **Heroes who solve incidents alone** — the team can't learn or recover sustainably.
+- **No runbooks** — every incident is a from-scratch investigation.
+- **Blame in the postmortem** — kills psychological safety, future incidents get hidden.
+- **No action items from postmortems** — same incident in 3 months.
+- **Action items without owners + due dates** — never done.
+- **Skipping the postmortem because "it was small"** — every SEV-1/2 gets one.
+- **Pre-incident testing of the rollback procedure missing** — first time you roll back is in the incident; the rollback fails.
+
+## Tooling
+
+Classify severity deterministically (no 3am debate):
+`python3 scripts/classify_severity.py --core-down --users-affected 70`
+
+## See also
+
+- [references/severity-and-runbook.md](references/severity-and-runbook.md) — SEV scale, roles, lifecycle, blameless postmortem.
+- [assets/incident-checklist.md](assets/incident-checklist.md) — the during-incident gate.
+- [observability](../observability/SKILL.md) — alerts come from here.
+- [thinking_os](../thinking_os/SKILL.md) — CHAOTIC quadrant routes here (act → stabilize → then diagnose).
+- [deployment-cicd](../deployment-cicd/SKILL.md) — rollback is the most-used incident action.
+- [security-web](../security-web/SKILL.md) — when the incident is a security event, different playbook (containment, evidence preservation, comms differ).

core/skills/incident-response/assets/incident-checklist.md

@@ -0,0 +1,29 @@
+<!-- domain:BACKEND | layer:asset | ssot:false | updated:2026-06-04 -->
+# Incident Response Checklist
+
+During an incident, in order.
+
+## Declare
+- [ ] Severity classified (`python3 scripts/classify_severity.py ...` — data loss/breach = SEV1).
+- [ ] Incident declared in the agreed channel; roles assigned (Commander / Ops / Comms).
+- [ ] Status page updated if user-facing.
+
+## Mitigate (before diagnosing)
+- [ ] Stop the bleeding — rollback the last deploy is the fastest common fix.
+- [ ] Only ONE person changes prod (Ops) to avoid collisions.
+- [ ] Mitigation confirmed via the telemetry that detected the issue.
+
+## Communicate
+- [ ] Stakeholder updates on a cadence (even "still investigating").
+- [ ] Customer-facing comms if SLA/contract requires.
+
+## Resolve & learn
+- [ ] Recovery confirmed; incident closed.
+- [ ] Blameless postmortem scheduled (use docs/governance/_templates/post-mortem-template.md).
+- [ ] Action items have an owner + a date; tracked to done.
+- [ ] A guardrail added (test/alert/check) so the same class can't recur.
+
+## Pre-launch (design the process before you need it)
+- [ ] On-call rotation + escalation defined.
+- [ ] Runbooks written for known failure modes (docs/governance/_templates/runbook-template.md).
+- [ ] Every alert links to a runbook and is actionable.

core/skills/incident-response/references/severity-and-runbook.md

@@ -0,0 +1,53 @@
+<!-- domain:BACKEND | layer:reference | ssot:true | updated:2026-06-04 -->
+# Severity, On-Call, Comms, Lifecycle
+
+> P: Run an incident from detection to postmortem with a clear severity scale and roles.
+> R: During an incident, or designing the on-call/runbook process before launch.
+> S: Instrumentation that detects the incident — that's [observability](../../observability/SKILL.md).
+> N: [SKILL.md](../SKILL.md), [incident-checklist.md](../assets/incident-checklist.md)
+
+> Nav: [Skill](../SKILL.md)
+
+The postmortem and runbook *document templates* are co-shipping SSOT — use them,
+don't recreate: [post-mortem-template.md](../../../../docs/governance/_templates/post-mortem-template.md),
+[runbook-template.md](../../../../docs/governance/_templates/runbook-template.md).
+This reference is the *process*.
+
+## Severity scale (tune per org)
+
+| SEV | Meaning | Response |
+|---|---|---|
+| 1 | core down for many users / data loss / security breach | page now, incident commander, status page, all-hands |
+| 2 | major degradation or core down with a workaround | page, dedicated responder, fix in hours |
+| 3 | partial / minor impact | ticket + owner, normal hours |
+| 4 | cosmetic / negligible | backlog |
+
+`scripts/classify_severity.py` turns impact facts into a level so there's no
+debate mid-incident. Data loss and security breach are **always** SEV1.
+
+## Roles (even a 2-person team needs the split)
+
+- **Incident Commander** — owns the response, makes calls, not hands-on-keyboard.
+- **Operations** — the one making changes (one person touches prod, to avoid collisions).
+- **Communications** — updates stakeholders/status page so the IC isn't interrupted.
+
+For a small team one person may wear two hats — but name them, so nothing falls
+through the gap.
+
+## Lifecycle
+
+1. **Detect** — an alert (symptom-based, links a runbook) or a report.
+2. **Triage** — classify severity, declare the incident, assign roles.
+3. **Mitigate first, diagnose later** — stop the bleeding (rollback is the most
+   common and fastest mitigation — see [deployment-cicd](../../deployment-cicd/SKILL.md));
+   root cause can wait until users are unblocked.
+4. **Communicate** — regular updates on a cadence, even "still investigating".
+5. **Resolve** — confirm recovery via the same telemetry that detected it.
+6. **Postmortem** — blameless, within days, using the template. Action items have
+   owners + dates, or they don't exist.
+
+## Blameless postmortem
+
+The question is "what about the system let this happen?", never "who messed up".
+A human error that a system allowed is a system gap — add the guardrail (a test,
+an alert, a check) so the next person can't make it. Track action items to done.

core/skills/incident-response/scripts/classify_severity.py

@@ -0,0 +1,86 @@
+"""Classify an incident's severity (SEV1-4) from its impact signals.
+
+PURPOSE:      Remove the "what SEV is this?" debate at 3am — feed the impact
+              facts, get a deterministic level + the response it implies.
+INPUT:        flags describing impact:
+                --users-affected <0-100>  percent of users impacted
+                --data-loss               irreversible data loss/corruption
+                --security-breach         confirmed/suspected breach
+                --core-down               a core user journey is unavailable
+                --workaround              a viable workaround exists
+              [--json]
+OUTPUT:       Level + rationale + response on stderr; "SEV<n>" on stdout.
+              Exit 0 always (classification is not a failure). Exit 2 on usage.
+DEPENDENCIES: stdlib only.
+NOTES:        Pure classify() is unit-testable. Adjust thresholds per your SEV
+              policy. Spec: docs/playbooks/skill-authoring.md; craft: ../SKILL.md.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+RESPONSE = {
+    1: "page on-call now · incident commander · status page · all-hands until mitigated",
+    2: "page on-call · dedicated responder · update stakeholders · fix within hours",
+    3: "ticket + owner · fix in normal working hours",
+    4: "backlog · fix when convenient",
+}
+
+
+def classify(
+    *,
+    users_affected: float,
+    data_loss: bool,
+    security_breach: bool,
+    core_down: bool,
+    workaround: bool,
+) -> tuple[int, str]:
+    if data_loss or security_breach:
+        return 1, "irreversible data loss or security breach — always SEV1"
+    if core_down and users_affected >= 50 and not workaround:
+        return 1, "core journey down for most users, no workaround"
+    if core_down or users_affected >= 25:
+        level = 3 if workaround else 2
+        why = "core journey impacted" if core_down else f"{users_affected:.0f}% of users impacted"
+        return level, f"{why}{' (workaround exists → lower)' if workaround else ''}"
+    if users_affected > 0:
+        return 3, f"{users_affected:.0f}% of users impacted, no core journey down"
+    return 4, "minimal/no user impact"
+
+
+def main(argv: list[str]) -> int:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    parser.add_argument("--users-affected", default=0.0, type=float)
+    parser.add_argument("--data-loss", action="store_true")
+    parser.add_argument("--security-breach", action="store_true")
+    parser.add_argument("--core-down", action="store_true")
+    parser.add_argument("--workaround", action="store_true")
+    parser.add_argument("--json", action="store_true", dest="as_json")
+    args = parser.parse_args(argv)
+
+    if not 0 <= args.users_affected <= 100:
+        print("error: --users-affected must be 0-100", file=sys.stderr)
+        return 2
+
+    level, why = classify(
+        users_affected=args.users_affected,
+        data_loss=args.data_loss,
+        security_breach=args.security_breach,
+        core_down=args.core_down,
+        workaround=args.workaround,
+    )
+    response = RESPONSE[level]
+    print(f"  SEV{level}: {why}", file=sys.stderr)
+    print(f"  response: {response}", file=sys.stderr)
+    if args.as_json:
+        print(json.dumps({"severity": level, "rationale": why, "response": response}))
+    else:
+        print(f"SEV{level}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

core/skills/linux-sysadmin/SKILL.md

@@ -0,0 +1,115 @@
+---
+name: linux-sysadmin
+tier: infra
+domain: [infra]
+description: Operate and harden Linux hosts — SSH, systemd services, users/permissions, package management, networking, firewall, log triage, and resource inspection. Use when configuring a server, writing a systemd unit, hardening SSH, debugging "the box is slow / a port won't bind / a service won't start", setting up a firewall, or triaging a host under load. Targets Debian/Ubuntu + RHEL/Fedora families. Triggers — "ssh", "systemd", "the server", "permission denied", "port in use", "service won't start", "harden the box", "set up the VPS", "linux". Pairs with deployment-cicd (containers/pipelines), incident-response (host on fire), security-web (app-side hardening), shell-scripting (the automation).
+globs: ""
+paths: []
+last_reviewed: "2026-06-04"
+versions_ref: versions.json
+---
+
+# Linux System Administration
+
+A server is a contract with production: it stays up, it's reachable only how you intend, and when it misbehaves you can see why in one pass. This skill is host-level operation + hardening. Containers and pipelines belong to [deployment-cicd](../deployment-cicd/SKILL.md); the incident *process* to [incident-response](../incident-response/SKILL.md); this is the box itself.
+
+> Triage a host in one compact report (disk, mem, load, failed units, ports):
+> `bash scripts/triage.sh --json`
+
+## SSH — the front door, hardened
+
+```bash
+# /etc/ssh/sshd_config.d/10-hardening.conf  (drop-in, survives package upgrades)
+PermitRootLogin no
+PasswordAuthentication no            # keys only — the single biggest win
+PubkeyAuthentication yes
+KbdInteractiveAuthentication no
+MaxAuthTries 3
+AllowUsers deploy                    # allow-list, not everyone
+```
+
+```bash
+# Wrong — edits the main file, lost on upgrade, no syntax check before reload
+vi /etc/ssh/sshd_config && systemctl restart sshd     # locks you out if typo'd
+
+# Correct — drop-in + validate + reload (keeps your current session alive)
+sshd -t && systemctl reload sshd                       # -t aborts on bad config
+```
+
+Keys not passwords, root login off, an `AllowUsers` allow-list, `fail2ban` for brute-force. Never restart sshd before `sshd -t` passes — a typo + restart locks you out of a remote box. Full hardening → [references/ssh-hardening.md](references/ssh-hardening.md).
+
+## systemd — services that restart themselves
+
+```ini
+# /etc/systemd/system/api.service
+[Unit]
+Description=API service
+After=network-online.target
+
+[Service]
+ExecStart=/usr/local/bin/api --port 8080
+Restart=on-failure
+RestartSec=2
+User=api                             # never run a service as root
+NoNewPrivileges=true                 # cheap hardening
+ProtectSystem=strict                 # read-only /usr, /boot, /etc
+
+[Install]
+WantedBy=multi-user.target
+```
+
+`systemctl daemon-reload` after editing a unit, then `enable --now`. Read failures with `systemctl status api` + `journalctl -u api -e --no-pager`. A service with no `Restart=` dies silently on the first crash. Recipes → [references/systemd-and-services.md](references/systemd-and-services.md).
+
+## Permissions — least privilege, by default
+
+```bash
+# Wrong — 777 "to make it work" is an open door
+chmod -R 777 /var/www
+
+# Correct — owner writes, group reads, world nothing; dirs need +x to enter
+chown -R deploy:www-data /var/www
+find /var/www -type d -exec chmod 750 {} \;
+find /var/www -type f -exec chmod 640 {} \;
+```
+
+`chmod 777` is never the answer — it's "I don't know who needs access" written as a vulnerability. Files `640`, dirs `750`, secrets `600` owned by the service user. `sudo` for specific commands via `/etc/sudoers.d/`, not blanket root.
+
+## The host-triage reflex (slow box / won't bind / OOM)
+
+| Symptom | First command |
+|---|---|
+| slow / high load | `uptime` (load avg vs core count), `top`/`htop` |
+| out of memory | `free -h`, `dmesg | grep -i oom` (the OOM killer logs here) |
+| disk full | `df -h`, `du -xhd1 / | sort -h | tail` |
+| port won't bind | `ss -ltnp | grep :8080` (who holds it) |
+| service down | `systemctl status X`, `journalctl -u X -e` |
+| can't reach it | `ss -ltn`, then firewall: `ufw status` / `firewall-cmd --list-all` |
+
+`scripts/triage.sh` runs the safe read-only subset of these in one shot and emits a compact summary — read three lines instead of running six commands.
+
+## Packages & firewall (Debian/Ubuntu ↔ RHEL/Fedora)
+
+| | Debian/Ubuntu | RHEL/Fedora |
+|---|---|---|
+| install | `apt install` | `dnf install` |
+| update index | `apt update` | (dnf auto) |
+| firewall | `ufw allow 443/tcp` | `firewall-cmd --add-service=https --permanent` |
+| service logs | `journalctl` | `journalctl` |
+
+Default-deny inbound, allow only the ports you serve. Pin OS versions in [versions.json](versions.json); LTS in production, not the latest interim release.
+
+## Anti-patterns (reject on sight)
+
+- `chmod 777` / `PasswordAuthentication yes` / `PermitRootLogin yes` → all open doors.
+- Restarting `sshd` without `sshd -t` first → remote lockout.
+- Editing `/etc/ssh/sshd_config` directly instead of a `sshd_config.d/` drop-in.
+- A long-running daemon launched from a shell / `nohup` instead of a systemd unit (no restart, no logs, dies on logout).
+- Running a service as `root` "because permissions" → fix the ownership, drop the privilege.
+- `curl … | sudo bash` from an unverified source → read it first.
+
+## See also
+
+- [references/ssh-hardening.md](references/ssh-hardening.md) — keys, drop-ins, fail2ban, bastion.
+- [references/systemd-and-services.md](references/systemd-and-services.md) — units, timers, sandboxing, journald.
+- [assets/hardening-checklist.md](assets/hardening-checklist.md) — the new-server ship gate.
+- [deployment-cicd](../deployment-cicd/SKILL.md) · [incident-response](../incident-response/SKILL.md) · [shell-scripting](../shell-scripting/SKILL.md).

core/skills/linux-sysadmin/assets/hardening-checklist.md

@@ -0,0 +1,29 @@
+<!-- domain:INFRA | layer:asset | ssot:false | updated:2026-06-04 -->
+# New-Server Hardening Checklist
+
+Run before a host serves production traffic.
+
+## Access
+- [ ] SSH: `PasswordAuthentication no`, `PermitRootLogin no`, `AllowUsers` allow-list (drop-in file).
+- [ ] `sshd -t` passes; reloaded (not restarted) with a second session open.
+- [ ] Keys: ed25519, `~/.ssh` 700, `authorized_keys` 600, one key per identity.
+- [ ] fail2ban (or equivalent) enabled on the sshd jail.
+- [ ] A non-root deploy user with scoped `sudoers.d/` — no blanket root.
+
+## Network
+- [ ] Firewall default-deny inbound; only served ports open (`ufw`/`firewalld`).
+- [ ] SSH restricted to known source CIDRs or behind a bastion.
+- [ ] No unexpected listeners — `ss -ltnp` reviewed.
+
+## Services
+- [ ] Every long-running process is a systemd unit with `Restart=on-failure`.
+- [ ] Services run as dedicated non-root users.
+- [ ] Sandboxing on: `NoNewPrivileges`, `ProtectSystem=strict`, `PrivateTmp`.
+- [ ] Logs go to journald; crash loops bounded with `StartLimit*`.
+
+## Hygiene
+- [ ] OS on a supported LTS (`make skills-check-versions` for the pin).
+- [ ] Unattended security updates enabled.
+- [ ] Permissions: dirs 750, files 640, secrets 600 — no `chmod 777` anywhere.
+- [ ] `scripts/triage.sh --json` baseline captured (disk/mem/load/ports/failed).
+- [ ] Backups configured as a systemd timer + restore tested.

core/skills/linux-sysadmin/references/ssh-hardening.md

@@ -0,0 +1,62 @@
+<!-- domain:INFRA | layer:reference | ssot:true | updated:2026-06-04 -->
+# SSH Hardening
+
+> P: Lock SSH to keys + allow-list, survive package upgrades, resist brute force.
+> R: Provisioning a server, exposing SSH to the internet, or reviewing access.
+> S: App-layer auth (JWT/OAuth) — that's [auth-patterns](../../auth-patterns/SKILL.md).
+> N: [SKILL.md](../SKILL.md), [systemd-and-services.md](systemd-and-services.md)
+
+> Nav: [Skill](../SKILL.md)
+
+## Use a drop-in, never the main file
+
+```bash
+# /etc/ssh/sshd_config.d/10-hardening.conf
+PermitRootLogin no
+PasswordAuthentication no
+PubkeyAuthentication yes
+KbdInteractiveAuthentication no
+MaxAuthTries 3
+MaxSessions 4
+LoginGraceTime 20
+AllowUsers deploy
+X11Forwarding no
+ClientAliveInterval 300
+ClientAliveCountMax 2
+```
+
+Drop-ins in `sshd_config.d/` survive `apt upgrade` (which may replace the main
+file). Always validate before reloading:
+
+```bash
+sshd -t && systemctl reload sshd      # -t = syntax check; reload keeps your session
+```
+
+`reload` (not `restart`) keeps existing sessions — so a mistake doesn't lock you
+out before you can fix it. Keep a second session open while changing SSH.
+
+## Keys done right
+
+```bash
+ssh-keygen -t ed25519 -C "deploy@host"     # ed25519 > rsa: smaller, faster, modern
+ssh-copy-id deploy@host                      # appends to ~/.ssh/authorized_keys
+chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys
+```
+
+`authorized_keys` must be `600`, `~/.ssh` `700`, owned by the user — sshd
+refuses keys on loose permissions. Rotate keys on offboarding; one key per
+person/machine so you can revoke precisely.
+
+## Brute-force + exposure
+
+- **fail2ban** — bans IPs after N failed auths (`/etc/fail2ban/jail.local`, `sshd` jail).
+- **Non-standard port** — cuts log noise, not real security (don't rely on it alone).
+- **Bastion / jump host** — production hosts have no public SSH; reach them via one audited `ProxyJump` bastion.
+- **Firewall** — `ufw allow from <office-cidr> to any port 22` restricts source.
+
+## Verify the lockdown
+
+```bash
+ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no deploy@host
+# MUST fail with "Permission denied (publickey)" — proves passwords are off.
+```