runsheets 0.0.1

data/examples/directory_based_runbooks/staging-teardown/rollback.md

@@ -0,0 +1,28 @@
+---
+title: Rollback
+---
+
+There is no rolling back a deleted stack; there is rebuilding from the
+snapshot. How far you got decides what to do.
+
+**Stopped before step 60 (nothing deleted).** Re-enable the pipeline and
+scale the services back up. The next deploy restores the desired counts;
+this just gets staging answering again sooner:
+
+```bash
+gh workflow enable deploy-staging.yml --repo acme/platform
+for svc in web worker scheduler; do
+  aws ecs update-service --cluster "$CLUSTER" --service "$svc" --desired-count 2
+done
+```
+
+**Step 60 failed partway (`DELETE_FAILED`).** The stack is in a state
+CloudFormation will not touch until the failing resource is dealt with.
+Find it with the last block of step 60, remove whatever references it
+outside the stack, and run the delete again. Do not retain resources with
+`--retain-resources` unless you intend to clean them up by hand.
+
+**Staging is needed again after step 60.** Restore the snapshot into a
+new instance and point a fresh stack at it. This is the rebuild runbook,
+not this one; the snapshot identifier is `$SNAPSHOT_ID` and the run
+record of the teardown has the row counts to compare against.

data/examples/directory_based_runbooks/staging-teardown/runbook.md

@@ -0,0 +1,57 @@
+---
+title: Staging infrastructure teardown
+when_to_use: >
+  The staging environment is being retired or rebuilt from scratch and
+  nothing in it needs to survive. Not for routine deploys, not for
+  production, never while a release is being verified on staging.
+prerequisites:
+  - AWS CLI v2 logged in through SSO to the staging account (`aws sso login`)
+  - GitHub CLI (`gh`) authenticated with permission to disable workflows
+  - Session Manager plugin for the SSM port forward in step 50
+  - jq on the PATH
+blast_radius: >
+  Every ECS service, the RDS instance, the load balancer and the DNS record
+  for staging. Anything in staging that was not snapshotted in step 20 is
+  gone for good. Production is untouched: a different account, a different
+  profile.
+escalation: The platform on-call engineer, via the #platform-oncall channel.
+tags: [aws, ecs, rds, staging, destructive]
+inputs:
+  - name: AWS_PROFILE
+    prompt: AWS profile for the staging account
+    default: acme-staging-admin
+  - name: AWS_REGION
+    prompt: Region the staging stack lives in
+    default: us-east-1
+  - name: CLUSTER
+    prompt: ECS cluster name
+    default: acme-staging
+  - name: DB_INSTANCE
+    prompt: RDS instance identifier
+    default: acme-staging-db
+  - name: SNAPSHOT_ID
+    prompt: Identifier for the final pre-teardown snapshot
+    default: acme-staging-final-preteardown
+  - name: STACK_NAME
+    prompt: CloudFormation stack that owns the environment
+    default: acme-staging
+---
+
+# Staging infrastructure teardown
+
+This runbook takes the staging environment down in an order that leaves a
+restorable database snapshot and no orphaned resources. It is the shape of
+a real teardown runbook: the commands are the ones you would type, the
+account names are made up.
+
+Read the blast radius above before starting a run. The steps are:
+
+1. Confirm with the team that nothing in staging needs to be kept.
+2. Take and verify a final RDS snapshot.
+3. Stop the deployment pipeline so nothing redeploys mid-teardown.
+4. Drain the ECS services to zero.
+5. Open a tunnel to the database and record the final row counts.
+6. Delete the CloudFormation stack.
+
+Every executed command, its output and your acknowledgements are recorded.
+Attach the run record to the change ticket when you are done.

data/examples/directory_based_runbooks/staging-teardown/steps/010-confirm-nothing-to-preserve.md

@@ -0,0 +1,18 @@
+---
+title: Confirm nothing in staging needs to be preserved
+kind: manual
+---
+
+Post in `#eng-announcements` that staging is being torn down, with the
+change ticket number, and wait for anyone with an in-flight test to
+respond. Give it the time the ticket says, usually an hour.
+
+Then check the things people forget:
+
+- Feature branches deployed to staging for review. Ask their owners.
+- Data loaded by hand for a demo. If it matters, export it now; the
+  snapshot in the next step keeps the database, not S3 uploads.
+- Scheduled jobs that write somewhere outside staging.
+
+Mark this step done with a note saying who confirmed and when. That note
+is the audit trail for "we asked".

data/examples/directory_based_runbooks/staging-teardown/steps/020-snapshot-the-database.md

@@ -0,0 +1,47 @@
+---
+title: Take a final snapshot of the database
+kind: automated
+timeout: 1800
+---
+
+A manual snapshot survives the deletion of the instance; automated ones do
+not. This is the only copy of staging data that will exist afterwards, so
+wait for it to reach `available` before moving on.
+
+```bash run
+set -euo pipefail
+aws rds create-db-snapshot \
+  --db-instance-identifier "$DB_INSTANCE" \
+  --db-snapshot-identifier "$SNAPSHOT_ID" \
+  --query 'DBSnapshot.{Id:DBSnapshotIdentifier,Status:Status}' --output table
+aws rds wait db-snapshot-available --db-snapshot-identifier "$SNAPSHOT_ID"
+aws rds describe-db-snapshots --db-snapshot-identifier "$SNAPSHOT_ID" \
+  --query 'DBSnapshots[0].{Id:DBSnapshotIdentifier,Status:Status,SizeGB:AllocatedStorage,Created:SnapshotCreateTime}' \
+  --output table
+```
+
+The first table shows the snapshot `creating`; the second is what you
+want to see, with `Status` at `available`. The borders, the time, the
+size and the snapshot id differ from run to run, so they are `{{any}}`:
+
+```text expect
+{{any}}
+|{{any}}CreateDBSnapshot{{any}}|
+{{any}}
+|{{any}}Id{{any}}Status{{any}}|
+{{any}}
+|  {{any}}|  creating  |
+{{any}}
+{{any}}
+|{{any}}DescribeDBSnapshots{{any}}|
+{{any}}
+|{{any}}Created{{any}}Id{{any}}SizeGB{{any}}Status{{any}}|
+{{any}}
+|  {{any}}|  available  |
+{{any}}
+```
+
+If `create-db-snapshot` fails with `DBSnapshotAlreadyExists`, someone ran
+this step already. Check the existing snapshot's date before reusing it;
+if it is older than today, pick a new `SNAPSHOT_ID` and start the run
+again.

data/examples/directory_based_runbooks/staging-teardown/steps/030-stop-the-pipeline.md

@@ -0,0 +1,32 @@
+---
+title: Stop the deployment pipeline
+kind: automated
+timeout: 120
+---
+
+Nothing must redeploy to staging while the services are being drained.
+Disable the deploy workflow rather than deleting it, so re-enabling it is
+one command when the environment is rebuilt.
+
+```bash run
+set -euo pipefail
+gh workflow disable deploy-staging.yml --repo acme/platform
+gh workflow list --repo acme/platform --all | grep -i staging
+```
+
+Expect the staging workflow to show `disabled_manually`. The columns are
+separated by tabs or spaces depending on the terminal, and the last one
+is the workflow id:
+
+```text expect
+Deploy to staging{{any}}disabled_manually{{any}}
+```
+
+Some teams also have a bot that re-enables workflows it finds disabled.
+If yours does, pause it in its own UI; that part is interactive, so do it
+in your terminal and confirm here:
+
+```bash terminal
+gh auth status
+open "https://github.com/acme/platform/actions/workflows/deploy-staging.yml"
+```

data/examples/directory_based_runbooks/staging-teardown/steps/040-drain-the-ecs-services.md

@@ -0,0 +1,35 @@
+---
+title: Drain the ECS services
+kind: automated
+timeout: 900
+---
+
+Scale every service in the cluster to zero and wait for the tasks to stop.
+The load balancer starts returning 503 as soon as the last task exits,
+which is expected; the DNS record goes away with the stack in step 60.
+
+```bash run
+set -euo pipefail
+services=$(aws ecs list-services --cluster "$CLUSTER" --query 'serviceArns[]' --output text)
+test -n "$services" || { echo "no services in $CLUSTER"; exit 0; }
+for svc in $services; do
+  name=${svc##*/}
+  echo "scaling $name to 0"
+  aws ecs update-service --cluster "$CLUSTER" --service "$name" --desired-count 0 --query 'service.desiredCount' --output text
+done
+aws ecs wait services-stable --cluster "$CLUSTER" --services $services
+aws ecs describe-services --cluster "$CLUSTER" --services $services \
+  --query 'services[].{Name:serviceName,Desired:desiredCount,Running:runningCount}' --output table
+```
+
+Every row of the final table should read `0` in both `Desired` and
+`Running`. If a service refuses to drain (a task stuck in `DEPROVISIONING`
+for more than ten minutes), stop its tasks directly:
+
+```bash
+aws ecs list-tasks --cluster "$CLUSTER" --service-name web --query 'taskArns[]' --output text \
+  | xargs -n1 aws ecs stop-task --cluster "$CLUSTER" --reason "staging teardown" --task
+```
+
+That block is display only on purpose: run it from your terminal against
+the one service that is stuck, not against all of them.

data/examples/directory_based_runbooks/staging-teardown/steps/050-record-the-final-row-counts.md

@@ -0,0 +1,41 @@
+---
+title: Record the final row counts
+kind: verify
+timeout: 300
+---
+
+Before the database goes, write its row counts into the run record. When
+staging is rebuilt from the snapshot, the same query proves the restore
+is complete.
+
+The database is only reachable through the bastion. Open a port forward
+and leave it running while the queries run; stop it from the Running
+panel, or let finishing the run stop it:
+
+```bash background
+aws ssm start-session \
+  --target "$(aws ec2 describe-instances --filters Name=tag:Role,Values=bastion Name=instance-state-name,Values=running \
+      --query 'Reservations[0].Instances[0].InstanceId' --output text)" \
+  --document-name AWS-StartPortForwardingSessionToRemoteHost \
+  --parameters "{\"host\":[\"$(aws rds describe-db-instances --db-instance-identifier "$DB_INSTANCE" \
+      --query 'DBInstances[0].Endpoint.Address' --output text)\"],\"portNumber\":[\"5432\"],\"localPortNumber\":[\"15432\"]}"
+```
+
+With the tunnel up, count the rows in the tables that matter. The password
+comes from the secret in the staging account, never from this file:
+
+```bash run
+set -euo pipefail
+export PGPASSWORD=$(aws secretsmanager get-secret-value --secret-id "staging/db/app" --query SecretString --output text | jq -r .password)
+psql -h localhost -p 15432 -U app -d app -X -At -c "
+  select relname, n_live_tup from pg_stat_user_tables order by n_live_tup desc limit 10;"
+```
+
+```text expect
+events|{{any}}
+orders|{{any}}
+users|{{any}}
+```
+
+The counts are `{{any}}` because they differ every time; the point is
+that the query runs and the record keeps whatever it printed.

data/examples/directory_based_runbooks/staging-teardown/steps/060-delete-the-stack.md

@@ -0,0 +1,56 @@
+---
+title: Delete the CloudFormation stack
+kind: automated
+destructive: true
+timeout: 3600
+---
+
+This is the step that cannot be undone. Everything the stack owns goes:
+the ECS cluster and services, the RDS instance (its final snapshot from
+step 20 stays), the load balancer, the DNS record, the security groups.
+
+Check once more that the snapshot exists and is `available`, in the same
+process that will delete the stack, so a wrong `AWS_PROFILE` fails here
+rather than on the delete:
+
+```bash run
+set -euo pipefail
+aws sts get-caller-identity --query '{Account:Account,Arn:Arn}' --output table
+aws rds describe-db-snapshots --db-snapshot-identifier "$SNAPSHOT_ID" --query 'DBSnapshots[0].Status' --output text
+```
+
+The caller identity table comes first, then the snapshot status. The
+account and ARN are yours to check by eye, so they are `{{any}}` here:
+
+```text expect
+{{any}}
+|{{any}}GetCallerIdentity{{any}}|
+{{any}}
+|{{any}}Account{{any}}Arn{{any}}|
+{{any}}
+|  {{any}}|  arn:aws:{{any}}|
+{{any}}
+available
+```
+
+Then delete. The confirmation code the page asks for is a speed bump; the
+real check is the table above showing the staging account, not
+production.
+
+```bash destructive
+set -euo pipefail
+aws cloudformation delete-stack --stack-name "$STACK_NAME"
+aws cloudformation wait stack-delete-complete --stack-name "$STACK_NAME"
+echo "stack $STACK_NAME deleted"
+```
+
+`wait stack-delete-complete` can take forty minutes for a stack with an
+RDS instance in it; the step timeout allows an hour. If it fails with
+`DELETE_FAILED`, the events tell you which resource refused, usually a
+security group something outside the stack still references:
+
+```bash run
+aws cloudformation describe-stack-events --stack-name "$STACK_NAME" \
+  --query 'StackEvents[?ResourceStatus==`DELETE_FAILED`].{Resource:LogicalResourceId,Reason:ResourceStatusReason}' \
+  --output table
+```

data/examples/directory_based_runbooks/staging-teardown/verify.md

@@ -0,0 +1,40 @@
+---
+title: Verify
+kind: verify
+timeout: 300
+---
+
+Run after the last step, or on their own from the **Checks** page at any
+time afterwards: these prove staging is really gone and the snapshot is
+really there.
+
+```bash run
+set -euo pipefail
+status=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --query 'Stacks[0].StackStatus' --output text 2>&1 || true)
+case "$status" in
+  *"does not exist"*) echo "stack $STACK_NAME: gone" ;;
+  *) echo "stack $STACK_NAME still reports: $status"; exit 1 ;;
+esac
+```
+
+```bash run
+set -euo pipefail
+aws ecs describe-clusters --clusters "$CLUSTER" --query 'clusters[0].status' --output text | grep -qx INACTIVE \
+  && echo "cluster $CLUSTER: inactive" || { echo "cluster $CLUSTER is still active"; exit 1; }
+```
+
+```bash run
+set -euo pipefail
+aws rds describe-db-snapshots --db-snapshot-identifier "$SNAPSHOT_ID" \
+  --query 'DBSnapshots[0].{Status:Status,SizeGB:AllocatedStorage,Encrypted:Encrypted}' --output table
+```
+
+```text expect
+--------------------------------------
+|         DescribeDBSnapshots        |
++-----------+----------+-------------+
+| Encrypted | SizeGB   |   Status    |
++-----------+----------+-------------+
+|  True     |  {{any}} |  available  |
++-----------+----------+-------------+
+```

data/examples/single_md_file_runbooks/db-maintenance.md

@@ -0,0 +1,163 @@
+---
+title: Monthly PostgreSQL maintenance
+when_to_use: >
+  The first Monday of the month, or when the bloat report in the monitoring
+  dashboard crosses 30% on any table. Run against one database at a time.
+prerequisites:
+  - psql 14 or newer on the PATH
+  - A role with ownership of the application tables (VACUUM and REINDEX need it)
+  - Nobody running a long report; REINDEX takes a lock
+escalation: The data platform channel; after hours, the database on-call.
+tags: [postgres, maintenance]
+inputs:
+  - name: PGHOST
+    prompt: Database host
+    default: localhost
+  - name: PGPORT
+    prompt: Port
+    default: "5432"
+  - name: PGDATABASE
+    prompt: Database name
+    default: app
+  - name: PGUSER
+    prompt: Role to connect as
+    default: app_owner
+  - name: PGPASSWORD
+    prompt: Password for that role
+    secret: true
+interpreters:
+  sql: psql -X -v ON_ERROR_STOP=1 --pset footer=off -f
+---
+
+# Monthly PostgreSQL maintenance
+
+A single-file runbook: this one markdown file holds the front matter, this
+preamble, and one `##` heading per step. It also shows `sql run` blocks.
+They execute because the front matter maps `sql` to a `psql` command; the
+file runsheets writes each block to (`<block-id>.<n>.cmd` in the run
+directory) is appended to that command, and `psql` picks the connection
+up from the `PG*` inputs in the environment. The password is a secret input: it reaches `psql`, never the
+run record, and is redacted if anything prints it.
+
+Step attributes live in an HTML comment right after each heading. GitHub
+and every other renderer hide the comment, so the file reads as ordinary
+markdown anywhere else.
+
+## Check the connection
+<!-- kind: verify, timeout: 30 -->
+
+Prove the inputs are right before anything else. This also runs on its
+own from the Checks page.
+
+```sql run
+select current_database() as database, current_user as role, version();
+```
+
+The version string, and so the width of its column, depends on the
+server; `{{any}}` stands in for both:
+
+```text expect
+ database |   role    |{{any}}version
+----------+-----------+{{any}}
+ app      | app_owner | PostgreSQL {{any}}
+```
+
+## Find the bloated tables
+<!-- kind: automated, timeout: 120 -->
+
+Dead tuples that autovacuum has not caught up with. Anything with more
+than a fifth of its rows dead is a candidate for the next step.
+
+```sql run
+select relname,
+       n_live_tup as live,
+       n_dead_tup as dead,
+       round(100.0 * n_dead_tup / greatest(n_live_tup + n_dead_tup, 1), 1) as dead_pct,
+       last_autovacuum
+  from pg_stat_user_tables
+ where n_dead_tup > 10000
+ order by dead_pct desc
+ limit 15;
+```
+
+Note the worst table in the step note; the next step takes its name as a
+literal because `VACUUM` cannot be parameterised.
+
+## Vacuum the worst table
+<!-- kind: automated, timeout: 1800 -->
+
+`VACUUM (VERBOSE, ANALYZE)` does not take an exclusive lock and can run
+while the application is up. Edit the table name before clicking Run;
+the exact text that ran is what the record keeps.
+
+```sql run
+vacuum (verbose, analyze) events;
+```
+
+Then confirm the dead count dropped:
+
+```sql run
+select relname, n_live_tup, n_dead_tup, last_vacuum
+  from pg_stat_user_tables
+ where relname = 'events';
+```
+
+## Reindex
+<!-- kind: automated, destructive: true, timeout: 3600 -->
+
+`REINDEX` takes a lock that blocks writes to the table for its duration.
+It is marked destructive so the page asks for a confirmation code and
+shows the escalation contact: the harm is downtime, not data loss.
+
+```sql destructive
+reindex (verbose) table concurrently events;
+```
+
+If `CONCURRENTLY` fails partway it leaves an invalid index behind. Find
+and drop it:
+
+```sql run
+select indexrelid::regclass as index, indisvalid
+  from pg_index
+ where indrelid = 'events'::regclass and not indisvalid;
+```
+
+## Tell the team
+<!-- kind: manual -->
+
+Post the before and after dead-tuple counts from steps 2 and 3 in the
+data platform channel. Mark this step done with a link to the post.
+
+## Verify
+
+Whole-procedure checks. Everything here is read-only and safe to run at
+any time from the Checks page.
+
+```sql run
+select count(*) filter (where not indisvalid) as invalid_indexes
+  from pg_index;
+```
+
+```text expect
+ invalid_indexes
+-----------------
+               0
+```
+
+```sql run
+select relname, n_dead_tup
+  from pg_stat_user_tables
+ where n_dead_tup > 100000
+ order by n_dead_tup desc;
+```
+
+## Rollback
+
+Nothing here destroys data. A `VACUUM` interrupted partway has done part
+of its work and left the rest for next time. A `REINDEX CONCURRENTLY`
+interrupted partway leaves an invalid index; drop it with the query in
+the Reindex step and run the reindex again when there is a quiet window:
+
+```sql
+drop index concurrently events_created_at_idx_ccnew;
+```

data/examples/single_md_file_runbooks/disk-space-triage.md

@@ -0,0 +1,90 @@
+---
+title: Disk space triage
+when_to_use: >
+  A host is close to full: an alert on free space, a build failing with
+  "No space left on device", or a log directory that keeps growing.
+prerequisites:
+  - A shell on the host, as a user that can read the directory to check
+  - The cleanup policy (linked below) open in another tab
+escalation: The owner of the host; for shared build machines, the platform channel.
+tags: [disk, safe, example]
+inputs:
+  - name: TARGET_DIR
+    prompt: Directory to check
+    default: /tmp
+  - name: THRESHOLD
+    prompt: Highest acceptable use, in percent
+    default: "90"
+---
+
+# Disk space triage
+
+A single-file runbook that leans on plain documents kept elsewhere in the
+examples directory. The background lives there, not in the steps:
+
+- [Disk usage glossary](disk-usage-glossary.md): what `df` and `du`
+  report and why they can disagree.
+- [Cleanup policy](policies/cleanup-policy.md): what may be deleted, and what
+  must never be.
+
+Those files have no front matter, so runsheets treats them as documents:
+they stay out of the library tree, and the links above open them as pages.
+A document can sit anywhere under the directory `runsheets` was started on;
+one is beside this file, the other in `policies/`. See also
+[about these examples](../about-the-examples.md).
+
+Every block in this runbook only reads; nothing is deleted.
+
+## Check free space
+<!-- kind: verify, timeout: 30 -->
+
+How full is the file system that holds `TARGET_DIR`? `df` answers for the
+whole file system, not the directory; see
+[df and du](disk-usage-glossary.md#df-and-du) if that is surprising.
+
+```bash run
+df -h "$TARGET_DIR"
+```
+
+## Find what is using it
+<!-- timeout: 120 -->
+
+The ten largest entries directly under `TARGET_DIR`. Entries you cannot
+read are skipped.
+
+```bash run
+du -sh "$TARGET_DIR"/* 2>/dev/null | sort -rh | head -10
+```
+
+If the total here is far below what `df` reported, the space is held
+somewhere else: another directory on the same file system, or deleted files
+a process still holds open. The glossary explains
+[deleted-but-open files](disk-usage-glossary.md#deleted-but-open-files).
+
+## Decide what to clean
+<!-- kind: manual -->
+
+Compare what you found with the [cleanup policy](policies/cleanup-policy.md).
+Delete only what it lists as safe, in your own terminal, and note here what
+you removed and how much it freed.
+
+```bash
+rm -rf "$TARGET_DIR"/some-build-cache
+```
+
+## Verify
+
+Use is at or below the threshold.
+
+```bash run
+used=$(df -P "$TARGET_DIR" | awk 'NR == 2 { sub("%", "", $5); print $5 }')
+echo "use ${used}%, threshold ${THRESHOLD}%"
+test "$used" -le "$THRESHOLD"
+```
+
+## Rollback
+
+Nothing in this runbook deletes anything. If a manual cleanup removed
+something it should not have, restore it from backup and tell the owner of
+the host; the [cleanup policy](policies/cleanup-policy.md#when-something-was-deleted-by-mistake)
+says who to call.

data/examples/single_md_file_runbooks/disk-usage-glossary.md

@@ -0,0 +1,48 @@
+# Disk usage glossary
+
+Background for the [disk space triage](disk-space-triage.md) runbook.
+This file has no front matter, so runsheets treats it as a plain document:
+it is not in the library tree, and nothing on this page executes.
+
+## File system and mount point
+
+A disk, or a slice of one, is formatted as a **file system** and attached
+to the directory tree at a **mount point**. Everything below a mount point
+lives on that file system until another mount point takes over.
+
+```bash
+df -h /            # the file system mounted at /
+mount | head       # what is mounted where
+```
+
+## df and du
+
+`df` asks the file system how many blocks are in use. `du` walks a directory
+and adds up the sizes of the files it can see. They answer different
+questions:
+
+- `df -h /tmp` reports the whole file system that holds `/tmp`, including
+  everything outside `/tmp`.
+- `du -sh /tmp` reports only what is under `/tmp`, and only what you can
+  read.
+
+So `du` coming in well below `df` is normal. A large gap is worth a look.
+
+## Deleted-but-open files
+
+Deleting a file removes its name. The space is freed only when no process
+still has the file open. A log file deleted while a server is writing to it
+keeps growing, invisible to `du`, until the server is restarted or reopens
+its logs.
+
+```bash
+lsof +L1           # open files with no remaining name
+```
+
+## Inodes
+
+Every file uses one inode. A file system can run out of inodes with plenty
+of free blocks, usually from millions of tiny files. `df -i` shows inode
+use.
+
+See the [cleanup policy](policies/cleanup-policy.md) for what to do about any of it.