skilldrop-cli 1.2.0 → 1.3.0

package/packs/infra-as-code/loops/provision/LOOP.md

@@ -0,0 +1,128 @@
+---
+name: provision
+description: Take an infrastructure change from a described need to a Terraform apply a person approved: a module with secure defaults, a review of the real plan that names every destroy, replace, widening and out-of-scope change, and a check right before apply that the plan being applied is the one reviewed, with a gate that sends the module back when the plan would destroy state or widen access unasked. Use when the user says "write the terraform and check it before we apply", "provision this safely", "run the provision loop", or wants new infrastructure from module to apply in one pass.
+---
+
+# provision
+
+One infrastructure change, from a need to an apply. A module with secure defaults can still
+produce a plan that replaces a database, because the plan depends on the state and the
+environment, not only on the code. So the loop reviews the real plan against the scope written
+when the module was, and checks at apply time that nothing moved between review and apply.
+
+## Stages
+
+| # | Stage | Type | Skills | Gate |
+|---|---|---|---|---|
+| 1 | `module` | generate | [`terraform-module`](../../skills/terraform-module/SKILL.md) | — |
+| 2 | `plan` | verify | [`terraform-plan-review`](../../skills/terraform-plan-review/SKILL.md) | **G13** review |
+| 3 | `apply` | gate | [`terraform-plan-review`](../../skills/terraform-plan-review/SKILL.md) | **G13.1** human |
+
+## How to run this loop
+
+1. **Write the module and the scope together.** Run `terraform-module` on the need. Beside it,
+   write `scope.md`: the resource addresses this change is meant to create, change or destroy,
+   and one sentence of intent. The scope is written now, before any plan exists, so the review
+   can't read the plan to fit it.
+2. **Get a saved plan.** The person, or you where credentials allow, runs the commands below.
+   A plan printed without `-out` can't be applied as reviewed, so it doesn't count.
+
+   ```bash
+   terraform init
+   terraform plan -out=tfplan
+   terraform show -json tfplan > plan.json
+   ```
+
+   Record the planfile's SHA-256 in `scope.md`. Without credentials or state, stop and say
+   exactly which commands to run and what to bring back.
+3. **Review the plan against the scope.** Run `terraform-plan-review` on `plan.json`, passing
+   every address in `scope.md` as the intended scope. Every change outside it is listed, even
+   when harmless. Save the review as `review.md`.
+4. **Judge the plan at G13.** Map the review's verdict and findings to the gate:
+   - `SAFE TO APPLY` with nothing out of scope is `PROCEED`.
+   - `APPLY WITH CARE` is `PROCEED WITH CONDITIONS`, and the review's pre-apply checks are the
+     conditions.
+   - `DO NOT APPLY` caused by the code (a forced replacement, a widened rule, a missing
+     `prevent_destroy`) is `REVISE`.
+   - `DO NOT APPLY` caused by the change itself, such as a stateful resource that has to be
+     replaced to reach the goal, is `RECONSIDER`.
+5. **On `REVISE`, fix the module, then re-plan.** Return to `module` with the forcing attribute
+   or the widened rule named. Re-plan from the fixed code; never edit `plan.json`. Each return
+   counts against the cap.
+6. **Check the apply at G13.1.** Right before apply, confirm two things:
+   - the planfile's SHA-256 matches the one recorded in `scope.md`;
+   - `terraform show -json tfplan` still lists the same resource changes `review.md` reviewed.
+
+   Any difference is `REVISE` back to `plan`: re-plan and re-review. Then the person who will
+   run `terraform apply tfplan` reads every destroy and replace in `review.md` and decides.
+   Only they choose the verdict. Never run `terraform apply` yourself in this loop.
+7. **Hand over the verdict and what apply will do.** Lead with what changes in plain words,
+   what is destroyed or replaced, and the checks to do first. The review goes underneath.
+   Close with each gate on its own line, `G13: <VERDICT>` and `G13.1: <VERDICT>` (or
+   `G13.1: awaiting <role>`), and one sentence of reasons.
+
+## Verdicts
+
+Defined in [`contracts/terminals.json`](../../../../contracts/terminals.json). G13 emits:
+
+- `PROCEED`: every change is in scope, nothing stateful is destroyed or replaced, no access is
+  widened, and no unexplained drift.
+- `PROCEED WITH CONDITIONS`: in-scope replacements of stateless resources, in-place changes to
+  stateful ones, or drift the plan overwrites. The review's pre-apply checks travel to G13.1.
+- `REVISE`: the code causes a destroy, a replace or a widening nobody asked for. Return to
+  `module`.
+- `RECONSIDER`: the change can only be made by replacing state. It needs a migration with a
+  backup and a rollback, not a module edit, so the loop ends here and names that work.
+- `BLOCKED`: no plan to review, or a plan that errored.
+
+G13.1 emits `PROCEED`, `REVISE` (back to `plan`, when the planfile or its changes moved) or
+`BLOCKED` (no planfile, or nobody named to run apply).
+
+On the cap-th `REVISE` at G13, emit `RECONSIDER`: the module isn't converging on a safe plan,
+so the change needs a person to rethink it.
+
+## Non-interactive runs
+
+Missing provider or Terraform versions degrade to `terraform-module`'s `[assumption]` tags.
+With no plan and no way to run one, the loop ends after `module` with
+`G13: BLOCKED: need a saved plan (terraform plan -out=tfplan, then terraform show -json tfplan)`.
+G13.1 always stops at the person who runs apply.
+
+## When a stage's skill is not installed
+
+Name the missing skill and skilldrop as its source, do the minimal inline version of that
+stage, and record that it ran degraded. A degraded `plan` still lists every resource whose
+actions include `delete`, every change to a security group, IAM policy or bucket policy, and
+every address outside `scope.md`, because G13 can't be judged without them. Never skip G13 or
+G13.1.
+
+## Run log (opt-in)
+
+Only when the environment variable `SKILLDROP_LOOP_LOG` is set to a file path: after each
+gate's verdict, append one JSON line to that file. Nothing else is recorded, and nothing is
+sent anywhere. When the variable isn't set, or you can't read it, skip this section.
+
+```bash
+printf '%s\n' '{"ts":"2026-11-03T10:15:00Z","loop":"provision","stage":"plan","gate":"G13","verdict":"REVISE","round":1}' >> "$SKILLDROP_LOOP_LOG"
+```
+
+Fields: `ts` (UTC), `loop`, `stage`, `gate` (`null` for a stage without one), `verdict` (the
+gate's exact word), and `round` (1 for the first attempt, counting returns under the cap).
+The log holds verdicts only: never resource names, plan contents or values.
+`skilldrop loop-stats` summarises the file.
+
+## Quality bar
+
+- **The scope is written before the plan exists,** as resource addresses. A scope written after reading the plan covers whatever the plan does.
+- **Only a saved planfile is reviewed.** A plan printed without `-out` can't be applied as reviewed.
+- **Every change outside the scope is listed,** even when harmless.
+- **The plan applied is the plan reviewed.** G13.1 checks the planfile's hash and its resource changes.
+- **The loop never runs `terraform apply`.** The person who runs it decides at G13.1.
+
+## Anti-patterns to avoid
+
+- ❌ **Reviewing the module instead of the plan.** Secure code can still replace a database when the state disagrees with it.
+- ❌ **Fixing a bad plan by editing plan.json.** Fix the code and re-plan.
+- ❌ **Passing G13 because the destroy is "probably fine".** A stateful destroy without a confirmed backup is `DO NOT APPLY`, and the loop maps it to `REVISE` or `RECONSIDER`.
+- ❌ **Applying a fresh plan after reviewing an old one.** Anything merged in between is unreviewed.
+- ❌ **Treating `RECONSIDER` as a bigger `REVISE`.** A change that has to replace state is a migration; another module edit won't make it safe.

package/packs/infra-as-code/loops/provision/evals/eval_queries.json

@@ -0,0 +1,9 @@
+[
+  {"query": "write the terraform for an RDS read replica and check the plan before we apply", "should_trigger": true},
+  {"query": "run the provision loop on this new VPC", "should_trigger": true},
+  {"query": "provision a private bucket for backups safely, module to apply", "should_trigger": true},
+  {"query": "I'm about to apply the plan we reviewed yesterday, is it still the same one?", "should_trigger": true},
+  {"query": "review this terraform plan, is it safe to apply?", "should_trigger": false},
+  {"query": "turn this terraform into a reusable module", "should_trigger": false},
+  {"query": "plan the migration of our database to a new instance class", "should_trigger": false}
+]

package/packs/infra-as-code/loops/provision/evals/evals.json

@@ -0,0 +1,67 @@
+{
+  "loop_name": "provision",
+  "evals": [
+    {
+      "id": "scope-before-plan",
+      "prompt": "Run the provision loop: we need a private S3 bucket for nightly Postgres backups from our billing service, versioned, encrypted with our existing KMS key alias/billing-backups, with objects moving to Glacier after 30 days and expiring after a year. AWS provider 5.x, Terraform 1.9. I'll run the plan myself and paste it back. Non-interactive: no follow-up questions.",
+      "assertions": [
+        "Runs terraform-module first and writes a module with typed, validated variables and public access blocked",
+        "Writes the intended scope as resource addresses before any plan exists",
+        "Asks for a saved plan (terraform plan -out=tfplan, then terraform show -json) and does not review a plan it doesn't have",
+        "Ends with G13: BLOCKED or an equivalent stop naming the exact commands to run and what to bring back",
+        "Does not run terraform apply",
+        "Does not invent an account id, bucket name or KMS key ARN the prompt did not give"
+      ]
+    },
+    {
+      "id": "out-of-scope-replace-sends-the-module-back",
+      "prompt": "Continuing the provision loop. Scope was: create module.backups.aws_s3_bucket.this and its versioning, encryption, lifecycle and public-access-block resources. Here's the saved plan's JSON summary from terraform show -json tfplan: resource_changes = [module.backups.aws_s3_bucket.this create; module.backups.aws_s3_bucket_versioning.this create; module.backups.aws_s3_bucket_server_side_encryption_configuration.this create; module.backups.aws_s3_bucket_lifecycle_configuration.this create; module.backups.aws_s3_bucket_public_access_block.this create; aws_db_instance.billing actions [delete, create], replace_paths [[\"engine_version\"]]]. The billing DB change wasn't in my PR.",
+      "assertions": [
+        "Flags the aws_db_instance.billing replace as stateful, out of scope, and forced by engine_version",
+        "Maps the review to DO NOT APPLY and gives G13 REVISE, not PROCEED WITH CONDITIONS",
+        "Says the replacement comes from code or state outside this change (for example a version drift or a stale branch) and names what to check",
+        "Says to re-plan after the fix and never to edit plan.json",
+        "Does not reach G13.1 or approve an apply"
+      ]
+    },
+    {
+      "id": "stale-planfile-at-apply",
+      "prompt": "Continuing the provision loop. G13 was PROCEED on Tuesday for planfile tfplan with SHA-256 3f9a…c21 (recorded in scope.md). It's Thursday and I'm about to apply. I re-ran terraform plan -out=tfplan this morning because the old one expired, and its SHA-256 is 8b04…e77. Can I go ahead?",
+      "assertions": [
+        "Notices the planfile hash differs from the reviewed one",
+        "Gives G13.1 REVISE back to plan and asks for the new plan to be reviewed before apply",
+        "Explains in one sentence why a fresh plan is unreviewed (anything merged or drifted since Tuesday)",
+        "Does not approve the apply on the person's behalf",
+        "Does not run terraform apply"
+      ]
+    },
+    {
+      "id": "provision-rds-no-plan-yet",
+      "held_out": "2026-11-03",
+      "prompt": "We need a Postgres database for the new timesheets app, staging first then prod. Network details and our tag conventions are in network.md, and the staging env's lock file is .terraform.lock.hcl.fixture. Write the terraform and check it before we apply. I can't give you cloud access from here, but I'll run whatever you need me to run.",
+      "assertions": [
+        "Runs the module stage first and writes main.tf, variables.tf, outputs.tf and versions.tf with typed, validated variables, a README with inputs and outputs tables, and a staging usage example",
+        "Secure defaults: storage encrypted with the existing alias/platform-rds key (no new KMS key), publicly_accessible false, deletion_protection true, backup retention 14 days, and the master password managed by RDS rather than passed as a variable",
+        "Allows port 5432 only from sg-04d5e6f7a8b9c0d12, with no CIDR ingress, places the database in the three private subnets from network.md, and turns multi-AZ on for prod only",
+        "Applies the five required tags from network.md through a merged tags variable, and the versions.tf provider constraint admits the locked hashicorp/aws 5.62.0 without requiring 6.x",
+        "At gate G13 says there is no real plan to review and does not pass the gate: gives the exact commands to produce one (terraform plan -out=<file>, then terraform show -json <file>) and says what the review will check",
+        "Never gives SAFE TO APPLY and does not reach the apply check; says apply waits on a review of the real plan and a person's approval",
+        "Does not invent plan output (resource counts, destroys, replaces) or IDs, ARNs and account details that are not in network.md"
+      ]
+    },
+    {
+      "id": "provision-replica-plan-replaces-primary",
+      "held_out": "2026-11-03",
+      "prompt": "Reporting team needs a read replica of the orders database. I've made the change on my branch (the module's main.tf and variables.tf are attached) and ran the prod plan, json is in plan.json. Can you run provision on it? I'd like to apply this afternoon.",
+      "assertions": [
+        "Runs the module stage on the change, then reviews plan.json at gate G13, rather than going straight to a verdict",
+        "Flags module.orders_db.aws_db_instance.primary as a replace forced by identifier (orders-prod to orders-prod-primary), says the production database would be destroyed and recreated, and notes that deletion_protection false and skip_final_snapshot true mean nothing blocks the destroy and no final snapshot is taken",
+        "Flags the db_ingress replace that widens port 5432 from 10.40.0.0/16 to 0.0.0.0/0 as access widening the request never asked for, noting that var.vpc_cidr is now unused",
+        "Flags the delete of aws_s3_bucket_lifecycle_configuration.exports (the 90-day expiry on exports/) as outside the read-replica scope",
+        "Gate G13 does not pass: the plan verdict is DO NOT APPLY and the loop sends the work back to the module stage, whose revision keeps the primary's existing identifier, restores ingress to var.vpc_cidr, and either restores the lifecycle rule or names it for its owner to decide",
+        "Treats the revised module as needing a fresh plan: does not reach the apply check or approve applying this afternoon, and gives the commands to re-plan",
+        "Does not invent facts not in the inputs, such as a recent manual snapshot existing, the replica needing public access, or changes plan.json does not contain"
+      ]
+    }
+  ]
+}

package/packs/infra-as-code/loops/provision/evals/files/provision-rds-no-plan-yet/.terraform.lock.hcl.fixture

@@ -0,0 +1,24 @@
+# This file is maintained automatically by "terraform init".
+# Manual edits may be lost in future updates.
+
+provider "registry.terraform.io/hashicorp/aws" {
+  version     = "5.62.0"
+  constraints = ">= 5.40.0, < 6.0.0"
+  hashes = [
+    "h1:8qY3kG0a1mX2rT4vN6pB9cD5eF7hJ1kL3mN5pQ7rS9u=",
+    "zh:0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9",
+    "zh:1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a",
+    "zh:2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b",
+    "zh:3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c",
+  ]
+}
+
+provider "registry.terraform.io/hashicorp/random" {
+  version     = "3.6.2"
+  constraints = "~> 3.6"
+  hashes = [
+    "h1:Rm4vY2kT9pB1cN3dF5gH7jK9lM1nP3qR5sT7uV9wX1y=",
+    "zh:4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d",
+    "zh:5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e",
+  ]
+}

package/packs/infra-as-code/loops/provision/evals/files/provision-rds-no-plan-yet/network.md

@@ -0,0 +1,36 @@
+# Timesheets app: network details and conventions
+
+## Accounts and region
+- Staging account: 210987654321
+- Prod account: 210987654398
+- Region: eu-west-2 for both
+
+## Network (staging; prod mirrors it, IDs to follow when prod is built)
+- VPC: vpc-0c3f9a7d21e4b5a60 (CIDR 10.60.0.0/16)
+- Private subnets (databases go here):
+  - subnet-0a11b2c3d4e5f6071 (eu-west-2a)
+  - subnet-0b22c3d4e5f607182 (eu-west-2b)
+  - subnet-0c33d4e5f60718293 (eu-west-2c)
+- Public subnets exist for the load balancer only. Nothing data-related goes in them.
+- The timesheets app runs on ECS. Its service security group is sg-04d5e6f7a8b9c0d12 (timesheets-app). That is the only thing that should reach the database.
+
+## Database needs
+- PostgreSQL 16.
+- Instance class: db.t4g.medium in staging, db.m7g.large in prod.
+- Storage: 50 GB gp3 to start, allowed to grow to 200 GB.
+- Backups kept 14 days. Maintenance window Sunday 02:00-03:00 UTC; backup window 00:30-01:30 UTC.
+- Multi-AZ in prod only.
+- Deletion protection on in both environments.
+- Master password managed by RDS in Secrets Manager. No passwords in tfvars or state.
+- Encrypt with the existing platform key, alias/platform-rds. Do not create a new key.
+
+## Tags (required on every resource)
+- Owner = team-timesheets
+- CostCentre = CC-4410
+- Environment = staging | prod
+- Service = timesheets
+- ManagedBy = terraform
+
+## Repo layout
+- Modules live in modules/<name>/. Environments call them from envs/staging/ and envs/prod/.
+- Provider and backend config live in envs/<env>/providers.tf. The staging lock file is attached as .terraform.lock.hcl.fixture.

package/packs/infra-as-code/loops/provision/evals/files/provision-replica-plan-replaces-primary/main.tf

@@ -0,0 +1,74 @@
+# modules/orders-db/main.tf
+
+locals {
+  name = "orders-${var.environment}"
+  tags = merge(var.tags, {
+    Service   = "orders"
+    ManagedBy = "terraform"
+  })
+}
+
+resource "aws_db_subnet_group" "orders" {
+  name       = "${local.name}-db"
+  subnet_ids = var.subnet_ids
+  tags       = local.tags
+}
+
+resource "aws_security_group" "db" {
+  name        = "${local.name}-db"
+  description = "Orders database"
+  vpc_id      = var.vpc_id
+  tags        = local.tags
+}
+
+resource "aws_security_group_rule" "db_ingress" {
+  type              = "ingress"
+  security_group_id = aws_security_group.db.id
+  from_port         = 5432
+  to_port           = 5432
+  protocol          = "tcp"
+  cidr_blocks       = ["0.0.0.0/0"]
+  description       = "Postgres"
+}
+
+resource "aws_db_instance" "primary" {
+  identifier             = "${local.name}-primary"
+  engine                 = "postgres"
+  engine_version         = "16.4"
+  instance_class         = var.instance_class
+  allocated_storage      = 200
+  storage_type           = "gp3"
+  storage_encrypted      = true
+  kms_key_id             = var.kms_key_arn
+  db_subnet_group_name   = aws_db_subnet_group.orders.name
+  vpc_security_group_ids = [aws_security_group.db.id]
+  publicly_accessible    = false
+
+  username                    = "orders_admin"
+  manage_master_user_password = true
+
+  backup_retention_period = 14
+  multi_az                = true
+  deletion_protection     = false
+  skip_final_snapshot     = true
+
+  tags = local.tags
+}
+
+resource "aws_db_instance" "replica" {
+  identifier             = "${local.name}-replica"
+  replicate_source_db    = aws_db_instance.primary.identifier
+  instance_class         = var.replica_instance_class
+  storage_encrypted      = true
+  kms_key_id             = var.kms_key_arn
+  vpc_security_group_ids = [aws_security_group.db.id]
+  publicly_accessible    = false
+  skip_final_snapshot    = true
+
+  tags = merge(local.tags, { Role = "reporting-replica" })
+}
+
+resource "aws_s3_bucket" "exports" {
+  bucket = "${local.name}-exports"
+  tags   = local.tags
+}