thinkwork-cli 0.13.2 → 1.0.0

package/dist/terraform/modules/app/deployment-control-plane/README.md

@@ -30,6 +30,270 @@ Full deploy/update plans fail closed when they would delete customer-domain
 resources or remove web CloudFront aliases unless
 `allowCustomerDomainRemoval=true` is set for an intentional reviewed retirement.
 
+## Running the runner inside the VPC (THINK-1082)
+
+The runner's schema step (`push_database_schema`) opens `psql` to the Aurora
+cluster. By default the CodeBuild project has no `vpc_config`, so that
+connection uses the cluster's public endpoint and depends on the world-open
+`0.0.0.0/0 -> 5432` ingress the `aurora-postgres` module seeds on the database
+security group. A customer that revokes that rule (deliberate lockdown) breaks
+every deploy at the psql step with `Connection timed out`, and Terraform never
+restores it (`ignore_changes = [ingress]`).
+
+`runner_in_vpc = true` (with `vpc_id`, `subnet_ids`, `db_security_group_id`)
+moves the build into the VPC's private subnets behind a dedicated security
+group (`thinkwork-<stage>-deployment-runner-sg`) and adds a standalone
+`aws_security_group_rule` granting that group tcp/5432 on the database security
+group — the same pattern the app modules use. The private subnets must route
+egress through a NAT gateway: a VPC-attached build has no public IP and still
+needs GitHub (release manifest), apt/pip, STS, S3, and Secrets Manager. The
+flag is ship-inert: `false` leaves the CodeBuild project byte-identical to
+today.
+
+### Turning it on for a stage
+
+There are two places the control plane can live, and the flag has to reach
+both the network (NAT, owned by the `thinkwork` module) and the CodeBuild
+project (owned by whichever root provisioned it):
+
+1. **The stage's `thinkwork` module input** — `deployment_runner_in_vpc = true`.
+   It keeps the NAT gateway on regardless of the deprecated `okf_wiki_*` flags
+   and, when this module is embedded (`enable_deployment_control_plane`, the
+   dev/greenfield path), passes the VPC, private subnets, and database security
+   group straight into it. For a greenfield root: set the
+   `deployment_runner_in_vpc` variable (`terraform.tfvars` or
+   `TF_VAR_deployment_runner_in_vpc=true`). For a controller-managed customer
+   stage the generated root reads it from the runner-secrets document:
+
+   ```bash
+   # /thinkwork/<stage>/deployment/runner-secrets — add the key, keep the rest
+   aws secretsmanager put-secret-value --secret-id /thinkwork/<stage>/deployment/runner-secrets \
+     --secret-string "$(aws secretsmanager get-secret-value --secret-id /thinkwork/<stage>/deployment/runner-secrets \
+       --query SecretString --output text | jq '. + {deploymentRunnerInVpc: true}')"
+   ```
+
+   (A `deploymentRunnerInVpc: true` key on the controller payload overrides
+   per-deploy; the secret is the durable home.)
+
+2. **The bootstrapped control plane** (customer accounts — the customer-update
+   root always applies with `enable_deployment_control_plane = false`, so the
+   customer deploy never touches its own CodeBuild project). Re-run the
+   bootstrap with the network named explicitly; it re-applies this module in
+   its own state (`deployment-control-plane/<stage>.tfstate`):
+
+   ```bash
+   thinkwork enterprise bootstrap --customer <slug> --stage <stage> \
+     --runner-in-vpc \
+     --runner-vpc-id vpc-... \
+     --runner-subnet-ids subnet-...,subnet-... \
+     --runner-db-security-group-id sg-...
+   ```
+
+   Use the VPC's **private** subnets (Name tag `thinkwork-<stage>-priv-<az>`)
+   and the database security group (Name tag `thinkwork-<stage>-db-sg`;
+   `module.database.db_security_group_id` in the stage's Terraform state).
+
+### Rollout: the first enable is a two-build story
+
+Order matters. Do step 1 first and let it deploy: it creates the NAT gateway
+(EIP + route on the private route tables) in the customer apply. Then do step 2.
+
+When the flag turns the embedded control plane on (step 1 alone on a
+dev/greenfield stage), the deploy that enables it updates the CodeBuild project
+during its own Terraform step — but that build is still executing in the OLD,
+non-VPC project. Its psql step therefore still needs the database reachable
+over the public endpoint one last time. If the world-open rule is already gone,
+that first run fails at the psql step after a successful apply; the RETRY runs
+inside the VPC and succeeds. Do not read the first failure as the change not
+working. The same holds for a customer stage if step 2 is applied while a
+deploy is in flight.
+
+What the first enable creates: the NAT gateway + EIP and private default
+routes (if not already present), `aws_security_group.runner`,
+`aws_security_group_rule.db_from_runner` (tcp/5432 on the database security
+group from the runner group), and `vpc_config` on the CodeBuild project. The
+CodeBuild role already holds `ec2:*`, which covers the ENI permissions a
+VPC-attached build needs; nothing else in IAM changes.
+
+## Closing the database port (THINK-1082)
+
+Every database-connected Lambda of the agent — the 100+ `thinkwork-<stage>-api-*`
+handlers (each carries `DATABASE_URL`), `compliance-anchor`,
+`compliance-export-runner`, `agentcore-admin`, the `agentcore-pi` runtime and
+the capability broker — reaches Aurora the same way the runner's `psql` does:
+over the cluster's public endpoint, through the world-open `0.0.0.0/0 -> 5432`
+rule the `aurora-postgres` module seeds. Revoking that rule on a running stage
+before those clients are inside the VPC takes the entire app down
+(`auth-me` -> DB timeout -> 401 "No tenant assigned"; two hours on a customer
+stage on 2026-08-25).
+
+Four stage-level inputs close the port safely. All are ship-inert (defaults =
+today) and live in the same homes as `deploymentRunnerInVpc`: the
+runner-secrets document `/thinkwork/<stage>/deployment/runner-secrets` is the
+durable home, a controller-payload key overrides per deploy, and a
+greenfield root sets the `thinkwork` module variable directly (`dbAccess` is
+runner-only — it has no module variable).
+
+| Input (module / runner-secrets key)                    | Default         | Effect when changed                                                                                                           |
+| ------------------------------------------------------ | --------------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| `api_lambdas_in_vpc` / `apiLambdasInVpc`               | `false`         | Every DB-connected Lambda attaches to the private subnets behind `thinkwork-<stage>-api-lambda-sg`; the DB SG admits it on 5432; NAT on. |
+| — / `dbAccess`                                         | `psql`          | `data-api`: the runner's schema steps (probes, migrations, ledger, seeds, first-admin bootstrap) run over the RDS Data API — HTTPS + IAM, no network path. The runner stays outside the VPC. |
+| `db_public_ingress_cidrs` / `dbPublicIngressCidrs`     | `["0.0.0.0/0"]` | CIDRs seeded as 5432 ingress on a **new** DB SG. `[]` = none. Does **not** revoke a live rule (the group ignores ingress drift). |
+| `db_publicly_accessible` / `dbPubliclyAccessible`      | `true`          | Public IP on the cluster instance. `false` removes it (immediate modify).                                                       |
+| `deployment_runner_in_vpc` / `deploymentRunnerInVpc`   | `false`         | The runner CodeBuild attaches to the VPC (section above). Not required once `dbAccess` is `data-api`; kept as an alternative. |
+
+### Order for a running stage
+
+Do the steps in this order and verify between them; each is its own deploy.
+
+1. **Lambdas into the VPC.** Set `apiLambdasInVpc: true` (runner-secrets, or
+   `api_lambdas_in_vpc = true` on a greenfield root) and deploy. The apply
+   creates the NAT gateway (if not already present), the api-lambda security
+   group, the `db_from_api_lambda` rule, and re-configures every function with
+   `vpc_config` (expect a few minutes of Lambda updates; ENI creation is
+   per-subnet, one-time). Verify the app: sign in, `GET /auth/me` returns the
+   tenant, a chat turn completes (Pi and the broker are on the same lane).
+   The Lambdas now reach the database privately; the public rule is no longer
+   load-bearing for them.
+
+   ```bash
+   aws secretsmanager put-secret-value --secret-id /thinkwork/<stage>/deployment/runner-secrets \
+     --secret-string "$(aws secretsmanager get-secret-value --secret-id /thinkwork/<stage>/deployment/runner-secrets \
+       --query SecretString --output text | jq '. + {apiLambdasInVpc: true}')"
+   ```
+
+2. **Runner onto the Data API.** Set `dbAccess: "data-api"` the same way and
+   deploy once. That deploy's schema step — the `to_regclass` probes, the
+   migration ledger, every pending migration file, the platform seed and the
+   first-admin bootstrap — runs over the RDS Data API (`rds-data:*` on the
+   cluster, granted to the CodeBuild role by this module; the cluster has
+   `enable_http_endpoint = true` from the `aurora-postgres` module). The
+   runner never opens a TCP connection to the database and stays outside the
+   VPC. Verify in the build log:
+
+   ```
+   [runner] database access: data-api
+   ```
+
+   and that the migration lines (`[migrations] applying …`) completed. The
+   public path is now used by nothing.
+
+   ```bash
+   aws secretsmanager put-secret-value --secret-id /thinkwork/<stage>/deployment/runner-secrets \
+     --secret-string "$(aws secretsmanager get-secret-value --secret-id /thinkwork/<stage>/deployment/runner-secrets \
+       --query SecretString --output text | jq '. + {dbAccess: "data-api"}')"
+   ```
+
+   How the lane bridges what `psql` did: scripts are split into single
+   statements (dollar-quote, quote and comment aware; drizzle
+   `--> statement-breakpoint` markers count as boundaries), `:'stage'`-style
+   variables are interpolated, and the psql meta-commands the migration files
+   use (`\set`, `\echo`, `\if :{?name}` / `\else` / `\endif`, `\quit`) are
+   interpreted — any other meta-command is refused by name. Every script runs
+   inside one Data API transaction (its own `BEGIN`/`COMMIT` map onto it;
+   `CREATE INDEX CONCURRENTLY` commits and runs standalone) so statements
+   serialise on one connection and a DDL statement that outlives the API's
+   45 s call window (`continueAfterTimeout`) can never race the next one.
+   The first failing statement rolls the script back and fails the deploy
+   before the file is recorded in `platform_schema_migrations` — the same
+   `ON_ERROR_STOP` contract as before. A statement over the Data API's 64 KB
+   limit (0230's generated RLS block) is shipped in pieces to a temp table on
+   the transaction's connection and executed there.
+
+3. **Close the port.** Set `dbPublicIngressCidrs: []` and
+   `dbPubliclyAccessible: false` and deploy. Terraform stops seeding the
+   world-open rule and drops the public IP, but because the DB security group
+   ignores ingress drift the live `0.0.0.0/0` rule is still there — revoke it
+   by hand, once:
+
+   ```bash
+   aws ec2 revoke-security-group-ingress --group-id <thinkwork-<stage>-db-sg id> \
+     --protocol tcp --port 5432 --cidr 0.0.0.0/0
+   ```
+
+   Then verify the app again (auth-me, a chat turn, a brief) **and** run one
+   more deploy so the runner's Data API schema step is proven with the port
+   closed.
+
+Notes:
+
+- The runner-in-VPC path (`deploymentRunnerInVpc`, section above) remains
+  available but is not required: with `dbAccess: "data-api"` the runner has
+  no database network path to protect. (Its first enable hung in Terraform's
+  plan phase on 2026-08-25, which is why the Data API lane exists.)
+- Every run logs its lane once — `[runner] database access: psql|data-api` —
+  right after the runner files are written; check it before trusting a
+  "closed port" deploy.
+- The RDS Data API (`enable_http_endpoint`) needs IAM, not a network path, so
+  operator SQL (`aws rds-data execute-statement`), the Pi session store and
+  every `db_cluster_arn` consumer keep working at every step.
+- ECS workloads (Twenty, n8n) already attach standalone
+  `aws_security_group_rule` grants for their own task security groups and are
+  unaffected by the close.
+- VPC-attached Lambdas route all internet egress (Bedrock, Anthropic, Slack,
+  GitHub, model providers, AWS APIs without an interface endpoint) through the
+  NAT gateway — NAT data-processing charges apply to that traffic. Interface
+  endpoints for secretsmanager, sts, logs, rds-data, athena, bedrock-agentcore
+  and xray plus the S3 gateway keep that in-VPC.
+- Rolling back is the reverse order: re-add the CIDR rule by hand (or set
+  `dbPublicIngressCidrs` back and recreate), set `dbAccess` back to `psql`,
+  then flip `apiLambdasInVpc` off.
+
+## Saved-plan approval
+
+High-risk foundation changes can use a two-execution controller protocol. An
+`action=plan` execution must set `operation.plan=true`,
+`operation.apply=false`, and a `targetAction`. It uploads the opaque saved plan,
+the rendered JSON plan, and a content-addressed approval descriptor. A later
+`action=apply` execution must set `operation.plan=false`,
+`operation.apply=true`, and reference that exact descriptor. The runner refuses
+the apply if the release pin, deployment configuration, Terraform variables,
+state lineage/serial/content, saved-plan digest, or rendered JSON digest differs.
+It applies the accepted plan directly and never replans.
+
+Existing deployment and managed-application callers retain their established
+behavior while their product approval surfaces migrate to this protocol. The
+separate saved-plan path is mandatory for incident recovery and other explicitly
+guarded operations.
+
+## TEI `.380` proof-plane recovery
+
+The runner contains a one-time, exact-scope recovery contract for the interrupted
+TEI `.380` apply. It is bound to the observed account, region, state object,
+lineage, serial, state hash, stale lock ID/digest, rejected plan hash, original
+Twenty OAuth secret, and original turn-assertion KMS key. The managed `recover`
+action:
+
+1. proves it is the sole deployment execution and build;
+2. restores the original secret without reading its value;
+3. cancels deletion and re-enables the original KMS key;
+4. invokes Terraform's backend-aware `force-unlock` for only the bound stale
+   lock (never a direct DynamoDB delete); and
+5. proves the state object and digest are unchanged while writing redacted
+   evidence.
+
+The subsequent proof-enabled plan accepts that evidence and declaratively
+imports the recovered secret and KMS key. It must use the saved-plan approval
+protocol and pass a zero-delete review before any apply.
+
+## Node.js 22 control-runtime transition
+
+The AgentCore control SDK requires Node.js 20 or newer, and the ThinkWork
+toolchain requires Node.js 22 or newer. New and reconciled CodeBuild projects
+therefore select Node.js 22 in the checked-in buildspec, and release CI executes
+every packaged control-runtime entrypoint under that exact major.
+
+An established deployment whose live CodeBuild project still embeds the older
+Node.js 18 buildspec upgrades in 2 managed stages. First, the existing Python
+runner loads the ESM-bounded release under Node.js 18 only long enough to produce
+and apply an independently accepted Terraform plan. That transition must perform
+no AgentCore SDK API operation; the plan may only update CodeBuild to Node.js 22
+in place alongside the intended application changes, with zero delete or replace
+actions. Then read back the live buildspec, refresh the runner through the
+release-remediation API using the manifest's standalone runner asset and digest,
+and rerun the controller under Node.js 22. Do not claim Node.js 18 support from
+this bounded transition. Subsequent updates are entirely Terraform-owned.
+
 Platform deploys stay on the root Terraform state key:
 `thinkwork/<stage>/terraform.tfstate`.
 

package/dist/terraform/modules/app/deployment-control-plane/buildspec.yml

@@ -2,10 +2,12 @@ version: 0.2
 
 phases:
   install:
+    runtime-versions:
+      nodejs: 22
     commands:
       - |
         if ! command -v terraform >/dev/null 2>&1; then
-          curl -fsSLo /tmp/terraform.zip https://releases.hashicorp.com/terraform/1.8.5/terraform_1.8.5_linux_amd64.zip
+          curl -fsSLo /tmp/terraform.zip https://releases.hashicorp.com/terraform/1.9.8/terraform_1.9.8_linux_amd64.zip
           unzip -oq /tmp/terraform.zip -d /usr/local/bin
         fi
         if ! command -v psql >/dev/null 2>&1; then
@@ -17,6 +19,7 @@ phases:
           fi
         fi
       - terraform version
+      - node --version
       - psql --version
   build:
     commands:

package/dist/terraform/modules/app/deployment-control-plane/main.tf

@@ -27,6 +27,13 @@ locals {
   state_machine_log_group  = "/aws/vendedlogs/states/${local.state_machine_name}"
   terraform_module_is_git  = startswith(var.terraform_module_source, "git::") || startswith(var.terraform_module_source, "github.com/")
   terraform_module_version = var.terraform_module_version != "" ? var.terraform_module_version : (local.terraform_module_is_git ? "" : trimprefix(var.release_version, "v"))
+  runner_sg_name           = "${local.codebuild_project_name}-sg"
+  # runner_in_vpc needs every network input; checked as a precondition on the
+  # CodeBuild project because cross-variable validation is not available at
+  # the repo's Terraform floor (1.8.5).
+  runner_vpc_inputs_complete = !var.runner_in_vpc || (
+    var.vpc_id != "" && length(var.subnet_ids) > 0 && var.db_security_group_id != ""
+  )
 
   deployment_parameters = {
     for key, value in {
@@ -297,6 +304,25 @@ resource "aws_iam_role_policy" "codebuild" {
           ["arn:aws:secretsmanager:${var.region}:${var.account_id}:secret:thinkwork/${var.stage}/deployment/*"]
         )
       },
+      {
+        # THINK-1082: the runner's database steps over the RDS Data API
+        # (runner-secrets `dbAccess: "data-api"`) — HTTPS + IAM instead of a
+        # psql connection through the cluster's public endpoint, so the
+        # world-open 5432 rule can close with the runner still outside the
+        # VPC. Ungated: harmless while dbAccess is psql. The credentials
+        # secret the Data API signs with (graphql_db_secret_arn) is already
+        # readable through the secretsmanager:* grant below — the same secret
+        # `psql` reads today.
+        Effect = "Allow"
+        Action = [
+          "rds-data:BatchExecuteStatement",
+          "rds-data:BeginTransaction",
+          "rds-data:CommitTransaction",
+          "rds-data:ExecuteStatement",
+          "rds-data:RollbackTransaction",
+        ]
+        Resource = "arn:aws:rds:${var.region}:${var.account_id}:cluster:thinkwork-${var.stage}-db"
+      },
       {
         Effect = "Allow"
         Action = [
@@ -345,6 +371,51 @@ resource "aws_iam_role_policy" "codebuild" {
   })
 }
 
+################################################################################
+# Runner-in-VPC network (THINK-1082)
+#
+# Off by default (ship-inert). When on, the build attaches to the VPC's private
+# subnets with this security group and the database security group grants it
+# tcp/5432 — the same standalone-rule pattern the app modules use, so the
+# grant survives the Aurora module's `ignore_changes = [ingress]`. The
+# CodeBuild role already carries ec2:* on Resource "*", which covers the ENI
+# permissions a VPC-attached build needs (CreateNetworkInterface,
+# CreateNetworkInterfacePermission, DeleteNetworkInterface, Describe*).
+################################################################################
+
+resource "aws_security_group" "runner" {
+  count = var.runner_in_vpc ? 1 : 0
+
+  name        = local.runner_sg_name
+  description = "ThinkWork deployment runner (CodeBuild in VPC) for ${var.stage}."
+  vpc_id      = var.vpc_id
+
+  egress {
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+
+  tags = {
+    Name    = local.runner_sg_name
+    Stage   = var.stage
+    Purpose = "deployment-runner"
+  }
+}
+
+resource "aws_security_group_rule" "db_from_runner" {
+  count = var.runner_in_vpc ? 1 : 0
+
+  type                     = "ingress"
+  description              = "ThinkWork deployment runner (VPC)"
+  from_port                = 5432
+  to_port                  = 5432
+  protocol                 = "tcp"
+  security_group_id        = var.db_security_group_id
+  source_security_group_id = aws_security_group.runner[0].id
+}
+
 resource "aws_codebuild_project" "runner" {
   name          = local.codebuild_project_name
   description   = "ThinkWork deployment runner for ${var.stage}."
@@ -470,6 +541,24 @@ resource "aws_codebuild_project" "runner" {
     buildspec = file("${path.module}/buildspec.yml")
   }
 
+  # Absent entirely when runner_in_vpc is false, so the default project is
+  # byte-identical to the pre-THINK-1082 shape.
+  dynamic "vpc_config" {
+    for_each = var.runner_in_vpc ? [1] : []
+    content {
+      vpc_id             = var.vpc_id
+      subnets            = var.subnet_ids
+      security_group_ids = [aws_security_group.runner[0].id]
+    }
+  }
+
+  lifecycle {
+    precondition {
+      condition     = local.runner_vpc_inputs_complete
+      error_message = "runner_in_vpc requires non-empty vpc_id, subnet_ids, and db_security_group_id."
+    }
+  }
+
   tags = {
     Name    = local.codebuild_project_name
     Stage   = var.stage
@@ -574,6 +663,11 @@ resource "aws_sfn_state_machine" "deployment" {
               Type      = "PLAINTEXT"
               "Value.$" = "$.sessionId"
             },
+            {
+              Name      = "THINKWORK_DEPLOYMENT_EXECUTION_ARN"
+              Type      = "PLAINTEXT"
+              "Value.$" = "$$.Execution.Id"
+            },
             {
               Name      = "THINKWORK_DEPLOYMENT_INPUT"
               Type      = "PLAINTEXT"

package/dist/terraform/modules/app/deployment-control-plane/outputs.tf

@@ -52,3 +52,8 @@ output "secret_arns" {
   description = "Placeholder deployment secret ARNs keyed by purpose."
   value       = { for key, secret in aws_secretsmanager_secret.deployment : key => secret.arn }
 }
+
+output "runner_security_group_id" {
+  description = "Security group attached to the deployment runner when runner_in_vpc is true; null otherwise."
+  value       = var.runner_in_vpc ? aws_security_group.runner[0].id : null
+}