cdk-github-runner-switch 0.2.1 → 0.3.0

package/README.md

@@ -42,10 +42,10 @@ flowchart LR
 |---|---|
 | Webhook (Lambda Function URL) | Checks the GitHub signature. Each queued job that asks for the AWS label gets one machine: Spot on every instance type × subnet first, then on-demand. |
 | Switch (every 30 min) | Reads the Actions minutes used this month and flips `RUNNER` to `aws` at the threshold (default 1,900 of 2,000). It flips back to `github` when the month resets. When it moves to AWS, it re-runs the runs that failed because the minutes ran out. |
-| Reconciliation (every 5 min) | Queued jobs without a machine get one. This covers a lost webhook and jobs that hit the machine cap. |
+| Reconciliation (every 5 min) | Queued jobs without a machine get one. This covers a lost webhook and jobs that hit the machine cap. A machine left without a job (its job was cancelled, or another runner took it) is terminated after 5 minutes. |
 | On-demand label | Jobs labeled `aws-ondemand` (for example, deploys) skip Spot, so an interruption never cuts a deploy in half. |
 | Runner image | Ubuntu 24.04 arm64 from Canonical with Docker, Node.js, Python, git, GitHub CLI, AWS CLI and the runner. EC2 Image Builder rebuilds it monthly. You can add cached container images and extra commands. |
-| Safety | Security group with no inbound rules, IMDSv2, encrypted disk, shutdown → terminate, a 2-hour guard against hung jobs. The function may only launch, tag and list instances. |
+| Safety | Security group with no inbound rules, IMDSv2, encrypted disk, shutdown → terminate, a 2-hour guard against hung jobs. The function may launch, tag and list instances, and terminate only the ones it launched (tag condition). |
 
 ## Quick start
 
@@ -98,6 +98,7 @@ To flip the switch by hand: `gh variable set RUNNER --body aws` (or `github`).
 | `maxInstances` | `6` | Machine cap. Jobs above it wait and are launched by the reconciliation. |
 | `switchVariable` | `RUNNER` | Repository variable that the construct flips. |
 | `maxJobDuration` | 2 hours | The machine shuts down after this, even if a job hangs. |
+| `idleGrace` | 5 min | A machine without a job older than this is terminated by the reconciliation. |
 | `rootVolumeSize` | `40` GiB | |
 | `nodeMajorVersion` | `24` | Node.js installed in the image. |
 | `cachedImages` | `[]` | Container images pulled into the AMI, for faster `services:`. |

package/README.pt-BR.md

@@ -23,10 +23,10 @@ Ninguém precisa lembrar de nada, a pipeline não trava no fim do mês, e você
 |---|---|
 | Webhook (Function URL da Lambda) | Confere a assinatura do GitHub. Cada job na fila que pede o rótulo da AWS ganha uma máquina: primeiro Spot, em cada tipo de máquina × sub-rede, depois sob demanda. |
 | Chave (a cada 30 min) | Lê os minutos do Actions usados no mês e vira `RUNNER` para `aws` no limite (padrão: 1.900 de 2.000). Volta para `github` quando o mês vira. Ao ir para a AWS, refaz as execuções que falharam por falta de minutos. |
-| Reconciliação (a cada 5 min) | Job na fila sem máquina ganha uma. Cobre aviso do GitHub que se perdeu e o teto de máquinas. |
+| Reconciliação (a cada 5 min) | Job na fila sem máquina ganha uma. Cobre aviso do GitHub que se perdeu e o teto de máquinas. Máquina que ficou sem job (o job foi cancelado ou outra máquina o pegou) é desligada depois de 5 minutos. |
 | Rótulo sob demanda | Jobs com `aws-ondemand` (ex.: deploy) não usam Spot, então uma interrupção nunca corta um deploy no meio. |
 | Imagem | Ubuntu 24.04 arm64 da Canonical com Docker, Node.js, Python, git, GitHub CLI, AWS CLI e o runner. O EC2 Image Builder refaz a imagem todo mês. Dá para incluir imagens de contêiner em cache e comandos extras. |
-| Segurança | Sem porta de entrada, IMDSv2, disco criptografado, desligar = encerrar, limite de 2 horas para job travado. A função só lança, marca e lista máquinas. |
+| Segurança | Sem porta de entrada, IMDSv2, disco criptografado, desligar = encerrar, limite de 2 horas para job travado. A função lança, marca e lista máquinas, e só desliga as que ela mesma lançou (condição pela etiqueta). |
 
 ## Começando
 

package/lambda/index.mjs

@@ -2,7 +2,8 @@
 //
 // 1. Webhook `workflow_job` from the GitHub App → one disposable EC2 machine (Spot first) with a single-use (JIT)
 //    runner registration for the queued job.
-// 2. Every few minutes (`reconcile`) → queued jobs that have no machine (lost webhook, cap reached) get one.
+// 2. Every few minutes (`reconcile`) → queued jobs that have no machine (lost webhook, cap reached) get one, and
+//    machines left without a job (job cancelled, or taken by another machine) are terminated after a grace period.
 // 3. Every half hour (`check-usage`) → flips the repository variable (default `RUNNER`) between `github` and `aws`
 //    according to the GitHub Actions minutes used this month, and re-runs the runs that failed for lack of minutes.
 //
@@ -30,6 +31,7 @@ export function config(env = process.env) {
     billing: env.BILLING === 'user' ? 'user' : 'organization',
     runnerGroupId: Number(env.RUNNER_GROUP_ID || 1),
     maxJobMinutes: Number(env.MAX_JOB_MINUTES || 120),
+    idleGraceMinutes: Number(env.IDLE_GRACE_MINUTES || 5),
     switchId: env.SWITCH_ID || 'default',
     repository: env.REPOSITORY || '',
   };
@@ -93,6 +95,38 @@ export function missingRunners({ queued, active, busy }, c = config()) {
   return out;
 }
 
+/**
+ * Machines without a job. A JIT runner is not tied to the job that launched it: when that job is cancelled, or another
+ * idle runner takes it first, the machine would wait until `maxJobMinutes`. Per label, the machines whose runner is not
+ * busy (idle, still offline, or not registered) beyond the queued jobs are surplus; only those launched more than
+ * `idleGraceMinutes` ago are terminated (a young machine is still booting or about to take a job), oldest first.
+ * Also returns offline registrations of our labels with no machine behind them (left by terminated machines).
+ *
+ * `instances`: `{ id, label, launchTime, runnerName?, jobId? }`; `runners`: GitHub runners (`id`, `name`, `status`, `busy`, `labels`).
+ */
+export function surplusRunners({ queued, instances, runners }, c = config(), now = new Date()) {
+  const ours = (r) => (r.labels ?? []).some((l) => [c.label, c.onDemandLabel].includes(l.name ?? l));
+  const runnerOf = (i) => runners.find((r) => (i.runnerName ? r.name === i.runnerName : i.jobId && r.name.startsWith(`aws-${i.jobId}-`)));
+  const terminate = [];
+  for (const label of [c.onDemandLabel, c.label]) {
+    const free = instances
+      .filter((i) => i.label === label)
+      .map((i) => ({ ...i, runner: runnerOf(i) }))
+      .filter((i) => !i.runner?.busy)
+      .sort((a, b) => new Date(a.launchTime) - new Date(b.launchTime));
+    let extra = free.length - (queued[label] ?? 0);
+    for (const i of free) {
+      if (extra <= 0) break;
+      if (now - new Date(i.launchTime) < c.idleGraceMinutes * 60_000) continue;
+      terminate.push({ instanceId: i.id, runnerId: i.runner?.id ?? null });
+      extra--;
+    }
+  }
+  const matched = new Set(instances.map(runnerOf).filter(Boolean).map((r) => r.id));
+  const orphans = runners.filter((r) => ours(r) && r.status === 'offline' && !r.busy && !matched.has(r.id)).map((r) => r.id);
+  return { terminate, orphans };
+}
+
 /** Boot script: runs exactly one job (JIT) and shuts down — the launch template turns shutdown into terminate. */
 export function userData(jitConfig, c = config()) {
   if (!/^[A-Za-z0-9+/=]+$/.test(jitConfig)) throw new Error('invalid JIT config');
@@ -160,20 +194,24 @@ async function ec2() {
   return { client: ec2Client, ...m };
 }
 
-async function activeByLabel(c) {
+async function instancesOf(c) {
   const { client, DescribeInstancesCommand } = await ec2();
   const r = await client.send(new DescribeInstancesCommand({
     Filters: [{ Name: 'tag:runner-switch', Values: [c.switchId] }, { Name: 'instance-state-name', Values: ['pending', 'running'] }],
   }));
-  const out = {};
-  for (const i of (r.Reservations ?? []).flatMap((x) => x.Instances ?? [])) {
-    const label = i.Tags?.find((t) => t.Key === 'runner-label')?.Value ?? c.label;
-    out[label] = (out[label] ?? 0) + 1;
-  }
-  return out;
+  return (r.Reservations ?? []).flatMap((x) => x.Instances ?? []).map((i) => {
+    const tag = (k) => i.Tags?.find((t) => t.Key === k)?.Value;
+    return { id: i.InstanceId, label: tag('runner-label') ?? c.label, launchTime: i.LaunchTime, runnerName: tag('runner-name'), jobId: tag('github-job') };
+  });
 }
 
-async function launch(c, jobId, onDemand, jit) {
+const countByLabel = (instances) => instances.reduce((out, i) => ({ ...out, [i.label]: (out[i.label] ?? 0) + 1 }), {});
+
+async function activeByLabel(c) {
+  return countByLabel(await instancesOf(c));
+}
+
+async function launch(c, jobId, onDemand, jit, runnerName) {
   const { SSMClient, GetParameterCommand } = await sdk.ssm();
   const ami = (await new SSMClient({}).send(new GetParameterCommand({ Name: process.env.AMI_PARAMETER }))).Parameter.Value;
   const { client, RunInstancesCommand } = await ec2();
@@ -191,7 +229,7 @@ async function launch(c, jobId, onDemand, jit) {
           ResourceType: 'instance',
           Tags: [
             { Key: 'Name', Value: `github-runner-${c.switchId}` }, { Key: 'runner-switch', Value: c.switchId }, { Key: 'github-job', Value: String(jobId) },
-            { Key: 'market', Value: p.spot ? 'spot' : 'on-demand' }, { Key: 'runner-label', Value: label },
+            { Key: 'market', Value: p.spot ? 'spot' : 'on-demand' }, { Key: 'runner-label', Value: label }, { Key: 'runner-name', Value: runnerName },
           ],
         }],
       }));
@@ -208,14 +246,15 @@ async function launch(c, jobId, onDemand, jit) {
 }
 
 async function launchFor(c, token, jobId, onDemand) {
+  const name = `aws-${jobId}-${Date.now().toString(36)}`;
   const jit = await gh(`/repos/${c.repository}/actions/runners/generate-jitconfig`, {
     token, method: 'POST',
     body: {
-      name: `aws-${jobId}-${Date.now().toString(36)}`, runner_group_id: c.runnerGroupId, work_folder: '_work',
+      name, runner_group_id: c.runnerGroupId, work_folder: '_work',
       labels: ['self-hosted', 'linux', 'arm64', onDemand ? c.onDemandLabel : c.label],
     },
   });
-  return launch(c, jobId, onDemand, jit.json.encoded_jit_config);
+  return launch(c, jobId, onDemand, jit.json.encoded_jit_config, name);
 }
 
 // ---------------------------------------------------------------- tasks
@@ -254,6 +293,25 @@ export async function reconcile(c = config()) {
     }
   }
   const runners = await gh(`/repos/${c.repository}/actions/runners?per_page=100`, { token });
+  const instances = await instancesOf(c);
+  const { terminate, orphans } = surplusRunners({ queued, instances, runners: runners.json?.runners ?? [] }, c);
+  const ended = new Set();
+  for (const t of terminate) {
+    // Remove the registration first: GitHub refuses it while the runner is busy, so a job taken meanwhile is safe.
+    if (t.runnerId) {
+      try {
+        await gh(`/repos/${c.repository}/actions/runners/${t.runnerId}`, { token, method: 'DELETE' });
+      } catch (e) {
+        console.log(JSON.stringify({ keep: t.instanceId, reason: e.message }));
+        continue;
+      }
+    }
+    const { client, TerminateInstancesCommand } = await ec2();
+    await client.send(new TerminateInstancesCommand({ InstanceIds: [t.instanceId] }));
+    ended.add(t.instanceId);
+  }
+  for (const id of orphans) await gh(`/repos/${c.repository}/actions/runners/${id}`, { token, method: 'DELETE' }).catch((e) => console.log(e.message));
+  if (ended.size || orphans.length) console.log(JSON.stringify({ surplus: { terminated: [...ended], orphans } }));
   const busy = {};
   for (const r of runners.json?.runners ?? []) {
     if (!r.busy) continue;
@@ -261,7 +319,7 @@ export async function reconcile(c = config()) {
     const l = names.includes(c.onDemandLabel) ? c.onDemandLabel : names.includes(c.label) ? c.label : null;
     if (l) busy[l] = (busy[l] ?? 0) + 1;
   }
-  const active = await activeByLabel(c);
+  const active = countByLabel(instances.filter((i) => !ended.has(i.id)));
   const missing = missingRunners({ queued, active, busy }, c);
   console.log(JSON.stringify({ reconcile: { queued, active, busy, missing } }));
   for (const [label, n] of Object.entries(missing)) {

package/lib/runner-switch.d.ts

@@ -35,6 +35,11 @@ export interface GitHubRunnerSwitchProps {
     readonly runnerGroupId?: number;
     /** Safety net: the machine shuts down after this long even if the job hangs. @default Duration.hours(2) */
     readonly maxJobDuration?: Duration;
+    /**
+     * A machine left without a job (its job was cancelled, or another runner took it) is terminated by the
+     * reconciliation once it is older than this. @default Duration.minutes(5)
+     */
+    readonly idleGrace?: Duration;
     /** Root disk in GiB. @default 40 */
     readonly rootVolumeSize?: number;
     /** Node.js major version installed in the image. @default 24 */

package/lib/runner-switch.js

@@ -179,6 +179,7 @@ class GitHubRunnerSwitch extends constructs_1.Construct {
                 SWITCH_VARIABLE: props.switchVariable ?? 'RUNNER',
                 RUNNER_GROUP_ID: String(props.runnerGroupId ?? 1),
                 MAX_JOB_MINUTES: String((props.maxJobDuration ?? aws_cdk_lib_1.Duration.hours(2)).toMinutes()),
+                IDLE_GRACE_MINUTES: String((props.idleGrace ?? aws_cdk_lib_1.Duration.minutes(5)).toMinutes()),
                 SWITCH_ID: name,
                 AMI_PARAMETER: this.amiParameterName,
                 LAUNCH_TEMPLATE_ID: template.launchTemplateId,
@@ -192,6 +193,11 @@ class GitHubRunnerSwitch extends constructs_1.Construct {
         fn.addToRolePolicy(new iam.PolicyStatement({ actions: ['ssm:GetParameter'], resources: [`arn:aws:ssm:${region}:${account}:parameter${this.amiParameterName}`] }));
         fn.addToRolePolicy(new iam.PolicyStatement({ actions: ['ec2:RunInstances', 'ec2:CreateTags', 'ec2:DescribeInstances'], resources: ['*'] }));
         fn.addToRolePolicy(new iam.PolicyStatement({ actions: ['iam:PassRole'], resources: [instanceRole.roleArn] }));
+        // Only its own machines (tagged at launch) can be terminated: the ones left without a job.
+        fn.addToRolePolicy(new iam.PolicyStatement({
+            actions: ['ec2:TerminateInstances'], resources: [`arn:aws:ec2:${region}:${account}:instance/*`],
+            conditions: { StringEquals: { 'aws:ResourceTag/runner-switch': name } },
+        }));
         this.webhookUrl = fn.addFunctionUrl({ authType: lambda.FunctionUrlAuthType.NONE }).url;
         new events.Rule(this, 'UsageCheck', {
             schedule: events.Schedule.rate(props.usageCheckInterval ?? aws_cdk_lib_1.Duration.minutes(30)),

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cdk-github-runner-switch",
-  "version": "0.2.1",
+  "version": "0.3.0",
   "description": "AWS CDK construct: GitHub Actions runners that switch from GitHub-hosted free minutes to disposable EC2 Spot (Graviton) machines when the monthly minutes run out — and back when the month resets.",
   "license": "Apache-2.0",
   "author": "uarch-solutions",