@evident-ai/runner-cdk 3.4.1-dev.cb56b3f → 3.4.1-dev.e91ca26

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -11,7 +11,7 @@ Reusable AWS CDK constructs for running an [Evident](https://evident.run) agent
11
11
  - **`EvidentMicrovmConstruct`** — a per-session AWS Lambda MicroVM that boots on demand
12
12
  and suspends between messages. Installing the `@dev` tag of this package is all you
13
13
  need: the image build context ships inside it (see [The MicroVM image](#the-microvm-image)
14
- below), so there is no checkout of `sroze/evident` involved. See
14
+ below), so there is no checkout of the source repository involved. See
15
15
  [the AWS runner doc](https://evident.run/docs/aws-runner) for the full picture of both
16
16
  strategies.
17
17
 
@@ -76,8 +76,8 @@ agent.selfStopRoleArn;
76
76
  agent.replicaPrefix;
77
77
  ```
78
78
 
79
- For a complete working example, see `infrastructure/evident-runner/` in this repo — it is
80
- the construct's own dogfooding consumer.
79
+ For a complete working example, see [`GETTING-STARTED.md`](./GETTING-STARTED.md) in this
80
+ package — it is the construct's own dogfooding consumer.
81
81
 
82
82
  ## Props
83
83
 
@@ -121,10 +121,11 @@ context, which `cdk deploy` zips and uploads. The Lambda MicroVM service then bu
121
121
  image in *your* account, from a base image ARN you discover there — so, unlike the Fargate
122
122
  strategy, there is no registry image to pull.
123
123
 
124
- The context has two halves, and this package ships the half that is ours: the Dockerfile,
125
- the per-phase hook scripts and the bundled hook server, published inside the tarball at
126
- `dist/microvm-image-context/`. The other half is **your** repository, which is baked in as
127
- the agent's workspace. `stageMicrovmImageContext()` puts the two together:
124
+ The context has three parts: this package supplies the Dockerfile, the per-phase hook
125
+ scripts and the bundled hook server, published inside the tarball at
126
+ `dist/microvm-image-context/`; **your** repository is baked in as the agent's workspace;
127
+ and an optional overlay supplies deployment-specific build steps. `stageMicrovmImageContext()`
128
+ puts them together:
128
129
 
129
130
  ```ts
130
131
  import path from 'node:path';
@@ -143,6 +144,8 @@ new EvidentMicrovmConstruct(this, 'Runner', {
143
144
  // token the agent pushes with arrives per-session, never baked in.
144
145
  originUrl: 'https://github.com/acme/widgets.git',
145
146
  destination: path.join(__dirname, '..', 'build', 'image'),
147
+ // Optional deployment-specific installs and build warm-up.
148
+ overlayDir: '/path/to/microvm-overlay',
146
149
  }),
147
150
  // Must match the published image these were baked into, so export them
148
151
  // rather than restating the numbers.
@@ -156,6 +159,11 @@ new EvidentMicrovmConstruct(this, 'Runner', {
156
159
  });
157
160
  ```
158
161
 
162
+ Pass `overlayDir` to add deployment-specific build steps to the staged image. The
163
+ directory provides `setup-root` for root-owned installs and `setup-workspace` for
164
+ workspace build warm-up after dependency installation. See the [MicroVM image
165
+ README](../../runner/docker-images/microvm/README.md) for the full overlay contract.
166
+
159
167
  Your repo does not have to be a pnpm workspace. If it commits a `pnpm-lock.yaml` the image
160
168
  pre-installs dependencies at build time (a faster first boot); if not, that step is skipped
161
169
  and the agent installs on first use.
@@ -164,16 +172,21 @@ and the agent installs on first use.
164
172
  | --- | --- |
165
173
  | `runnerSecret` | No GitHub or MCP credentials are exported at `/run`. When supplied, `/run` reads the JSON secret and exports every non-empty value whose key is a valid environment-variable name. |
166
174
  | `runnerOpencodeConfigPath` | OpenCode uses the baked project configuration. Relative paths resolve from the workspace; absolute paths resolve in the image. |
175
+ | `extraImageEnvironment` | Non-secret values baked into every VM launched from this image version; every VM can read them, so never put secrets here. |
167
176
  | `gitUserName` / `gitUserEmail` | The hook uses the Evident bot defaults for git identity. |
168
177
  | `evidentApiUrl` / `evidentRunnerId` | No deploy-time report of the published image version — a byte-identical template to not having these props at all. Both non-secret (a URL, a public pool UUID) and both required together: set them if you have an Evident runner id configured for this provisioner, so each deploy that publishes a new image version tells Evident about it (a CloudFormation custom resource, HMAC-signed with `doorbellSecret`). A failure to report never fails your deploy — it warns and Evident reads that provisioner as *unknown*, never as up to date or stale. |
169
178
 
179
+ `extraImageEnvironment` is the runtime counterpart to an overlay. Use it when an
180
+ overlay-installed component needs an environment variable after the VM boots. The
181
+ values are baked into every VM launched from that image version and are readable by
182
+ all of them, so never put a secret in this prop.
183
+
170
184
  ## Status / limitations
171
185
 
172
186
  - **Published to npm** as `@evident-ai/runner-cdk` (MIT). Install the `@dev` tag —
173
187
  `npm install @evident-ai/runner-cdk@dev` (or `pnpm add`/`yarn add`); no checkout of
174
188
  this repo is required. It builds to a self-contained `dist/` (`pnpm --filter
175
189
  @evident-ai/runner-cdk build`) with no `workspace:`/`@evident/*` runtime dependency.
176
- This repo's own consumer (`infrastructure/evident-runner`) still consumes it as a
177
- `workspace:*` dependency for dogfooding.
190
+ Evident's own consumer still consumes it as a `workspace:*` dependency for dogfooding.
178
191
  - **AWS/ECS-specific.** Non-AWS clouds are an open question on the epic, not a supported
179
192
  path.
@@ -49854,7 +49854,8 @@ function parseDoorbell(rawBody) {
49854
49854
  occurred_at: occurredAt,
49855
49855
  microvm_id: microvmId,
49856
49856
  run_payload: runPayload,
49857
- shape
49857
+ shape,
49858
+ recreate_on_outdated_image: recreateOnOutdatedImage
49858
49859
  } = body;
49859
49860
  if (type !== SUSPEND_EVENT_TYPE && type !== SHAPES_EVENT_TYPE && !isWakeType(type)) {
49860
49861
  return { ok: false, reason: DOORBELL_REJECTION.unsupportedType };
@@ -49885,10 +49886,14 @@ function parseDoorbell(rawBody) {
49885
49886
  if (Buffer.byteLength(runHookPayload, "utf8") > RUN_HOOK_PAYLOAD_MAX_BYTES) {
49886
49887
  return { ok: false, reason: DOORBELL_REJECTION.runPayloadTooLarge };
49887
49888
  }
49888
- return {
49889
- ok: true,
49890
- doorbell: shape === void 0 ? { ...fields, type, runHookPayload } : { ...fields, type, shape, runHookPayload }
49891
- };
49889
+ const doorbell = { ...fields, type, runHookPayload };
49890
+ if (shape !== void 0) {
49891
+ doorbell.shape = shape;
49892
+ }
49893
+ if (recreateOnOutdatedImage === true) {
49894
+ doorbell.recreateOnOutdatedImage = true;
49895
+ }
49896
+ return { ok: true, doorbell };
49892
49897
  }
49893
49898
 
49894
49899
  // src/microvm/controller/throttle-retry.ts
@@ -50128,7 +50133,7 @@ async function handleWakeDoorbell(doorbell, shape, microvm, timing) {
50128
50133
  imageVersion: described.imageVersion
50129
50134
  });
50130
50135
  case "SUSPENDED":
50131
- if (await shouldRecreateForNewerImage(doorbell, shape, described.imageVersion, microvm)) {
50136
+ if (doorbell.recreateOnOutdatedImage === true && await shouldRecreateForNewerImage(doorbell, shape, described.imageVersion, microvm)) {
50132
50137
  return runMicrovm(
50133
50138
  doorbell,
50134
50139
  shape,
@@ -17,7 +17,7 @@ export type EvidentScaleToZeroConstructProps = {
17
17
  * (`agents/<id>`) and is the single expected `agent_id` the waker checks.
18
18
  */
19
19
  evidentAgentId: string;
20
- /** GitHub `owner/repo` the agent clones and works on (e.g. `sroze/evident`). */
20
+ /** GitHub `owner/repo` the agent clones and works on (e.g. `my-org/my-repo`). */
21
21
  gitRepo: string;
22
22
  /** Branch to clone. Defaults to `main` when omitted. */
23
23
  gitBranch?: string;
@@ -168,9 +168,8 @@ class EvidentScaleToZeroConstruct extends constructs_1.Construct {
168
168
  cluster,
169
169
  taskDefinition,
170
170
  serviceName,
171
- // Starts at 1; the runner self-scales to 0 on idle. A deploy resets this to
172
- // 1 (CFN), waking a sleeping agent that then re-naps (benign).
173
- desiredCount: 1,
171
+ // Deliberately omit DesiredCount: CloudFormation leaves the existing service's
172
+ // self-stop/waker count alone; either 1 or 0 would re-assert it on every deploy.
174
173
  enableExecuteCommand: true,
175
174
  assignPublicIp: true,
176
175
  vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
@@ -55,6 +55,12 @@ export interface EvidentMicrovmConstructProps {
55
55
  readonly gitUserName?: string;
56
56
  /** Optional git email the `/run` hook configures for agent commits. */
57
57
  readonly gitUserEmail?: string;
58
+ /**
59
+ * Non-secret values baked into every VM launched from this image version,
60
+ * typically values a deployment's overlay needs at runtime. Every VM can
61
+ * read them, so this must never contain a secret.
62
+ */
63
+ readonly extraImageEnvironment?: Record<string, string>;
58
64
  /**
59
65
  * TCP port the image's hook server listens on, baked into both the image
60
66
  * (`HOOKS_PORT` env var) and the `MicrovmImage`'s `hooks.port` — a mismatch
@@ -141,6 +141,7 @@ class EvidentMicrovmConstruct extends constructs_1.Construct {
141
141
  : {}),
142
142
  ...(props.gitUserName ? { GIT_USER_NAME: props.gitUserName } : {}),
143
143
  ...(props.gitUserEmail ? { GIT_USER_EMAIL: props.gitUserEmail } : {}),
144
+ ...props.extraImageEnvironment,
144
145
  };
145
146
  // ...and the hooks configuration every shape's image is built with.
146
147
  const imageHooks = {
@@ -40,6 +40,11 @@ export type WakeDoorbell = DoorbellFields & {
40
40
  * shape" — the controller resolves it against the shape catalogue.
41
41
  */
42
42
  shape?: string;
43
+ /**
44
+ * Set only when the body carries the literal `true`. When absent, the
45
+ * suspended VM is always resumed because recreating it destroys its filesystem.
46
+ */
47
+ recreateOnOutdatedImage?: boolean;
43
48
  };
44
49
  export type Doorbell = WakeDoorbell | (DoorbellFields & {
45
50
  type: typeof SUSPEND_EVENT_TYPE;
@@ -63,7 +63,7 @@ function parseDoorbell(rawBody) {
63
63
  if (typeof body !== 'object' || body === null || Array.isArray(body)) {
64
64
  return { ok: false, reason: exports.DOORBELL_REJECTION.invalidJson };
65
65
  }
66
- const { type, runner_id: runnerId, occurred_at: occurredAt, microvm_id: microvmId, run_payload: runPayload, shape, } = body;
66
+ const { type, runner_id: runnerId, occurred_at: occurredAt, microvm_id: microvmId, run_payload: runPayload, shape, recreate_on_outdated_image: recreateOnOutdatedImage, } = body;
67
67
  if (type !== SUSPEND_EVENT_TYPE && type !== exports.SHAPES_EVENT_TYPE && !isWakeType(type)) {
68
68
  return { ok: false, reason: exports.DOORBELL_REJECTION.unsupportedType };
69
69
  }
@@ -95,13 +95,14 @@ function parseDoorbell(rawBody) {
95
95
  if (Buffer.byteLength(runHookPayload, 'utf8') > constants_1.RUN_HOOK_PAYLOAD_MAX_BYTES) {
96
96
  return { ok: false, reason: exports.DOORBELL_REJECTION.runPayloadTooLarge };
97
97
  }
98
- // Omit `shape` entirely when absent, rather than setting it to `undefined`,
99
- // so a body with no shape parses to exactly the same object as before this
100
- // field existed.
101
- return {
102
- ok: true,
103
- doorbell: shape === undefined
104
- ? { ...fields, type, runHookPayload }
105
- : { ...fields, type, shape, runHookPayload },
106
- };
98
+ // Omit optional fields entirely when absent, rather than setting them to
99
+ // `undefined`, so older bodies keep the same parsed shape.
100
+ const doorbell = { ...fields, type, runHookPayload };
101
+ if (shape !== undefined) {
102
+ doorbell.shape = shape;
103
+ }
104
+ if (recreateOnOutdatedImage === true) {
105
+ doorbell.recreateOnOutdatedImage = true;
106
+ }
107
+ return { ok: true, doorbell };
107
108
  }
@@ -129,9 +129,8 @@ function compareImageVersions(a, b) {
129
129
  return 0;
130
130
  }
131
131
  /**
132
- * Whether the SUSPENDED VM must be thrown away and recreated because a newer
133
- * image has been published since it booted — a resume would otherwise keep
134
- * that VM on its old baked-in hooks and repo checkout for up to 8 hours.
132
+ * Whether an opted-in doorbell should throw away and recreate the SUSPENDED VM
133
+ * because a newer image has been published since it booted.
135
134
  *
136
135
  * FAIL-SAFE, one direction only: recreating destroys the VM's filesystem
137
136
  * (only the credentials and `opencode.db` in the object store survive), so
@@ -287,11 +286,12 @@ async function handleWakeDoorbell(doorbell, shape, microvm, timing) {
287
286
  imageVersion: described.imageVersion,
288
287
  });
289
288
  case 'SUSPENDED':
290
- // A newer image has been published since this VM booted: resuming would
291
- // keep it on the old baked-in hooks and repo checkout for up to 8 h, so
292
- // take the same recreate path the arm below takes — which, passing no
293
- // `imageVersion`, boots the latest (handler.ts's `run`).
294
- if (await shouldRecreateForNewerImage(doorbell, shape, described.imageVersion, microvm)) {
289
+ // False/absent disables silent automatic roll-forward; an explicit restart
290
+ // request (#1906) can still set the same wire flag. Recreate loses VM-local
291
+ // filesystem state; durable provider credentials survive in object storage,
292
+ // but a manual paste/upload not yet synced there may be lost (#2071).
293
+ if (doorbell.recreateOnOutdatedImage === true &&
294
+ (await shouldRecreateForNewerImage(doorbell, shape, described.imageVersion, microvm))) {
295
295
  return runMicrovm(doorbell, shape, microvm, timing, polled, REASON.imageVersionOutdated, state);
296
296
  }
297
297
  return resumeMicrovm(doorbell, doorbell.microvmId, shape, microvm, timing, polled, described.startedAt, described.imageVersion);
@@ -14,6 +14,12 @@ export interface StageMicrovmImageContextOptions {
14
14
  readonly originUrl: string;
15
15
  /** Directory to write the build context to. Removed and recreated. */
16
16
  readonly destination: string;
17
+ /**
18
+ * Overlay scripts the image build runs as root and as `runner`. The overlay
19
+ * ships inside the shared snapshot, so it may install software but must carry
20
+ * no credential. When omitted, deterministic no-op scripts are written.
21
+ */
22
+ readonly overlayDir?: string;
17
23
  /**
18
24
  * The published template to copy from. Defaults to the one inside this
19
25
  * package; named directly by the tests, which assert against a template they
@@ -23,11 +29,14 @@ export interface StageMicrovmImageContextOptions {
23
29
  }
24
30
  /**
25
31
  * Writes the build context AWS unpacks — the Dockerfile, the hook server, the
26
- * per-phase hook scripts and the repository — and returns its path. The result
27
- * is what `EvidentMicrovmConstruct`'s `imageSource` takes.
32
+ * per-phase hook scripts, the deployment overlay and the repository — and
33
+ * returns its path. The result is what `EvidentMicrovmConstruct`'s `imageSource`
34
+ * takes.
28
35
  *
29
- * The first three come from this package's published `dist/`, so a consumer
30
- * needs no checkout of `sroze/evident` and no copy of the Dockerfile or hook
31
- * scripts (#1528). Only `repositoryPath` is theirs to supply.
36
+ * The Dockerfile, hook server and phase hooks come from this package's
37
+ * published `dist/`, so a consumer needs no checkout of the source repository and no
38
+ * copy of those files (#1528). A caller may supply an overlay; the default is
39
+ * deterministic no-op scripts. `repositoryPath` is the only required caller
40
+ * input.
32
41
  */
33
42
  export declare function stageMicrovmImageContext(options: StageMicrovmImageContextOptions): string;
@@ -106,17 +106,36 @@ function stageRepository(repositoryPath, destination, originUrl) {
106
106
  console.log(`[stage] ${repositoryPath} has no pnpm-lock.yaml — the image build will skip dependency installation`);
107
107
  }
108
108
  }
109
+ function stageScriptDirectory(source, destination) {
110
+ (0, node_fs_1.mkdirSync)(destination);
111
+ for (const entry of (0, node_fs_1.readdirSync)(source, { withFileTypes: true })) {
112
+ const sourcePath = path.join(source, entry.name);
113
+ const staged = path.join(destination, entry.name);
114
+ if (entry.isDirectory()) {
115
+ stageScriptDirectory(sourcePath, staged);
116
+ continue;
117
+ }
118
+ (0, node_fs_1.copyFileSync)(sourcePath, staged);
119
+ // Set modes here rather than inheriting them from the template, so they stay
120
+ // correct after an npm pack, zip, CI-cache or hand-copy round trip. The image
121
+ // executes extensionless scripts and sources `.sh` files.
122
+ (0, node_fs_1.chmodSync)(staged, entry.name.endsWith('.sh') ? 0o644 : 0o755);
123
+ }
124
+ }
109
125
  /**
110
126
  * Writes the build context AWS unpacks — the Dockerfile, the hook server, the
111
- * per-phase hook scripts and the repository — and returns its path. The result
112
- * is what `EvidentMicrovmConstruct`'s `imageSource` takes.
127
+ * per-phase hook scripts, the deployment overlay and the repository — and
128
+ * returns its path. The result is what `EvidentMicrovmConstruct`'s `imageSource`
129
+ * takes.
113
130
  *
114
- * The first three come from this package's published `dist/`, so a consumer
115
- * needs no checkout of `sroze/evident` and no copy of the Dockerfile or hook
116
- * scripts (#1528). Only `repositoryPath` is theirs to supply.
131
+ * The Dockerfile, hook server and phase hooks come from this package's
132
+ * published `dist/`, so a consumer needs no checkout of the source repository and no
133
+ * copy of those files (#1528). A caller may supply an overlay; the default is
134
+ * deterministic no-op scripts. `repositoryPath` is the only required caller
135
+ * input.
117
136
  */
118
137
  function stageMicrovmImageContext(options) {
119
- const { originUrl, templateDir = TEMPLATE_DIR } = options;
138
+ const { originUrl, overlayDir, templateDir = TEMPLATE_DIR } = options;
120
139
  // `file://` and the clone below only mean anything against absolute paths,
121
140
  // and a caller may reasonably pass either.
122
141
  const repositoryPath = path.resolve(options.repositoryPath);
@@ -132,16 +151,27 @@ function stageMicrovmImageContext(options) {
132
151
  (0, node_fs_1.copyFileSync)(path.join(templateDir, 'hook-server.js'), path.join(destination, 'hook-server.js'));
133
152
  const hooksSource = path.join(templateDir, 'hooks');
134
153
  const hooksStage = path.join(destination, 'hooks');
135
- (0, node_fs_1.mkdirSync)(hooksStage);
136
- for (const entry of (0, node_fs_1.readdirSync)(hooksSource)) {
137
- const staged = path.join(hooksStage, entry);
138
- (0, node_fs_1.copyFileSync)(path.join(hooksSource, entry), staged);
139
- // The runtime only runs a hook it can execute. Set here rather than
140
- // inherited from the template, so the staged mode is a property of THIS
141
- // function rather than of however the template reached disk — `npm pack`
142
- // does carry the bit, but a zip-based vendoring, a CI cache restore or a
143
- // hand-copied directory need not.
144
- (0, node_fs_1.chmodSync)(staged, entry.endsWith('.sh') ? 0o644 : 0o755);
154
+ stageScriptDirectory(hooksSource, hooksStage);
155
+ const overlayStage = path.join(destination, 'overlay');
156
+ if (overlayDir === undefined) {
157
+ (0, node_fs_1.mkdirSync)(overlayStage);
158
+ for (const name of ['setup-root', 'setup-workspace']) {
159
+ const staged = path.join(overlayStage, name);
160
+ (0, node_fs_1.writeFileSync)(staged, '#!/usr/bin/env bash\n# No overlay supplied.\n', { mode: 0o755 });
161
+ (0, node_fs_1.chmodSync)(staged, 0o755);
162
+ }
163
+ }
164
+ else {
165
+ const resolvedOverlayDir = path.resolve(overlayDir);
166
+ if (!(0, node_fs_1.existsSync)(resolvedOverlayDir)) {
167
+ throw new Error(`overlay directory does not exist: ${resolvedOverlayDir}`);
168
+ }
169
+ for (const required of ['setup-root', 'setup-workspace']) {
170
+ if (!(0, node_fs_1.existsSync)(path.join(resolvedOverlayDir, required))) {
171
+ throw new Error(`overlay directory ${resolvedOverlayDir} is missing required script ${required}`);
172
+ }
173
+ }
174
+ stageScriptDirectory(resolvedOverlayDir, overlayStage);
145
175
  }
146
176
  stageRepository(repositoryPath, path.join(destination, 'repo'), originUrl);
147
177
  return destination;
@@ -1,7 +1,9 @@
1
- # Evident per-session MicroVM image (#558, epic #556)
1
+ # Generic MicroVM runner image
2
2
  #
3
- # Same installed footprint as the ECS runner image, built for linux/arm64
4
- # (Lambda MicroVMs run on Graviton only) and with ONE process: the hook server.
3
+ # Built for linux/arm64 (Lambda MicroVMs run on Graviton only) and with ONE
4
+ # process: the hook server. `overlay/setup-root` and
5
+ # `overlay/setup-workspace` are the extension point for deployment-specific
6
+ # packages and build warm-up.
5
7
  #
6
8
  # THE BOOT SPLIT — the rule that governs everything below. AWS snapshots this
7
9
  # image ONCE at build and every MicroVM resumes from that same snapshot, so
@@ -9,7 +11,8 @@
9
11
  # this image version. Therefore this image must NOT start the tunnel
10
12
  # (`evident run`), clone the repo, or carry any credential, machine ID or
11
13
  # generated secret. Identity arrives per-VM in the `/run` hook payload.
12
- # `src/image/dockerfile.test.ts` fails the build if this regresses.
14
+ # `aws/runner-cdk/src/microvm/image/dockerfile.test.ts` fails the build if this
15
+ # regresses.
13
16
  #
14
17
  # No `--platform` on FROM: AWS builds this natively on Graviton. Build it
15
18
  # locally with `docker build --platform linux/arm64`.
@@ -27,9 +30,8 @@ RUN corepack enable && corepack prepare pnpm@9.15.0 --activate
27
30
 
28
31
  # System tooling + GitHub CLI + AWS CLI, in a single layer.
29
32
  # - awscli (v2): required — durable-state persistence uses `aws s3`.
30
- # - procps, postgresql(+contrib), direnv: required at RUNTIME so the VM is a
31
- # full dev box for this repo (local Postgres on 5433 as the non-root runner,
32
- # no sudo). Listed explicitly so apt keeps them.
33
+ # - procps: required at RUNTIME for process inspection and debugging. Listed
34
+ # explicitly so apt keeps it.
33
35
  # - gnupg, unzip: BUILD-ONLY (gh apt key / AWS CLI bundle); purged in this layer.
34
36
  #
35
37
  # Purge build-only packages BY NAME — do NOT use `apt-get autoremove`, which
@@ -38,7 +40,6 @@ RUN corepack enable && corepack prepare pnpm@9.15.0 --activate
38
40
  RUN apt-get update \
39
41
  && apt-get install -y --no-install-recommends \
40
42
  ca-certificates curl git jq less procps psmisc ripgrep \
41
- postgresql postgresql-contrib direnv \
42
43
  gnupg unzip \
43
44
  # --- GitHub CLI (`gh`) from the official apt repo ---
44
45
  && mkdir -p -m 755 /etc/apt/keyrings \
@@ -54,10 +55,6 @@ RUN apt-get update \
54
55
  && unzip -q /tmp/awscliv2.zip -d /tmp \
55
56
  && /tmp/aws/install \
56
57
  && aws --version \
57
- # --- Put the PostgreSQL server binaries on PATH ---
58
- # On Debian initdb/pg_ctl/postgres/psql live under /usr/lib/postgresql/
59
- # <ver>/bin, not on PATH; symlink them into /usr/local/bin.
60
- && ln -s /usr/lib/postgresql/*/bin/* /usr/local/bin/ 2>/dev/null || true \
61
58
  # --- Drop the build-only packages (BY NAME, no autoremove) + caches ---
62
59
  && apt-get purge -y gnupg unzip \
63
60
  && rm -rf /tmp/aws /tmp/awscliv2.zip /var/lib/apt/lists/*
@@ -87,7 +84,7 @@ RUN curl -fsSL -o /tmp/litestream.tar.gz \
87
84
  # floating `opencode-ai@latest` install was implicated in unbounded runner
88
85
  # memory growth / OOM kills. Keep it in lockstep with the ECS runner image's.
89
86
  # EVIDENT_CLI_VERSION is DELIBERATELY the floating `dev` tag, matching the ECS
90
- # runner image (infrastructure/evident-runner/src/base-image.ts). This is a
87
+ # runner image. This is a
91
88
  # development environment: it should track the head of the CLI so changes are
92
89
  # testable without a version bump in every PR that touches a hook.
93
90
  # RUNNER_SYNCHRONISER_VERSION floats on `dev` for the identical reason.
@@ -104,27 +101,10 @@ RUN npm install -g \
104
101
  && evident --version \
105
102
  && npm cache clean --force
106
103
 
107
- # Playwright chromium + OS deps for E2E, baked at build time to avoid a slow
108
- # first-boot download.
109
- #
110
- # CRITICAL: this version MUST match the Playwright version the repo resolves for
111
- # `@playwright/test` (pnpm-lock.yaml → 1.58.0). The browser build is coupled to
112
- # the package version; if it drifts, a boot-time `playwright install`
113
- # re-downloads a different chromium. Bump in lockstep with @playwright/test.
114
- #
115
- # Browsers go to a SHARED PLAYWRIGHT_BROWSERS_PATH readable by the runtime
116
- # runner user (uid 10001), not root's ~/.cache.
117
- ENV PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright
118
- RUN npx playwright@1.58.0 install --with-deps chromium \
119
- && chown -R 10001:10001 /opt/ms-playwright \
120
- # Assert the browsers landed in the shared path (guard against a silent failure).
121
- && test -n "$(ls -A /opt/ms-playwright 2>/dev/null)" \
122
- || { echo "FATAL: Playwright browsers dir /opt/ms-playwright missing or empty after install" >&2; exit 1; }
123
-
124
104
  # Assert every REQUIRED runtime binary survived every layer above (fail loud).
125
105
  # Placed after the last install so it covers the apt purge, the litestream
126
106
  # extraction and the global npm installs in one pass.
127
- RUN for bin in ps pgrep pkill curl aws gh git jq rg psql pg_ctl initdb direnv litestream runner-synchroniser opencode evident timeout; do \
107
+ RUN for bin in ps pgrep pkill curl aws gh git jq rg litestream runner-synchroniser opencode evident timeout; do \
128
108
  command -v "$bin" >/dev/null 2>&1 || { echo "FATAL: required binary '$bin' missing from image" >&2; exit 1; }; \
129
109
  done \
130
110
  && curl -fsS -o /dev/null https://cli.github.com \
@@ -162,8 +142,6 @@ COPY hooks /etc/evident/hooks
162
142
  # per-VM, so the agent is what brings this checkout up to date.
163
143
  ENV WORKSPACE=/workspace
164
144
  ENV HOME=/home/runner
165
- # Runner-owned PGDATA parent for the local dev Postgres cluster (initdb +
166
- # pg_ctl at boot, no sudo).
167
145
  #
168
146
  # The machine-id files are created EMPTY and runner-owned so that /run can
169
147
  # actually rewrite them: the hooks run as uid 10001, and neither `/etc` nor
@@ -171,10 +149,18 @@ ENV HOME=/home/runner
171
149
  # permanent no-op and every VM from this snapshot shares one machine id. Empty
172
150
  # is the correct unset state — it is the absence of an identity, so nothing
173
151
  # per-VM-unique enters the shared snapshot.
174
- RUN mkdir -p /var/lib/runner-pg /var/lib/dbus /home/runner/.local/state/evident \
152
+ RUN mkdir -p /var/lib/dbus /home/runner/.local/state/evident \
175
153
  && install -o runner -g runner -m 0644 /dev/null /etc/machine-id \
176
154
  && install -o runner -g runner -m 0644 /dev/null /var/lib/dbus/machine-id \
177
- && chown -R runner:runner /var/lib/runner-pg /home/runner
155
+ && chown -R runner:runner /home/runner
156
+
157
+ # Deployment additions are always present because staging writes no-op defaults
158
+ # when no overlay is supplied.
159
+ COPY overlay /etc/evident/overlay
160
+ # HOME is already /home/runner, so root-warmed npx/npm state must be runner-owned
161
+ # or the runner's first npm/pnpm call would fail with EACCES.
162
+ RUN /etc/evident/overlay/setup-root \
163
+ && chown -R runner:runner /home/runner
178
164
  COPY --chown=10001:10001 repo ${WORKSPACE}
179
165
  WORKDIR ${WORKSPACE}
180
166
  USER runner
@@ -185,8 +171,8 @@ USER runner
185
171
  # `runner` (not root) so the rebuilt index is runner-owned and writable later.
186
172
  RUN git reset --quiet
187
173
 
188
- # Dependencies and the shared builds everything else depends on. As the runner
189
- # user, so node_modules is writable by the agent's own later installs.
174
+ # Dependencies for the baked workspace. Install as the runner user so
175
+ # node_modules is writable by the agent's own later installs.
190
176
  #
191
177
  # ONE layer on purpose: pnpm hard-links node_modules into its content-addressed
192
178
  # store (which it puts at ${WORKSPACE}/.pnpm-store, gitignored), and hard links
@@ -199,22 +185,18 @@ RUN git reset --quiet
199
185
  # at a repository that is not a pnpm workspace at all: it is baked uninstalled
200
186
  # instead of failing the build, and the agent installs on first use.
201
187
  #
202
- # Nothing environment-specific is built: `vite build` inlines VITE_* at build
203
- # time and the migrations need a running Postgres, so the app builds and the dev
204
- # database stay per-VM (see entrypoint.sh's pre-warm in evident-runner).
188
+ # Anything beyond dependency installation is deployment-specific and belongs in
189
+ # `overlay/setup-workspace`.
205
190
  RUN if [ -f pnpm-lock.yaml ]; then \
206
- pnpm install --frozen-lockfile \
207
- && pnpm run build --filter='./packages/*' \
208
- --filter='./aws/runner-cdk' \
209
- --filter='./aws/lambda-microvm-cdk' \
210
- --filter='./aws/lambda-microvm-runtime'; \
191
+ pnpm install --frozen-lockfile; \
211
192
  else \
212
- echo "[workspace-prep] no pnpm-lock.yaml in the baked repository — skipping the dependency install and the workspace build; the agent installs on first use."; \
193
+ echo "[workspace-prep] no pnpm-lock.yaml in the baked repository — skipping the dependency install; the agent installs on first use."; \
213
194
  fi
214
195
 
215
- # Data dir for the local dev Postgres cluster; listens on 5433 to match this
216
- # repo's .envrc DSN postgres://postgres:postgres@localhost:5433/evident.
217
- ENV PGDATA=/var/lib/runner-pg/data
196
+ # Runs as `runner` in `${WORKSPACE}`, after the install, so a project's build
197
+ # output is writable by the agent.
198
+ RUN /etc/evident/overlay/setup-workspace
199
+
218
200
  # Port AWS calls the MicroVM hooks on.
219
201
  ENV HOOKS_PORT=8080
220
202
  # OpenCode loopback port the CLI tunnels to (matches `evident run` default).
@@ -21,11 +21,15 @@ OPENCODE_PORT="${OPENCODE_PORT:-4096}"
21
21
  # /proc/<pid>/environ, which runs as that same uid. /terminate removes it.
22
22
  # shellcheck disable=SC2034 # read by the scripts that source this file
23
23
  CONTEXT_FILE="/dev/shm/evident-run-context"
24
+
25
+ # The runtime injects MICROVM_ID only into /run, never /resume. This tmpfs file
26
+ # survives suspend/resume so /resume can restore the id for every fresh CLI
27
+ # process to self-report and acknowledge a fulfilled recycle request (#1906).
28
+ # shellcheck disable=SC2034 # read by the scripts that source this file
29
+ MICROVM_ID_FILE="/dev/shm/evident-microvm-id"
24
30
  TUNNEL_PID_FILE="/dev/shm/evident-tunnel.pid"
25
31
  OPENCODE_PID_FILE="/dev/shm/evident-opencode.pid"
26
32
  LITESTREAM_PID_FILE="/dev/shm/evident-litestream.pid"
27
- CREDS_SYNC_PID_FILE="/dev/shm/evident-creds-sync.pid"
28
- CREDS_SYNC_LAST_ERROR_FILE="/dev/shm/evident-creds-sync.last-error"
29
33
 
30
34
  # Where the runner's credential store lives inside the durable-state bucket.
31
35
  # The BUCKET is the same for every VM from an image version, so the stack bakes
@@ -47,6 +51,10 @@ LITESTREAM_CONFIG_FILE="/dev/shm/evident-litestream.yml"
47
51
  # `/terminate` removes it with the rest of the per-VM state.
48
52
  SESSION_DB_NO_REPLICATE_MARKER="/dev/shm/evident-session-db-no-replicate"
49
53
 
54
+ # Completion evidence for the CLI-owned credential flush. It lives in tmpfs,
55
+ # is removed before every handshake, and is also removed by /terminate.
56
+ CREDENTIAL_FLUSH_MARKER_FILE="/dev/shm/evident-credential-flush"
57
+
50
58
  hook_name() { printf '%s' "${0##*/}"; }
51
59
  log() { echo "[hook:$(hook_name)] $*"; }
52
60
  warn() { echo "[hook:$(hook_name)] $*" >&2; }
@@ -318,7 +326,6 @@ regenerate_machine_id() {
318
326
  tunnel_is_running() { is_running "${TUNNEL_PID_FILE}"; }
319
327
  opencode_is_running() { is_running "${OPENCODE_PID_FILE}"; }
320
328
  litestream_is_running() { is_running "${LITESTREAM_PID_FILE}"; }
321
- creds_sync_is_running() { is_running "${CREDS_SYNC_PID_FILE}"; }
322
329
 
323
330
  # `jq -e` alone is not enough: its exit status reflects the LAST OUTPUT VALUE,
324
331
  # and an interpolation of a missing field is still a non-empty string, so a
@@ -463,149 +470,6 @@ kill_litestream() {
463
470
  }
464
471
  # --- litestream replicate (end) ----------------------------------------------
465
472
 
466
- # --- credential sync loop (#1868 WI-3, ECS parity) ---------------------------
467
- #
468
- # sync_credentials (above) covers the three boundary flushes /run's restore,
469
- # /suspend and /terminate already call. What it does NOT cover is a VM that
470
- # runs for a long time between those boundaries: a provider re-authenticated
471
- # through the proxied UI hours into a run would sit unflushed until the next
472
- # suspend/terminate, and a VM that dies without one (a crash, an OOM kill)
473
- # loses everything since boot. runner/docker-images/fargate/entrypoint.sh's
474
- # own sync_credentials_loop is the ECS side of the identical gap; this is the
475
- # same fix, backgrounded like the other long-lived services so it
476
- # outlives this hook process, `( … ) &` rather than `setsid`: a plain
477
- # backgrounded subshell is reparented to init and keeps running once its
478
- # parent hook script exits (verified: PPID=1, still alive, with no controlling
479
- # terminal in this image to send it a stray SIGHUP), and it inherits every
480
- # function this file defines, so it can call run_synchroniser directly with no
481
- # re-exec.
482
-
483
- # Bounded confirmation window `stop_credential_sync` polls after signalling the
484
- # loop. The loop's current child is one fast `run_synchroniser sync-once` call,
485
- # so this stays a short backstop rather than a graceful drain.
486
- CREDS_SYNC_STOP_WAIT_SECONDS=2
487
-
488
- # Best-effort per tick, exactly like sync_credentials above: a failed tick
489
- # must never end the loop, or a single transient S3 error would silently
490
- # disable sync for the rest of the VM's life.
491
- start_credential_sync() {
492
- if [ -z "${PERSISTENCE_BUCKET:-}" ]; then
493
- warn "CREDS-SYNC-DISABLED: LITESTREAM_BUCKET/LITESTREAM_PREFIX are not both set; no interval credential sync this boot."
494
- return 0
495
- fi
496
-
497
- if creds_sync_is_running; then
498
- warn "credential sync loop already running (pid $(cat "${CREDS_SYNC_PID_FILE}")); reusing it"
499
- return 0
500
- fi
501
-
502
- # This is the same environment input runner-synchroniser validates. The
503
- # variable is normally absent, so that ordinary case uses the documented
504
- # default without a warning; a present invalid value is named and rejected.
505
- local interval="${CREDS_SYNC_INTERVAL:-60}"
506
- if [ -n "${CREDS_SYNC_INTERVAL+x}" ] && [[ ! "${CREDS_SYNC_INTERVAL}" =~ ^[1-9][0-9]*$ ]]; then
507
- warn "CREDS-SYNC-INTERVAL-INVALID: CREDS_SYNC_INTERVAL='${CREDS_SYNC_INTERVAL}' is not a positive integer; using 60s"
508
- interval=60
509
- fi
510
-
511
- rm -f "${CREDS_SYNC_LAST_ERROR_FILE}"
512
-
513
- (
514
- # Releases the fds this subshell inherited from the hook process before
515
- # settling in for the VM's whole remaining life: nothing here writes to
516
- # them (every synchroniser call already redirects its own), so there is
517
- # no reason to keep holding the hook's original stdout/stderr open. A
518
- # long-lived process that instead inherited a pipe's write end (a test
519
- # harness reading the hook's own output, for one) would keep that pipe
520
- # from ever reporting EOF — testing-guide.mdc's own lesson, and the same
521
- # reason the long-lived services never inherit stdio either. That
522
- # redirect also means `warn`/`log`/`error` calls in here go nowhere, so a
523
- # failed sync-once is instead recorded to CREDS_SYNC_LAST_ERROR_FILE and
524
- # surfaced by stop_credential_sync, which DOES have live stdio.
525
- exec >/dev/null 2>&1 </dev/null
526
-
527
- # A TERM this subshell receives (from stop_credential_sync, below) only
528
- # kills THIS wrapper by default — its currently-running child (`sleep`,
529
- # or a `run_synchroniser sync-once` call) is a separate process that
530
- # would otherwise be orphaned and keep running, free to upload STALE
531
- # credentials to S3 after the boundary flush that /suspend and
532
- # /terminate perform immediately following the stop. Tracking the
533
- # current child explicitly and forwarding the signal closes that race.
534
- creds_sync_child_pid=""
535
- trap 'trap - TERM; [ -n "${creds_sync_child_pid}" ] && kill -TERM "${creds_sync_child_pid}" 2>/dev/null; exit 0' TERM
536
-
537
- while true; do
538
- sleep "${interval}" &
539
- creds_sync_child_pid=$!
540
- wait "${creds_sync_child_pid}" 2>/dev/null
541
- creds_sync_child_pid=""
542
-
543
- run_synchroniser sync-once claude &
544
- creds_sync_child_pid=$!
545
- wait "${creds_sync_child_pid}" 2>/dev/null || echo "claude" >"${CREDS_SYNC_LAST_ERROR_FILE}"
546
- creds_sync_child_pid=""
547
-
548
- run_synchroniser sync-once opencode &
549
- creds_sync_child_pid=$!
550
- wait "${creds_sync_child_pid}" 2>/dev/null || echo "opencode" >"${CREDS_SYNC_LAST_ERROR_FILE}"
551
- creds_sync_child_pid=""
552
- done
553
- ) &
554
-
555
- echo $! >"${CREDS_SYNC_PID_FILE}"
556
- log "CREDS-SYNC-STARTED: pid=$! interval=${interval}s"
557
- }
558
-
559
- # Signals the loop, then confirms (bounded — see CREDS_SYNC_STOP_WAIT_SECONDS)
560
- # that it and its current child are actually gone before returning: /suspend
561
- # and /terminate start their own boundary flush immediately after this call,
562
- # and an orphaned in-flight sync-once surviving past that point can overwrite
563
- # fresher credentials with stale ones. The TERM trap inside the loop (above)
564
- # forwards the signal to its current child almost instantly — this poll is a
565
- # defensive confirmation, not the primary mechanism, so it stays short; a
566
- # SIGKILL backstop covers a child that ignores TERM entirely.
567
- #
568
- # The DIED branch is a liveness report, not a no-op: every recovery/no-op path
569
- # must say what it found (development-workflow.mdc) — a stopped-before-called
570
- # loop and a died-on-its-own loop are different facts an operator needs told
571
- # apart, not the same "nothing to stop" line.
572
- stop_credential_sync() {
573
- if [ ! -s "${CREDS_SYNC_PID_FILE}" ]; then
574
- log "CREDS-SYNC-NOT-RUNNING: no credential sync loop to stop"
575
- return 0
576
- fi
577
-
578
- local pid
579
- pid="$(cat "${CREDS_SYNC_PID_FILE}")"
580
- if ! process_is_alive "${pid}"; then
581
- rm -f "${CREDS_SYNC_PID_FILE}"
582
- warn "CREDS-SYNC-DIED: credential sync loop (pid=${pid}) had already exited before this stop"
583
- return 0
584
- fi
585
-
586
- kill -TERM "${pid}" 2>/dev/null || true
587
- rm -f "${CREDS_SYNC_PID_FILE}"
588
-
589
- local waited_ms=0
590
- while process_is_alive "${pid}" && [ "${waited_ms}" -lt $((CREDS_SYNC_STOP_WAIT_SECONDS * 1000)) ]; do
591
- sleep 0.1
592
- waited_ms=$((waited_ms + 100))
593
- done
594
-
595
- if process_is_alive "${pid}"; then
596
- kill -KILL "${pid}" 2>/dev/null || true
597
- warn "CREDS-SYNC-STOP-TIMEOUT: pid=${pid} still alive after ${CREDS_SYNC_STOP_WAIT_SECONDS}s; sent SIGKILL"
598
- fi
599
-
600
- if [ -s "${CREDS_SYNC_LAST_ERROR_FILE}" ]; then
601
- warn "CREDS-SYNC-HAD-FAILURES: sync-once failed at least once for: $(tr '\n' ' ' <"${CREDS_SYNC_LAST_ERROR_FILE}")"
602
- rm -f "${CREDS_SYNC_LAST_ERROR_FILE}"
603
- fi
604
-
605
- log "CREDS-SYNC-STOPPED: pid=${pid}"
606
- }
607
- # --- credential sync loop (end) -----------------------------------------------
608
-
609
473
  # --- flush_session_db (#812 WI-4) -------------------------------------------
610
474
  #
611
475
  # The checked, synchronous flush /suspend and /terminate need before they
@@ -677,16 +541,19 @@ flush_session_db() {
677
541
  # --- flush_session_db (end) --------------------------------------------------
678
542
 
679
543
  # How long the guest CLI runs with no activity before it exits itself
680
- # (`evident run --idle-timeout`, apps/cli/src/commands/run.ts), which is what
544
+ # (`evident run --idle-timeout`), which is what
681
545
  # turns a truly-abandoned VM into the clean-offline POST that lets Evident
682
546
  # suspend it (#732). Sized from the measured cost of guessing wrong rather than
683
547
  # the saving: suspend reaches SUSPENDED in ~7 s and a resume is RUNNING in
684
548
  # ~0.6 s with the tunnel back ~2 s later (README, "Measured, end to end"), so
685
- # napping a VM whose user comes straight back costs ~10 s — against ~$0.30/h
686
- # that is cheap enough that the balance sits far nearer the floor than the
687
- # ceiling. Not AT the floor, though: the CLI's idle detector needs 2 clear poll
688
- # cycles (`run.ts`'s `idlePolls >= 2`, ≥4 s of real time), so a value near that
689
- # would spend more time suspending/resuming than idle.
549
+ # napping a VM whose user comes straight back costs ~10 s — against the
550
+ # legacy conservative baseline input of ~$0.30/h, AWS can burst a loaded VM
551
+ # to a 16 GB / 8 vCPU peak (~$1.06/h at sustained full load—a worst-case
552
+ # ceiling, not an expectation); that is cheap enough that the balance
553
+ # sits far nearer the floor than the ceiling. Not AT the floor, though: the
554
+ # CLI's idle detector needs 2 clear poll cycles (≥4 s of real time), so a value
555
+ # near that would spend more time
556
+ # suspending/resuming than idle.
690
557
  # ECS's waker uses 900 s instead only because *its* cold start is far slower
691
558
  # than this VM's ~2 s resume — not evidence this default should match it.
692
559
  #
@@ -698,7 +565,9 @@ flush_session_db() {
698
565
  # runner/docker-images/fargate/entrypoint.sh uses for the ECS runner's own
699
566
  # idle-timeout flag, so an operator who knows one knows the other. A
700
567
  # non-numeric override must never silently DROP the flag — that degrades to
701
- # an always-on VM burning ~$2.40/day, exactly the bug this closes so it
568
+ # an always-on VM burning ~$7.17/day at baseline, or up to ~$25.49/day at
569
+ # sustained full peak as load increases (a worst-case ceiling, not an
570
+ # expectation), exactly the bug this closes — so it
702
571
  # warns and falls back to the default instead.
703
572
  #
704
573
  # Deliberately NOT in the /run payload yet: doing so would touch the doorbell
@@ -766,6 +635,7 @@ start_tunnel() {
766
635
  --litestream-config "${LITESTREAM_CONFIG_FILE}" \
767
636
  --litestream-pid-file "${LITESTREAM_PID_FILE}" \
768
637
  --session-db-no-replicate-marker "${SESSION_DB_NO_REPLICATE_MARKER}" \
638
+ --credential-sync-marker "${CREDENTIAL_FLUSH_MARKER_FILE}" \
769
639
  "${credential_flags[@]}" \
770
640
  "${session_db_flags[@]}" \
771
641
  "${opencode_config_flags[@]}" \
@@ -777,22 +647,20 @@ start_tunnel() {
777
647
  }
778
648
 
779
649
  # The worst case `evident run` can take to shut down gracefully on SIGTERM, in
780
- # whole seconds: 25 s in-flight drain (SHUTDOWN_DRAIN_TIMEOUT_MS,
781
- # apps/cli/src/commands/run.ts) + 2 s offline POST (notifyAgentDisconnected,
782
- # apps/cli/src/commands/agent-lookup.ts) + 5 s telemetry flush
783
- # (TELEMETRY_SHUTDOWN_TIMEOUT_MS, run.ts). Each of the three is bounded there, so
784
- # this is a ceiling rather than a typical cost — an idle suspend finishes in a
785
- # couple of seconds. This is the ONE place the budget is written down; the CLI
786
- # only points back here, because restating it in three places produced #657.
650
+ # whole seconds: 25 s in-flight drain (SHUTDOWN_DRAIN_TIMEOUT_MS) + 2 s offline
651
+ # POST (notifyAgentDisconnected) + 5 s telemetry flush
652
+ # (TELEMETRY_SHUTDOWN_TIMEOUT_MS) + 8.5 s pre-drain credential flush
653
+ # + 8.5 s post-drain credential flush. Each phase is bounded there, so this is
654
+ # a ceiling rather than a typical cost — an idle suspend finishes in a couple
655
+ # of seconds. This is the ONE place the hand-maintained budget is written down;
656
+ # a guard in the source repository keeps this number in step with the CLI's
657
+ # declared bounds.
787
658
  #
788
- # DOCUMENTATION ONLY — nothing is derived from this any more. The wait below
789
- # used to be pinned above it, which is the reasoning that talked #699 into 40 s;
790
- # since #718 an exited CLI is noticed on the first poll, so the wait is a
791
- # backstop chosen on its own merits and the two numbers are unrelated. Nothing
792
- # cross-checks the 32 against apps/cli either: it is a hand-maintained sum of the
793
- # three bounds cited above, so if one of them moves, update it here.
659
+ # DOCUMENTATION ONLY — nothing is derived from this value. It is the hand-maintained
660
+ # sum of the five bounded phases above (25 + 2 + 5 + 8.5 + 8.5); update it here if
661
+ # any of those bounds changes.
794
662
  # shellcheck disable=SC2034 # documentation; deliberately read by nothing
795
- CLI_SHUTDOWN_CEILING_SECONDS=32
663
+ CLI_SHUTDOWN_CEILING_SECONDS=49
796
664
 
797
665
  # How long stop_tunnel waits for that shutdown before the SIGKILL backstop.
798
666
  # Since #718 this binds ONLY for a CLI that is still draining: one that has
@@ -868,12 +736,91 @@ stop_tunnel() {
868
736
  log "tunnel stopped ${outcome}"
869
737
  }
870
738
 
739
+ # The CLI owns the interval loop and its boundary flush. Two seconds of slack
740
+ # over the CLI's 8s flush deadline keeps this marker handshake inside the 55s
741
+ # hook ceiling while leaving the fallback flush and tunnel stop budget intact.
742
+ #
743
+ # This bounds only the CLI's pre-drain flush, which always runs first and
744
+ # unconditionally (run.ts's cleanup(), before the channel-work drain) — not
745
+ # the CLI's post-drain second pass, which can take up to
746
+ # SHUTDOWN_DRAIN_TIMEOUT_MS longer than this wait covers. A drain that
747
+ # consumes the whole window loses the SECOND pass, not the credential
748
+ # guarantee itself: the pre-drain flush already persisted everything on disk
749
+ # at signal time, exactly what the old bash `sync_credentials` guaranteed in
750
+ # one synchronous call — so the worst case here is no worse than before this
751
+ # handshake existed, never a fresh data-loss window. See
752
+ # docs/decisions/0063-microvm-boot-orchestration-in-cli.md's two-phase-flush
753
+ # section for the full reasoning.
754
+ CREDENTIAL_FLUSH_WAIT_SECONDS="${EVIDENT_CREDENTIAL_FLUSH_WAIT_SECONDS:-10}"
755
+
756
+ # Remove the previous answer, signal the same CLI that stop_tunnel handles, and
757
+ # wait for either its marker or its death. A fallback sync runs only after the
758
+ # CLI is known to be absent, never alongside a live CLI that may still write.
759
+ stop_runner_and_flush_credentials() {
760
+ rm -f "${CREDENTIAL_FLUSH_MARKER_FILE}"
761
+
762
+ if ! tunnel_is_running; then
763
+ warn "CREDS-FLUSH-NO-RUNNER: no live CLI at handshake entry"
764
+ stop_tunnel
765
+ sync_credentials
766
+ return 0
767
+ fi
768
+
769
+ local pid
770
+ pid="$(cat "${TUNNEL_PID_FILE}")"
771
+ kill -TERM "${pid}" 2>/dev/null || true
772
+
773
+ local waited_ms=0 marker_found=false runner_exited=false
774
+ while [ "${waited_ms}" -lt $((CREDENTIAL_FLUSH_WAIT_SECONDS * 1000)) ]; do
775
+ if [ -s "${CREDENTIAL_FLUSH_MARKER_FILE}" ]; then
776
+ marker_found=true
777
+ break
778
+ fi
779
+ if ! process_is_alive "${pid}"; then
780
+ runner_exited=true
781
+ break
782
+ fi
783
+ sleep 0.1
784
+ waited_ms=$((waited_ms + 100))
785
+ done
786
+
787
+ # The CLI can publish the marker and exit inside one poll tick. A final
788
+ # marker check after a death break preserves that answer instead of falling
789
+ # through to the fallback.
790
+ if [ "${runner_exited}" = true ] && [ -s "${CREDENTIAL_FLUSH_MARKER_FILE}" ]; then
791
+ marker_found=true
792
+ runner_exited=false
793
+ fi
794
+
795
+ if [ "${marker_found}" = true ]; then
796
+ local failures
797
+ failures="$(grep -Ev '^(claude|opencode)=ok$' "${CREDENTIAL_FLUSH_MARKER_FILE}" || true)"
798
+ if [ -n "${failures}" ]; then
799
+ warn "CREDS-FLUSH-HAD-FAILURES: ${failures//$'\n'/ }"
800
+ else
801
+ log "CREDS-FLUSH-OK"
802
+ fi
803
+ stop_tunnel
804
+ return 0
805
+ fi
806
+
807
+ if [ "${runner_exited}" = true ]; then
808
+ warn "CREDS-FLUSH-RUNNER-EXITED: CLI exited without a completion marker"
809
+ stop_tunnel
810
+ sync_credentials
811
+ return 0
812
+ fi
813
+
814
+ warn "CREDS-FLUSH-TIMEOUT: live CLI did not write a completion marker within ${CREDENTIAL_FLUSH_WAIT_SECONDS}s"
815
+ stop_tunnel
816
+ }
817
+
871
818
  # --- check_runner_key (#1172) ------------------------------------------------
872
819
  #
873
820
  # Answers exactly one question before opencode/the tunnel start spending this
874
821
  # boot's SIGTERM budget on a key that cannot work: "does the runner key in this
875
822
  # payload authenticate against Evident?" Delegates entirely to the CLI's own
876
- # `evident status --json` (apps/cli/src/commands/status.ts) rather than
823
+ # `evident status --json` rather than
877
824
  # reimplementing its auth logic here — that command's `reason` field is the
878
825
  # published contract this function reads, and its own header states the
879
826
  # absent-vs-contrary distinction this function must honour.
@@ -2,10 +2,8 @@
2
2
  #
3
3
  # Re-dial with the identity /run left behind. There is no step that could fetch
4
4
  # a fresh runner key, so the one from /run is what resumes. The CLI delegated by
5
- # start_tunnel owns the session-DB replicator's post-resume start. The interval
6
- # credential sync loop (#1868 WI-3) stays here because /suspend stops it too, so
7
- # a resumed VM that never restarted it here would never sync credentials again
8
- # for the rest of its life.
5
+ # start_tunnel owns the credential loop and the session-DB replicator's
6
+ # post-resume start.
9
7
  set -euo pipefail
10
8
 
11
9
  # shellcheck source=./common.sh
@@ -39,12 +37,18 @@ fi
39
37
  read -r runner_key
40
38
  } <"${CONTEXT_FILE}"
41
39
 
40
+ # Restore the id so the resumed CLI can self-report and acknowledge any
41
+ # outstanding recycle request (#1906). Older images may not have this file.
42
+ if [ -s "${MICROVM_ID_FILE}" ]; then
43
+ MICROVM_ID="$(cat "${MICROVM_ID_FILE}")"
44
+ export MICROVM_ID
45
+ fi
46
+
42
47
  # Tolerant, never fatal: a resume that fails costs the user their whole
43
48
  # session, so a broken durable-state config degrades to "no session-DB
44
49
  # replication" rather than a failed resume. The CLI's own guards handle the
45
50
  # rest — this needs no logic of its own.
46
51
  load_state_config || warn "could not resolve durable-state config; the session DB will not resume replicating"
47
- start_credential_sync
48
52
 
49
53
  # Diagnostic-only, unlike /run's gate: a failed resume costs the user their
50
54
  # whole session, so this never exits — it only converts a silent "resumed but
@@ -69,9 +73,8 @@ start_credential_sync
69
73
  # list, so a later edit cannot accidentally foreground the check again.
70
74
  ( check_runner_key "${runner_key}" "${api_url}" || true ) &
71
75
 
72
- # Not waited on: the CLI backs off and keeps retrying its own dial with no
73
- # attempt cap (apps/cli/src/lib/tunnel/connection.ts), and a control-plane
74
- # reclaim pass (the api-worker lifecycle cron) is what decides whether a
76
+ # Not waited on: the Evident CLI backs off and keeps retrying its own dial with no
77
+ # attempt cap, and Evident's control-plane reclaim pass is what decides whether a
75
78
  # runner that never reconnects gets suspended.
76
79
  start_tunnel "${runner_key}" "${api_url}" "${tunnel_url}"
77
80
 
@@ -28,7 +28,6 @@ cleanup() {
28
28
  stop_tunnel || warn "stop_tunnel failed while cleaning up"
29
29
  stop_opencode || warn "stop_opencode failed while cleaning up"
30
30
  kill_litestream || warn "kill_litestream failed while cleaning up"
31
- stop_credential_sync || warn "stop_credential_sync failed while cleaning up"
32
31
  }
33
32
  trap cleanup EXIT
34
33
 
@@ -73,9 +72,12 @@ load_state_config || exit 1
73
72
  # it warns and this VM still boots.
74
73
  check_runner_key "${runner_key}" "${api_url}" || exit 1
75
74
 
76
- # 6 — the interval credential sync (#1868 WI-3). It stays before the context file
77
- # is written. Bare: every guard inside start_credential_sync is its own `return 0`.
78
- start_credential_sync
75
+ # 6 — preserve the VM id across suspend/resume. The runtime always supplies it
76
+ # for a /run hook, and the subshell keeps the persisted value protected.
77
+ (
78
+ umask 077
79
+ printf '%s\n' "${MICROVM_ID}" >"${MICROVM_ID_FILE}"
80
+ )
79
81
 
80
82
  # 7 — the first per-VM identity on the wire. The subshell's umask makes the file
81
83
  # unreadable to anyone else from the moment it exists, before the key is in it.
@@ -85,9 +87,8 @@ start_credential_sync
85
87
  >"${CONTEXT_FILE}"
86
88
  )
87
89
 
88
- # Not waited on: the CLI backs off and keeps retrying its own dial with no
89
- # attempt cap (apps/cli/src/lib/tunnel/connection.ts), and a control-plane
90
- # reclaim pass (the api-worker lifecycle cron) is what decides whether a
90
+ # Not waited on: the Evident CLI backs off and keeps retrying its own dial with no
91
+ # attempt cap, and Evident's control-plane reclaim pass is what decides whether a
91
92
  # runner that never connects gets suspended.
92
93
  start_tunnel "${runner_key}" "${api_url}" "${tunnel_url}" true true
93
94
 
@@ -8,11 +8,10 @@ set -euo pipefail
8
8
  # shellcheck source=./common.sh
9
9
  source "$(dirname "$0")/common.sh"
10
10
 
11
- # Stopped BEFORE the boundary flush (#1868 WI-3): the interval loop and this
12
- # flush must not race each other on the same credential stores.
13
- stop_credential_sync
14
- sync_credentials
15
- stop_tunnel
11
+ # load_state_config exports PERSISTENCE_BUCKET, which flush_session_db below
12
+ # requires; the CLI flush completes, or is proven absent, before teardown continues.
13
+ load_state_config || warn "could not resolve durable-state config; neither the credential flush nor the session-DB flush below can run"
14
+ stop_runner_and_flush_credentials
16
15
 
17
16
  # opencode is deliberately NOT stopped here (#812 WI-4) — it is inside the
18
17
  # snapshot and must resume with it — so a turn still in flight may not be
@@ -8,11 +8,10 @@ set -uo pipefail
8
8
  # shellcheck source=./common.sh
9
9
  source "$(dirname "$0")/common.sh"
10
10
 
11
- # Stopped BEFORE the boundary flush (#1868 WI-3): the interval loop and this
12
- # flush must not race each other on the same credential stores.
13
- stop_credential_sync
14
- sync_credentials
15
- stop_tunnel
11
+ # load_state_config exports PERSISTENCE_BUCKET, which flush_session_db below
12
+ # requires; the CLI flush completes, or is proven absent, before teardown continues.
13
+ load_state_config || warn "could not resolve durable-state config; neither the credential flush nor the session-DB flush below can run"
14
+ stop_runner_and_flush_credentials
16
15
 
17
16
  # Writers stopped BEFORE litestream's final sync (#812 WI-4, the
18
17
  # ordered-shutdown invariant, plan §6): otherwise litestream could snapshot
@@ -32,6 +31,6 @@ flush_session_db
32
31
  # this hook runs under `set -u` (above), so an unbound one aborts the script
33
32
  # AT THIS LINE, leaving the runner key sitting in CONTEXT_FILE on a VM that is
34
33
  # being torn down. That is the one thing this line exists to prevent.
35
- rm -f "${CONTEXT_FILE}" "${SESSION_DB_NO_REPLICATE_MARKER}"
34
+ rm -f "${CONTEXT_FILE}" "${SESSION_DB_NO_REPLICATE_MARKER}" "${CREDENTIAL_FLUSH_MARKER_FILE}"
36
35
 
37
36
  exit 0
@@ -60,8 +60,7 @@ class EvidentWaker extends constructs_1.Construct {
60
60
  // dist/waker-lambda/handler.js, so neither this package's published npm
61
61
  // artifact (which ships dist/ already built) nor a consuming app needs
62
62
  // that private workspace package, or esbuild, on synth's PATH. Requires
63
- // a build before synth when consuming this package via `workspace:*` —
64
- // see infrastructure/evident-runner's README "local gate".
63
+ // a build before synth when consuming this package via `workspace:*`.
65
64
  //
66
65
  // Resolved from the PACKAGE ROOT (../.. ), not `__dirname` directly:
67
66
  // `__dirname` is `dist/waker` at runtime (compiled) but `src/waker` when
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@evident-ai/runner-cdk",
3
- "version": "3.4.1-dev.cb56b3f",
3
+ "version": "3.4.1-dev.e91ca26",
4
4
  "description": "Reusable CDK constructs for an Evident agent runner: a single scale-to-zero Fargate runner (task + service + per-agent self-stop role + waker Lambda), or a per-session AWS Lambda MicroVM that boots on demand and suspends between messages. Instantiate once per agent from your own stack.",
5
5
  "main": "./dist/index.js",
6
6
  "types": "./dist/index.d.ts",
@@ -15,7 +15,7 @@
15
15
  ],
16
16
  "scripts": {
17
17
  "build": "pnpm run build-bundled-deps && ts-node scripts/build.ts",
18
- "//build-bundled-deps": "scripts/build.ts esbuild-bundles runner/docker-images/microvm/hook-server.ts, which imports @evident-ai/lambda-microvm-runtime by its built `main`. @evident-ai/lambda-microvm-cdk is a workspace:* devDependency whose types resolve through its own dist/, so tsc --project tsconfig.build.json cannot compile src/microvm/{shapes,construct}.ts without it. Chained into `build` rather than left to turbo's `^build` because publish-runner-cdk.yaml and infrastructure/evident-runner's build-runner-cdk-dep invoke `pnpm --filter @evident-ai/runner-cdk build` directly, which bypasses turbo entirely.",
18
+ "//build-bundled-deps": "scripts/build.ts esbuild-bundles runner/docker-images/microvm/hook-server.ts, which imports @evident-ai/lambda-microvm-runtime by its built `main`. @evident-ai/lambda-microvm-cdk is a workspace:* devDependency whose types resolve through its own dist/, so tsc --project tsconfig.build.json cannot compile src/microvm/{shapes,construct}.ts without it. Chained into `build` rather than left to turbo's `^build` because callers outside turbo invoke `pnpm --filter @evident-ai/runner-cdk build` directly, which bypasses turbo entirely.",
19
19
  "build-bundled-deps": "pnpm --filter @evident-ai/lambda-microvm-cdk build && pnpm --filter @evident-ai/lambda-microvm-runtime build",
20
20
  "typecheck": "tsc --noEmit",
21
21
  "test": "node --test --require ts-node/register 'src/**/*.test.ts'",