@xemahq/repo-build-tooling 0.21.0 → 0.21.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -31,8 +31,8 @@ pnpm exec xema-scrub-swagger-paths dist
31
31
  | `xema-scrub-swagger-paths <dist-dir>...` | Rewrites `@nestjs/swagger` plugin output that names a private install path, then fails the build if any leak remains (`--self-test` available) |
32
32
  | `xema-composed-image <command>` | Build-time half of a composed-process image: `members`, `filters`, `dirs`, `pins`, `index`, `load`, `stage`, `pnpm-context`, `sync-injected`, `prisma-check`, `migrate`, `coordinate`, `load-image`, `smoke-refusal`, `dockerfile` |
33
33
  | `xema-arm64-supplement-needed` | Decides whether an arm64 supplement is still owed at `REGISTRY/IMAGE:CONTENT_TAG` (env: `REGISTRY`, `IMAGE`, `CONTENT_TAG`, `REGISTRY_USER`, `REGISTRY_TOKEN`). `needed=false` only when the coordinate serves an index whose linux/arm64 child is proven by its own image config; any other resolvable state is `needed=true`; a 401/403/5xx or an unresolvable child is an error. Writes `needed=true\|false` to `$GITHUB_OUTPUT`. Replaces the generated `arm64-supplement-needed.mjs` copy. |
34
- | `xema-promote-image --service <name> --dev-ref <repo> --release-ref <repo> --tag <tag> --commit <sha> --main-ref <ref> --signer-repository <owner/repo> --develop-workflow <file> --release-workflow <file>... --promotion-predicate-out <path> -- <tag command>` | Publishes the develop image's digest under the release tag instead of rebuilding: `reused` (the release digest carries an `image-promotion/v1` attestation signed by a `--release-workflow` identity on main, with exactly one verified statement about the digest and a `ghcr.io/<owner>/dev/<name>` source; no dev-ref probe), `copied` (the proven develop digest, digest-preserving; fails closed if the release digest differs) or `miss` (build it; a reuse whose release digest has no `image-promotion/v1` attestation, a legacy image, is a miss with `reason=reuse-no-promotion` on the stdout line; a signature or identity failure, or a source outside `dev/`, is an error; a copy whose dev digest carries no SLSA v1 provenance at all, an image reused on develop from before provenance, is a miss with `reason=copy-no-develop-provenance`, while provenance that is present but fails a leg is an error). A develop digest is proven only if its SLSA v1 provenance is signed by the develop workflow at `refs/heads/develop`, its revision is an ancestor of the main ref, and the tag command prints the same tag at that revision and at `--commit`; any failed leg is an error, never a miss. On every non-miss result it writes the `https://xema.dev/attestations/image-promotion/v1` predicate for the caller to attest: a copy states the facts just proved; a reuse states this run's `release` and `promotedCommit` and carries `source` and `developProvenance` from the earlier promotion. Writes `result`, `digest`, `source`, `revision`, `evidence`, `provenance_digest`, `promotion_predicate`, `promotion_predicate_type` to `$GITHUB_OUTPUT`. Needs `docker`, `cosign` (major v2, the same as the develop attest step and xema-deploy's verify; it runs `cosign version` first and refuses any other major or an unparsable version), `git` and full history. It does not sign |
35
- | `xema-buildx-builder ensure` | Ensures the runner's persistent, capped buildkitd container (`xema-buildkit-signed-<runner>`, state volume `-state`) is up and is the current buildx builder. Env: `RUNNER_NAME`, `BUILDKIT_IMAGE`, `BUILDKIT_LIMITS` (JSON `cpus`, `memory`, `maxParallelism`, `gcMaxUsedSpace`, all required and positive), optional `BUILDKIT_READY_DEADLINE_SECONDS` (positive integer, default 180). Pinned to the runner's cgroup cpuset; recreated once when the image, caps or config change. Prints `buildx-boot path= secs= builder=`. |
34
+ | `xema-promote-image --service <name> --dev-ref <repo> --release-ref <repo> --tag <tag> --commit <sha> --main-ref <ref> --signer-repository <owner/repo> --develop-workflow <file> --release-workflow <file>... [--release-index-workflow <file>] --promotion-predicate-out <path> -- <tag command>` | Publishes the develop image's digest under the release tag instead of rebuilding: `reused` (the release digest carries an `image-promotion/v1` attestation signed by a `--release-workflow` identity on main, with exactly one verified statement about the digest and a `ghcr.io/<owner>/dev/<name>` source; no dev-ref probe), `copied` (the proven develop digest, digest-preserving; fails closed if the release digest differs) or `miss` (build it; a reuse whose release digest has no `image-promotion/v1` attestation, a legacy image, is a miss with `reason=reuse-no-promotion` on the stdout line; with `--release-index-workflow`, a release tag that names an image index signed by that workflow on main, with exactly one `linux/amd64` child whose own digest passes that same reuse check, is instead `reused` with `evidence=release-index`, `digest` = the amd64 child A, `index_digest` = the index D and no predicate, but only when A's selected promotion statement promotes `--commit` itself (a rerun; a later commit with the same content takes the existing miss and attests its own promotion). On `evidence=release-index` the caller skips sign/attest, tags `:<sha>` and `main-<sha>` to the amd64 subject `digest` (release-arm64 resolves `:<sha>` to it and verifies its promotion), and points only the content tag (and `latest`, if it moves it) at the index `index_digest`. Any failed step keeps the miss, and two amd64 children are an error; a signature or identity failure, or a source outside `dev/`, is an error; a copy whose dev digest carries no SLSA v1 provenance at all, an image reused on develop from before provenance, is a miss with `reason=copy-no-develop-provenance`, while provenance that is present but fails a leg is an error). A develop digest is proven only if its SLSA v1 provenance is signed by the develop workflow at `refs/heads/develop`, its revision is an ancestor of the main ref, and the tag command prints the same tag at that revision and at `--commit`; any failed leg is an error, never a miss. On every non-miss result it writes the `https://xema.dev/attestations/image-promotion/v1` predicate for the caller to attest: a copy states the facts just proved; a reuse states this run's `release` and `promotedCommit` and carries `source` and `developProvenance` from the earlier promotion. Writes `result`, `digest`, `source`, `revision`, `evidence`, `provenance_digest`, `promotion_predicate`, `promotion_predicate_type` to `$GITHUB_OUTPUT`. Needs `docker`, `cosign` (major v2, the same as the develop attest step and xema-deploy's verify; it runs `cosign version` first and refuses any other major or an unparsable version), `git` and full history. It does not sign |
35
+ | `xema-buildx-builder ensure` | Ensures the runner's persistent, capped buildkitd container (`xema-buildkit-signed-<runner>`, state volume `-state`) is up and is the current buildx builder. Env: `RUNNER_NAME`, `BUILDKIT_IMAGE`, `BUILDKIT_LIMITS` (JSON `cpus`, `memory`, `maxParallelism`, `gcMaxUsedSpace`, all required and positive), optional `BUILDKIT_READY_DEADLINE_SECONDS` (positive integer, default 180). The slot is `RUNNER_NAME` with a trailing `-<hostname>` stripped only when it equals this container's own hostname, so a recreated runner container reuses its builder and cache. Pinned to the runner's cgroup cpuset; caps are clamped so every `xema-buildkit-signed-*` builder on the host fits (memory at most half of `docker info` MemTotal divided by N, CPUs at most NCPU divided by N). On each run it removes this verb's labelled builders, and their `-state` volumes, whose recorded runner container is no longer running; unlabelled containers are never touched. Recreated once when the image, effective caps, config or runner container change. Prints `buildx-boot path= secs= builder=`. |
36
36
  | `xema-check-no-vendored-deps` | Fails on a `workspace:` range for a package the repository does not own |
37
37
  | `xema-prepush-sanity` | Lefthook `pre-push`: turbo `lint` and `typecheck` on what the push changes plus dependents (root `lint` only without turbo; typecheck is CI-only there); CI is the proof |
38
38
  | `xema-check-workspace-ranges` | Fails on a `workspace:` range the local package does not satisfy |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@xemahq/repo-build-tooling",
3
- "version": "0.21.0",
3
+ "version": "0.21.2",
4
4
  "description": "Dev-time build tooling shared by every Xema repository. Ships as plain ESM, depending only on @xemahq/release-policy for registry access, so the published artifact is the reviewed source.",
5
5
  "license": "Apache-2.0",
6
6
  "author": "Neuralchowder Inc. <developer@xema.dev> (https://xema.dev)",
@@ -52,6 +52,6 @@
52
52
  "test": "node --test src/buildx-builder.test.mjs && node src/scrub-swagger-plugin-paths.mjs --self-test && node --test src/check-workspace-range-matches-local.test.mjs && node --test src/check-no-vendored-deps.test.mjs && node --test src/readiness.test.mjs && node --test src/composed-image.test.mjs && node --test src/composed-image-coordinate-cli.test.mjs && node --test src/declaration-refusal.test.mjs && node --test src/smoke-refusal.test.mjs && node --test src/load-image.test.mjs && node --test src/composed-image-coordinate.test.mjs && node --test src/composed-image-dockerfile.test.mjs && node --test src/service-bootstrap.test.mjs && node --test src/service-bootstrap-protocols.test.mjs && node --test src/composed-image-prebuilt-dockerfile.test.mjs && node --test src/composed-image-in-image-closure.test.mjs && node --test src/composed-image-stage.test.mjs && node --test src/composed-image-pnpm-context.test.mjs && node --test src/composed-image-sync-injected.test.mjs && node --test src/composed-image-prisma-check.test.mjs && node --test src/composed-migrate.test.mjs && node --test src/promote-image.test.mjs && node --test src/arm64-supplement-needed.test.mjs && node --test src/prepush-sanity.test.mjs"
53
53
  },
54
54
  "xemaRelease": {
55
- "sourceDigest": "sha256:bc219d66a33857aaeb74bad77ae50b0f53228368249eb16803ecfd7ef6a8fe03"
55
+ "sourceDigest": "sha256:417658f5270347a7cfc99992c7004050eb9f0603498c75094f9610da66f6ccb2"
56
56
  }
57
57
  }
@@ -0,0 +1,61 @@
1
+ // Pure host-level decisions of xema-buildx-builder: the stable slot name, the host clamp on the caps and the
2
+ // orphan selection. No docker and no I/O here; the verb feeds in what `docker` reported.
3
+
4
+ export const BUILDER_PREFIX = 'xema-buildkit-signed-';
5
+ export const OWNER_LABEL = 'xema.buildkit.owner';
6
+ export const OWNER_VALUE = 'xema-buildx-builder';
7
+ export const HOST_LABEL = 'xema.buildkit.runner-host';
8
+
9
+ const UNIT = { '': 1, b: 1, k: 1024, m: 1024 ** 2, g: 1024 ** 3 };
10
+ const MEBIBYTE = 1024 ** 2;
11
+ const MEMORY_SHARE = 0.5;
12
+ const CONTAINER_ID = /^[0-9a-f]{12}$/;
13
+
14
+ /**
15
+ * The runner's slot: RUNNER_NAME with a trailing `-<hostname>` stripped, only when that suffix is exactly this
16
+ * container's own hostname (the runner container id GitHub appends to the name). A recreated runner container
17
+ * then keeps its builder and cache. Anything else keeps RUNNER_NAME unchanged: never a guess.
18
+ */
19
+ export function runnerSlot(runnerName, hostname) {
20
+ if (runnerName === undefined || runnerName === '') throw new Error('RUNNER_NAME is empty or unset: the builder key is derived from it');
21
+ const suffix = `-${hostname}`;
22
+ const stripped = hostname !== '' && runnerName.endsWith(suffix) && runnerName.length > suffix.length ? runnerName.slice(0, -suffix.length) : runnerName;
23
+ return stripped.toLowerCase().replace(/[^a-z0-9_.-]/g, '-');
24
+ }
25
+
26
+ export function builderName(slot) {
27
+ return `${BUILDER_PREFIX}${slot}`;
28
+ }
29
+
30
+ /** A validated size such as `8g` in bytes. */
31
+ export function memoryBytes(size) {
32
+ const match = /^([0-9]+)([bkmgBKMG]?)$/.exec(size);
33
+ return Number(match[1]) * UNIT[match[2].toLowerCase()];
34
+ }
35
+
36
+ /**
37
+ * The caps actually applied: the configured limit, clamped so N builders together stay inside the host
38
+ * (half of its memory, all of its CPUs). N counts every signed builder on the host, this one included.
39
+ * Memory is whole MiB (docker's `m` unit); CPUs are rounded down to 2 decimals.
40
+ */
41
+ export function effectiveCaps(limits, host, builders) {
42
+ if (!Number.isInteger(builders) || builders < 1) throw new Error(`builder count must be a positive integer, got ${builders}`);
43
+ const memoryMiB = Math.floor(Math.min(memoryBytes(limits.memory), (host.memTotal * MEMORY_SHARE) / builders) / MEBIBYTE);
44
+ const cpus = Math.floor(Math.min(Number(limits.cpus), host.ncpu / builders) * 100) / 100;
45
+ if (memoryMiB < 1 || cpus <= 0) throw new Error(`host (${host.memTotal} bytes, ${host.ncpu} cpus) is too small for ${builders} signed builders`);
46
+ return { cpus: String(cpus), memory: `${memoryMiB}m` };
47
+ }
48
+
49
+ /**
50
+ * Builders that carry this verb's labels and whose recorded runner container is no longer running.
51
+ * `known` rows: { name, owner, host }; `runningIds`: ids of the running containers on the host.
52
+ * A recorded host that is not a container id (a host-network runner records the machine's hostname) cannot
53
+ * be judged and is never an orphan. Unlabelled containers are never touched; `keep` is the caller's own.
54
+ */
55
+ export function orphanBuilders(known, runningIds, keep) {
56
+ return known.filter((row) =>
57
+ row.name !== keep &&
58
+ row.owner === OWNER_VALUE &&
59
+ CONTAINER_ID.test(row.host) &&
60
+ !runningIds.some((id) => id.startsWith(row.host) || row.host.startsWith(id)));
61
+ }
@@ -33,14 +33,8 @@ export function buildkitdToml(limits) {
33
33
  return `[worker.oci]\n gc = true\n maxUsedSpace = "${limits.gcMaxUsedSpace}"\n max-parallelism = ${limits.maxParallelism}\n`;
34
34
  }
35
35
 
36
- /** The label value: a change of image, caps, cpuset or config recreates the container exactly once. */
37
- export function configHash(image, limits, cpuset, toml, sha256) {
38
- const caps = `cpus=${limits.cpus} memory=${limits.memory} memory-swap=${limits.memory} cpuset=${cpuset}`;
39
- return sha256(`${image}\n${caps}\n${toml}`);
40
- }
41
-
42
- /** The builder key: `xema-buildkit-signed-*` is reserved for signed lanes (T133), so a PR build can never share its cache mounts. The lowercased runner name with anything outside [a-z0-9_.-] mapped to '-'. */
43
- export function builderName(runnerName) {
44
- if (runnerName === undefined || runnerName === '') throw new Error('RUNNER_NAME is empty or unset: the builder key is derived from it');
45
- return `xema-buildkit-signed-${runnerName.toLowerCase().replace(/[^a-z0-9_.-]/g, '-')}`;
36
+ /** The label value: `caps` are the EFFECTIVE caps (after the host clamp); a change of image, caps, cpuset or config recreates the container exactly once. */
37
+ export function configHash(image, caps, cpuset, toml, sha256) {
38
+ const capText = `cpus=${caps.cpus} memory=${caps.memory} memory-swap=${caps.memory} cpuset=${cpuset}`;
39
+ return sha256(`${image}\n${capText}\n${toml}`);
46
40
  }
@@ -13,11 +13,12 @@
13
13
  import { spawnSync } from 'node:child_process';
14
14
  import { createHash } from 'node:crypto';
15
15
  import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
16
- import { tmpdir } from 'node:os';
16
+ import { hostname as osHostname, tmpdir } from 'node:os';
17
17
  import { join } from 'node:path';
18
18
 
19
19
  import { invokedDirectly } from './readiness.mjs';
20
- import { buildkitdToml, builderName, configHash, parseLimits } from './buildx-builder-limits.mjs';
20
+ import { BUILDER_PREFIX, HOST_LABEL, OWNER_LABEL, OWNER_VALUE, builderName, effectiveCaps, orphanBuilders, runnerSlot } from './buildx-builder-host.mjs';
21
+ import { buildkitdToml, configHash, parseLimits } from './buildx-builder-limits.mjs';
21
22
 
22
23
  const ADMITTED_EVENTS = ['push', 'workflow_dispatch'];
23
24
  const ADMITTED_REFS = ['refs/heads/main', 'refs/heads/develop'];
@@ -57,7 +58,8 @@ export async function ensure(env, log = console.log) {
57
58
  const image = env.BUILDKIT_IMAGE;
58
59
  if (image === undefined || image === '') throw new Error('BUILDKIT_IMAGE is required');
59
60
  const limits = parseLimits(env.BUILDKIT_LIMITS);
60
- const key = builderName(env.RUNNER_NAME);
61
+ const hostname = osHostname();
62
+ const key = builderName(runnerSlot(env.RUNNER_NAME, hostname));
61
63
  const cpusetFile = env.BUILDKIT_CPUSET_FILE ?? DEFAULT_CPUSET_FILE;
62
64
  let cpuset = '';
63
65
  try {
@@ -68,31 +70,51 @@ export async function ensure(env, log = console.log) {
68
70
  if (cpuset === '') throw new Error(`${cpusetFile} is missing or empty: cannot bound the builder to the runner's cpuset (cgroup v2 required)`);
69
71
  const deadlineSeconds = parseDeadlineSeconds(env.BUILDKIT_READY_DEADLINE_SECONDS);
70
72
  const toml = buildkitdToml(limits);
71
- const cfg = configHash(image, limits, cpuset, toml, sha256);
72
73
 
73
74
  const scratch = mkdtempSync(join(tmpdir(), 'xema-buildkitd-'));
74
75
  try {
75
76
  const configFile = join(scratch, 'buildkitd.toml');
76
77
  writeFileSync(configFile, toml);
77
78
  const start = Date.now();
79
+
80
+ // Orphans first: a builder whose runner container is gone keeps its caps on the host for ever otherwise.
81
+ const listed = () => docker(['ps', '-a', '--filter', `name=^/${BUILDER_PREFIX}`, '--format', `{{.Names}}|{{.Label "${OWNER_LABEL}"}}|{{.Label "${HOST_LABEL}"}}`]).stdout
82
+ .split('\n').filter((l) => l !== '').map((l) => { const [name, owner, host] = l.split('|'); return { name, owner, host }; });
83
+ const runningIds = docker(['ps', '--format', '{{.ID}}']).stdout.split('\n').filter((l) => l !== '');
84
+ for (const orphan of orphanBuilders(listed(), runningIds, key)) {
85
+ log(`::notice::removing orphaned builder ${orphan.name} (runner container ${orphan.host} is gone) and its state volume`);
86
+ docker(['rm', '-f', orphan.name], { allowFailure: true });
87
+ docker(['volume', 'rm', `${orphan.name}-state`], { allowFailure: true });
88
+ }
89
+
90
+ // Clamp to the host: N builders share it, this one included.
91
+ const info = docker(['info', '--format', '{{.MemTotal}} {{.NCPU}}']).stdout.split(' ').map(Number);
92
+ if (info.length !== 2 || !info.every((n) => Number.isInteger(n) && n > 0)) throw new Error(`docker info did not report MemTotal and NCPU: '${info.join(' ')}'`);
93
+ const present = listed();
94
+ const builders = present.some((row) => row.name === key) ? present.length : present.length + 1;
95
+ const caps = effectiveCaps(limits, { memTotal: info[0], ncpu: info[1] }, builders);
96
+ const cfg = configHash(image, caps, cpuset, toml, sha256);
97
+
78
98
  const create = () => {
79
99
  docker(['create', '--name', key, '--restart', 'unless-stopped', '--privileged',
80
- '--cpus', limits.cpus, '--memory', limits.memory, '--memory-swap', limits.memory, '--cpuset-cpus', cpuset,
100
+ '--cpus', caps.cpus, '--memory', caps.memory, '--memory-swap', caps.memory, '--cpuset-cpus', cpuset,
81
101
  '-v', `${key}-state:/var/lib/buildkit`, '--label', `${LABEL}=${cfg}`,
102
+ '--label', `${OWNER_LABEL}=${OWNER_VALUE}`, '--label', `${HOST_LABEL}=${hostname}`,
82
103
  image, '--config', '/etc/buildkitd.toml']);
83
104
  docker(['cp', configFile, `${key}:/etc/buildkitd.toml`]);
84
105
  docker(['start', key]);
85
106
  };
86
107
 
87
108
  let path;
88
- if (docker(['ps', '-a', '--filter', `name=^/${key}$`, '--format', '{{.Names}}']).stdout === '') {
109
+ if (!present.some((row) => row.name === key)) {
89
110
  path = 'created';
90
111
  create();
91
112
  } else {
92
113
  const have = docker(['inspect', '-f', `{{ index .Config.Labels "${LABEL}" }}`, key]).stdout;
93
114
  const haveImage = docker(['inspect', '-f', '{{.Config.Image}}', key]).stdout;
94
- if (have !== cfg || haveImage !== image) {
95
- log(`::notice::buildkitd ${key} config or image changed (${have || 'none'} ${haveImage} -> ${cfg} ${image}): recreating once, state volume kept`);
115
+ const haveHost = docker(['inspect', '-f', `{{ index .Config.Labels "${HOST_LABEL}" }}`, key]).stdout;
116
+ if (have !== cfg || haveImage !== image || haveHost !== hostname) {
117
+ log(`::notice::buildkitd ${key} config, image or runner container changed (${have || 'none'} ${haveImage} ${haveHost} -> ${cfg} ${image} ${hostname}): recreating once, state volume kept`);
96
118
  docker(['rm', '-f', key]);
97
119
  path = 'recreated';
98
120
  create();
@@ -103,7 +125,7 @@ export async function ensure(env, log = console.log) {
103
125
  docker(['start', key]);
104
126
  }
105
127
  }
106
- log(`buildx-boot ${key}: ${path}; caps cpus=${limits.cpus} memory=${limits.memory} cpuset=${cpuset} max-parallelism=${limits.maxParallelism}; waiting for workers (deadline ${deadlineSeconds}s)`);
128
+ log(`buildx-boot ${key}: ${path}; caps cpus=${caps.cpus} memory=${caps.memory} (limit ${limits.cpus}/${limits.memory}, ${builders} builders on the host) cpuset=${cpuset} max-parallelism=${limits.maxParallelism}; waiting for workers (deadline ${deadlineSeconds}s)`);
107
129
 
108
130
  let check = 0;
109
131
  for (;;) {
@@ -131,7 +153,7 @@ export async function ensure(env, log = console.log) {
131
153
  docker(['buildx', 'inspect', '--bootstrap', key]);
132
154
 
133
155
  const secs = Math.floor((Date.now() - start) / 1000);
134
- const line = `buildx-boot path=${path} secs=${secs} builder=${key}`;
156
+ const line = `buildx-boot path=${path} secs=${secs} builder=${key} cpus=${caps.cpus} memory=${caps.memory}`;
135
157
  log(line);
136
158
  return { path, secs, builder: key, line };
137
159
  } finally {
@@ -1,12 +1,18 @@
1
1
  import assert from 'node:assert/strict';
2
2
  import { spawnSync } from 'node:child_process';
3
3
  import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
4
- import { tmpdir } from 'node:os';
4
+ import { hostname, tmpdir } from 'node:os';
5
5
  import { join } from 'node:path';
6
6
  import test from 'node:test';
7
7
  import { fileURLToPath } from 'node:url';
8
8
 
9
+ import { runnerSlot } from './buildx-builder-host.mjs';
10
+
9
11
  const BIN = fileURLToPath(new URL('./buildx-builder.mjs', import.meta.url));
12
+ const HOST = hostname();
13
+ const OTHER_HOST = `${HOST}-other`;
14
+ const OWN = 'xema-buildkit-signed-owl-runner-1';
15
+ const ownRow = (host = HOST) => `${OWN}|xema-buildx-builder|${host}`;
10
16
  const LIMITS = '{"cpus":"4","memory":"8g","maxParallelism":4,"gcMaxUsedSpace":"5GB"}';
11
17
 
12
18
  // A fake docker on PATH: appends every call to calls.log; `ps` answers $FAKE_PS, `inspect -f ...Labels...` answers $FAKE_LABEL,
@@ -18,8 +24,9 @@ function sandbox() {
18
24
  writeFileSync(fake, `#!/bin/sh
19
25
  echo "$@" >> "${log}"
20
26
  case "$1" in
21
- ps) printf '%s' "$FAKE_PS" ;;
22
- inspect) case "$*" in *Labels*) printf '%s' "$FAKE_LABEL" ;; *Config.Image*) printf '%s' "\${FAKE_IMAGE:-moby/buildkit:v0.1}" ;; *Running*) printf '%s' "$FAKE_RUNNING" ;; esac ;;
27
+ ps) case "$*" in *'{{.ID}}'*) printf '%s' "$FAKE_RUNNING_IDS" ;; *) printf '%s' "$FAKE_LIST" ;; esac ;;
28
+ info) printf '%s' "\${FAKE_INFO:-68719476736 10}" ;;
29
+ inspect) case "$*" in *runner-host*) printf '%s' "$FAKE_HOST" ;; *Labels*) printf '%s' "$FAKE_LABEL" ;; *Config.Image*) printf '%s' "\${FAKE_IMAGE:-moby/buildkit:v0.1}" ;; *Running*) printf '%s' "$FAKE_RUNNING" ;; esac ;;
23
30
  exec) if [ "\${FAKE_EXEC_FAIL:-0}" = 1 ]; then echo "no workers" >&2; exit 1; fi ;;
24
31
  buildx) [ "$2" = inspect ] && [ "$3" != --bootstrap ] && exit 1; exit 0 ;;
25
32
  esac
@@ -31,7 +38,7 @@ exit 0
31
38
  const run = (extra = {}) => {
32
39
  const env = {
33
40
  PATH: `${dir}:${process.env.PATH}`, RUNNER_NAME: 'Owl Runner/1', BUILDKIT_IMAGE: 'moby/buildkit:v0.1',
34
- BUILDKIT_LIMITS: LIMITS, BUILDKIT_CPUSET_FILE: cpuset, GITHUB_EVENT_NAME: 'push', GITHUB_REF: 'refs/heads/develop', ...extra,
41
+ BUILDKIT_LIMITS: LIMITS, FAKE_HOST: HOST, BUILDKIT_CPUSET_FILE: cpuset, GITHUB_EVENT_NAME: 'push', GITHUB_REF: 'refs/heads/develop', ...extra,
35
42
  };
36
43
  for (const [k, v] of Object.entries(env)) if (v === undefined) delete env[k];
37
44
  const r = spawnSync(process.execPath, [BIN, 'ensure'], { env, encoding: 'utf8' });
@@ -97,32 +104,32 @@ test('missing or invalid limits, runner name, image and cpuset fail before any d
97
104
  test('first run creates the capped container pinned to the runner cpuset, then prints the buildx-boot line', () => {
98
105
  const s = sandbox();
99
106
  try {
100
- const r = s.run({ FAKE_PS: '' });
107
+ const r = s.run({});
101
108
  assert.equal(r.status, 0, r.stderr);
102
109
  const create = r.calls.split('\n').find((l) => l.startsWith('create '));
103
110
  assert.match(create, /--name xema-buildkit-signed-owl-runner-1 /);
104
- assert.match(create, /--cpus 4 --memory 8g --memory-swap 8g --cpuset-cpus 10-27/);
111
+ assert.match(create, /--cpus 4 --memory 8192m --memory-swap 8192m --cpuset-cpus 10-27/);
105
112
  assert.match(create, /--privileged/);
106
113
  assert.match(create, /-v xema-buildkit-signed-owl-runner-1-state:\/var\/lib\/buildkit/);
107
114
  assert.match(r.calls, /cp .*buildkitd\.toml xema-buildkit-signed-owl-runner-1:\/etc\/buildkitd\.toml/);
108
115
  assert.match(r.calls, /buildx create --name xema-buildkit-signed-owl-runner-1 --driver remote/);
109
116
  assert.match(r.calls, /buildx use xema-buildkit-signed-owl-runner-1/);
110
- assert.match(r.stdout, /^buildx-boot path=created secs=\d+ builder=xema-buildkit-signed-owl-runner-1$/m);
117
+ assert.match(r.stdout, /^buildx-boot path=created secs=\d+ builder=xema-buildkit-signed-owl-runner-1 cpus=4 memory=8192m$/m);
111
118
  } finally { rmSync(s.dir, { recursive: true, force: true }); }
112
119
  });
113
120
 
114
121
  test('a changed hash recreates once, keeping the state volume; the same hash reuses', () => {
115
122
  const s = sandbox();
116
123
  try {
117
- const first = s.run({ FAKE_PS: '' });
124
+ const first = s.run({});
118
125
  const label = /--label xema\.buildkit\.cfg=([0-9a-f]{64})/.exec(first.calls)[1];
119
126
  rmSync(join(s.dir, 'calls.log'));
120
- const recreated = s.run({ FAKE_PS: 'xema-buildkit-signed-owl-runner-1', FAKE_LABEL: 'stale' });
127
+ const recreated = s.run({ FAKE_LIST: ownRow(), FAKE_LABEL: 'stale' });
121
128
  assert.match(recreated.calls, /rm -f xema-buildkit-signed-owl-runner-1/);
122
129
  assert.doesNotMatch(recreated.calls, /volume rm/);
123
130
  assert.match(recreated.stdout, /path=recreated/);
124
131
  rmSync(join(s.dir, 'calls.log'));
125
- const reused = s.run({ FAKE_PS: 'xema-buildkit-signed-owl-runner-1', FAKE_LABEL: label, FAKE_RUNNING: 'true' });
132
+ const reused = s.run({ FAKE_LIST: ownRow(), FAKE_LABEL: label, FAKE_RUNNING: 'true' });
126
133
  assert.doesNotMatch(reused.calls, /^(create|rm) /m);
127
134
  assert.match(reused.stdout, /path=running/);
128
135
  } finally { rmSync(s.dir, { recursive: true, force: true }); }
@@ -131,19 +138,93 @@ test('a changed hash recreates once, keeping the state volume; the same hash reu
131
138
  test('a matching label but a different running image recreates once', () => {
132
139
  const s = sandbox();
133
140
  try {
134
- const first = s.run({ FAKE_PS: '' });
141
+ const first = s.run({});
135
142
  const label = /--label xema\.buildkit\.cfg=([0-9a-f]{64})/.exec(first.calls)[1];
136
143
  rmSync(join(s.dir, 'calls.log'));
137
- const r = s.run({ FAKE_PS: 'xema-buildkit-signed-owl-runner-1', FAKE_LABEL: label, FAKE_RUNNING: 'true', FAKE_IMAGE: 'moby/buildkit:old' });
144
+ const r = s.run({ FAKE_LIST: ownRow(), FAKE_LABEL: label, FAKE_RUNNING: 'true', FAKE_IMAGE: 'moby/buildkit:old' });
138
145
  assert.match(r.calls, /rm -f xema-buildkit-signed-owl-runner-1/);
139
146
  assert.match(r.stdout, /path=recreated/);
140
147
  } finally { rmSync(s.dir, { recursive: true, force: true }); }
141
148
  });
142
149
 
150
+ test('a recreated runner container (new hostname) recreates its builder once, keeping the state volume', () => {
151
+ const s = sandbox();
152
+ try {
153
+ const first = s.run({});
154
+ const label = /--label xema\.buildkit\.cfg=([0-9a-f]{64})/.exec(first.calls)[1];
155
+ rmSync(join(s.dir, 'calls.log'));
156
+ const r = s.run({ FAKE_LIST: ownRow(OTHER_HOST), FAKE_HOST: OTHER_HOST, FAKE_LABEL: label, FAKE_RUNNING: 'true' });
157
+ assert.match(r.stdout, /path=recreated/);
158
+ assert.doesNotMatch(r.calls, /volume rm/);
159
+ } finally { rmSync(s.dir, { recursive: true, force: true }); }
160
+ });
161
+
162
+ test('runnerSlot strips a trailing -<hostname> only when it is exactly the injected hostname', () => {
163
+ assert.equal(runnerSlot('xema-fox1-2f7dad65784b', '2f7dad65784b'), 'xema-fox1');
164
+ assert.equal(runnerSlot('xema-fox1-2f7dad65784b', 'aaaaaaaaaaaa'), 'xema-fox1-2f7dad65784b');
165
+ assert.equal(runnerSlot('xema-fox1-2f7dad65784b', 'f7dad65784b'), 'xema-fox1-2f7dad65784b', 'a suffix of the id is not the id');
166
+ assert.equal(runnerSlot('xema-fox1-2f7dad65784b', 'ad65784b'), 'xema-fox1-2f7dad65784b');
167
+ assert.equal(runnerSlot('2f7dad65784b', '2f7dad65784b'), '2f7dad65784b', 'the whole name is kept: there is no slot left to strip to');
168
+ assert.equal(runnerSlot('Owl Runner/1', ''), 'owl-runner-1');
169
+ assert.throws(() => runnerSlot('', 'host'), /RUNNER_NAME/);
170
+ });
171
+
172
+ test('the verb names the builder and its volume from the stripped slot', () => {
173
+ const s = sandbox();
174
+ try {
175
+ // HOST is only the value the process under test will read; the case is true whatever it is.
176
+ const r = s.run({ RUNNER_NAME: `xema-fox1-${HOST}` });
177
+ assert.match(r.stdout, /builder=xema-buildkit-signed-xema-fox1 /);
178
+ assert.match(r.calls, /-v xema-buildkit-signed-xema-fox1-state:/);
179
+ } finally { rmSync(s.dir, { recursive: true, force: true }); }
180
+ });
181
+
182
+ test('orphan GC removes only labelled builders whose runner container is gone, with their state volume', () => {
183
+ const s = sandbox();
184
+ try {
185
+ const list = [
186
+ ownRow(),
187
+ 'xema-buildkit-signed-dead|xema-buildx-builder|aaaaaaaaaaaa',
188
+ 'xema-buildkit-signed-alive|xema-buildx-builder|bbbbbbbbbbbb',
189
+ 'xema-buildkit-signed-hostnet|xema-buildx-builder|fox-host',
190
+ 'xema-buildkit-signed-foreign||cccccccccccc',
191
+ ].join('\n');
192
+ const r = s.run({ FAKE_LIST: list, FAKE_RUNNING_IDS: 'bbbbbbbbbbbb\ndddddddddddd', FAKE_HOST: HOST });
193
+ assert.match(r.calls, /^rm -f xema-buildkit-signed-dead$/m);
194
+ assert.match(r.calls, /^volume rm xema-buildkit-signed-dead-state$/m);
195
+ assert.doesNotMatch(r.calls, /rm -f xema-buildkit-signed-(alive|hostnet|foreign)/);
196
+ assert.doesNotMatch(r.stdout, /removing orphaned builder xema-buildkit-signed-owl-runner-1/);
197
+ assert.doesNotMatch(r.calls, /volume rm xema-buildkit-signed-(alive|hostnet|foreign)/);
198
+ } finally { rmSync(s.dir, { recursive: true, force: true }); }
199
+ });
200
+
201
+ test('the caps are clamped so N signed builders fit the host, and the effective caps are printed', () => {
202
+ const s = sandbox();
203
+ try {
204
+ const list = [1, 2, 3].map((n) => `xema-buildkit-signed-other${n}|xema-buildx-builder|bbbbbbbbbbbb`).join('\n');
205
+ // 16 GiB, 8 cpus, 4 builders (3 others + this one): memory 16/2/4 = 2 GiB, cpus 8/4 = 2.
206
+ const r = s.run({ FAKE_LIST: list, FAKE_RUNNING_IDS: 'bbbbbbbbbbbb', FAKE_INFO: '17179869184 8' });
207
+ const create = r.calls.split('\n').find((l) => l.startsWith('create '));
208
+ assert.match(create, /--cpus 2 --memory 2048m --memory-swap 2048m/);
209
+ assert.match(r.stdout, /builder=xema-buildkit-signed-owl-runner-1 cpus=2 memory=2048m$/m);
210
+ } finally { rmSync(s.dir, { recursive: true, force: true }); }
211
+ });
212
+
213
+ test('changed effective caps recreate the builder once', () => {
214
+ const s = sandbox();
215
+ try {
216
+ const first = s.run({});
217
+ const label = /--label xema\.buildkit\.cfg=([0-9a-f]{64})/.exec(first.calls)[1];
218
+ rmSync(join(s.dir, 'calls.log'));
219
+ const r = s.run({ FAKE_LIST: ownRow(), FAKE_LABEL: label, FAKE_RUNNING: 'true', FAKE_INFO: '8589934592 8' });
220
+ assert.match(r.stdout, /path=recreated/);
221
+ } finally { rmSync(s.dir, { recursive: true, force: true }); }
222
+ });
223
+
143
224
  test('a builder that never becomes ready fails at the deadline with the container logs', () => {
144
225
  const s = sandbox();
145
226
  try {
146
- const r = s.run({ FAKE_PS: '', FAKE_EXEC_FAIL: '1', BUILDKIT_READY_DEADLINE_SECONDS: '1' });
227
+ const r = s.run({ FAKE_EXEC_FAIL: '1', BUILDKIT_READY_DEADLINE_SECONDS: '1' });
147
228
  assert.equal(r.status, 1);
148
229
  assert.match(r.stderr, /not ready after 1s/);
149
230
  assert.match(r.calls, /logs --tail 50/);
@@ -22,7 +22,7 @@
22
22
  // nearest commit, is an error; the selected statements must all yield the same source and developProvenance.
23
23
 
24
24
  import { createHash } from 'node:crypto';
25
- import { describeFailure, runTool } from './promote-image-registry.mjs';
25
+ import { DIGEST_PATTERN, describeFailure, isImageIndex, readManifestByDigest, runTool } from './promote-image-registry.mjs';
26
26
 
27
27
  export const GITHUB_SERVER = 'https://github.com';
28
28
  export const GITHUB_OIDC_ISSUER = 'https://token.actions.githubusercontent.com';
@@ -145,7 +145,7 @@ function attestationTypes(subject, label, run) {
145
145
  * Returns `undefined` when the digest carries no promotion attestation (a legacy image); a registry error, a signature or identity
146
146
  * failure, or a source outside `<registry>/<owner>/dev/`, is an error.
147
147
  */
148
- export function reuseEarlierPromotion({ releaseRef, tag, digest, repository, releaseWorkflows, commit }, run = runTool) {
148
+ export function reuseEarlierPromotion({ releaseRef, tag, digest, repository, releaseWorkflows, commit, exactCommit = false }, run = runTool) {
149
149
  const subject = `${releaseRef}@${digest}`;
150
150
  const label = 'reuse';
151
151
  // (a) presence, without identity verification: only the predicate types are read
@@ -165,6 +165,8 @@ export function reuseEarlierPromotion({ releaseRef, tag, digest, repository, rel
165
165
  const nearest = nearestPromotedCommit(
166
166
  new Set(about.map(promoted).filter((value) => typeof value === 'string' && COMMIT.test(value))), commit, subject, label, run,
167
167
  );
168
+ // `exactCommit`: only a statement promoting this very commit counts; an ancestor's is `undefined` (the caller's miss)
169
+ if (exactCommit && nearest !== commit) return undefined;
168
170
  const selected = about.filter((statement) => promoted(statement) === nearest);
169
171
  const admitted = new Set(selected.map((statement) => JSON.stringify(admissible(statement, { subject, tag, digest, releaseRef, repository, label }))));
170
172
  if (admitted.size !== 1) {
@@ -201,6 +203,60 @@ function admissible(statement, { subject, tag, digest, releaseRef, repository, l
201
203
  };
202
204
  }
203
205
 
206
+ // cosign v2's definite "this digest is not signed by that identity" answers; any other failure is an error.
207
+ const NO_MATCHING_SIGNATURE = /\bno (?:matching )?signatures\b/i;
208
+
209
+ /**
210
+ * Whether `subject` carries a signature by `pattern` (GitHub OIDC issuer). A definite "no matching signature" is
211
+ * `false`; a failure to run cosign or any other cosign failure is an error.
212
+ */
213
+ function hasSignature(subject, pattern, label, run) {
214
+ const verified = run('cosign', ['verify', '--certificate-identity-regexp', pattern, '--certificate-oidc-issuer', GITHUB_OIDC_ISSUER, subject]);
215
+ if (verified.error) throw new Error(`${label}: could not run cosign to verify ${subject}: ${verified.error.message}`);
216
+ if (verified.status === 0) return true;
217
+ const detail = describeFailure(verified);
218
+ if (verified.status === 1 && NO_MATCHING_SIGNATURE.test(detail)) return false;
219
+ throw new Error(`${label}: verifying the signature of ${subject} failed: ${detail}`);
220
+ }
221
+
222
+ /**
223
+ * Reuse of a release tag that release-arm64 already moved from main's amd64 digest A to a multi-arch index D. Every
224
+ * step fails closed: `undefined` (the caller's existing miss) unless ALL of
225
+ * (1) `releaseDigest` (what `<releaseRef>:<tag>` resolves to), read by digest, is an image index D;
226
+ * (2) D is signed by `indexWorkflow` of `repository` on main;
227
+ * (3) D has exactly one linux/amd64 child A (two are refused as ambiguous);
228
+ * (4) A carries a verified image-promotion/v1 statement under main's `releaseWorkflows` whose selected (nearest)
229
+ * promotedCommit IS `commit` (a rerun; an ancestor's is the miss), bound to this repository, tag and digest A
230
+ * (`reuseEarlierPromotion`, whose refusals stay errors).
231
+ * Returns `{ digest: A, index: D, source, developProvenance }`.
232
+ */
233
+ export function reuseThroughReleaseIndex({ releaseRef, tag, releaseDigest, repository, releaseWorkflows, indexWorkflow, commit }, run = runTool) {
234
+ const label = 'reuse via release index';
235
+ const index = releaseDigest;
236
+ // (1) an index, read by digest and hash-checked
237
+ const manifest = readManifestByDigest(releaseRef, index, run);
238
+ if (!isImageIndex(manifest)) return undefined;
239
+ // (2) signed by the index workflow on main
240
+ if (!hasSignature(`${releaseRef}@${index}`, workflowIdentityPattern(repository, [indexWorkflow], 'main'), label, run)) return undefined;
241
+ // (3) exactly one linux/amd64 child
242
+ const amd64 = (Array.isArray(manifest.manifests) ? manifest.manifests : [])
243
+ .filter((child) => child?.platform?.os === 'linux' && child.platform.architecture === 'amd64');
244
+ if (amd64.length === 0) return undefined;
245
+ if (amd64.length > 1) {
246
+ throw new Error(`${label}: ambiguous: index ${releaseRef}@${index} has ${amd64.length} linux/amd64 children (${amd64.map((child) => child.digest).join(', ')})`);
247
+ }
248
+ const digest = amd64[0].digest;
249
+ if (typeof digest !== 'string' || !DIGEST_PATTERN.test(digest) || digest === index) {
250
+ throw new Error(`${label}: index ${releaseRef}@${index} names no sha256 linux/amd64 child digest: ${digest}`);
251
+ }
252
+ // (4) the existing promotion verification of A, select-by-promotedCommit included
253
+ // ... restricted to a statement promoting this very `commit` (a rerun): a later commit with the same content must
254
+ // take the existing path and attest its own promotion, never reuse one that carries none
255
+ const earlier = reuseEarlierPromotion({ releaseRef, tag, digest, repository, releaseWorkflows, commit, exactCommit: true }, run);
256
+ if (earlier === undefined) return undefined;
257
+ return { digest, index, ...earlier };
258
+ }
259
+
204
260
  /**
205
261
  * Whether `<devRef>@<digest>` carries any SLSA v1 provenance attestation, listed without verifying a signature. A digest
206
262
  * with none (an image reused on develop from before provenance) is not promotable but not an error; a registry error is.
@@ -9,6 +9,7 @@
9
9
  // behind a rebuild.
10
10
 
11
11
  import { spawnSync } from 'node:child_process';
12
+ import { createHash } from 'node:crypto';
12
13
 
13
14
  export const DIGEST_PATTERN = /^sha256:[0-9a-f]{64}$/;
14
15
 
@@ -61,3 +62,28 @@ export function copyByDigest(source, target, run = runTool) {
61
62
  throw new Error(`copying ${source} to ${target} failed: ${describeFailure(result)}`);
62
63
  }
63
64
  }
65
+
66
+ const INDEX_MEDIA_TYPES = Object.freeze(['application/vnd.oci.image.index.v1+json', 'application/vnd.docker.distribution.manifest.list.v2+json']);
67
+
68
+ /**
69
+ * Reads the manifest `<repository>@<digest>` BY DIGEST (`imagetools inspect --raw`) and returns it parsed, refusing
70
+ * unless the bytes hash to `digest` (one trailing newline the printer may add is not part of the manifest).
71
+ */
72
+ export function readManifestByDigest(repository, digest, run = runTool) {
73
+ const reference = `${repository}@${digest}`;
74
+ const result = run('docker', ['buildx', 'imagetools', 'inspect', reference, '--raw']);
75
+ if (result.error || result.status !== 0) throw new Error(`reading the manifest of ${reference} failed: ${describeFailure(result)}`);
76
+ const raw = String(result.stdout ?? '');
77
+ const hashes = (text) => `sha256:${createHash('sha256').update(text, 'utf8').digest('hex')}` === digest;
78
+ if (!hashes(raw) && !(raw.endsWith('\n') && hashes(raw.slice(0, -1)))) {
79
+ throw new Error(`the manifest read for ${reference} does not hash to ${digest}`);
80
+ }
81
+ try {
82
+ return JSON.parse(raw);
83
+ } catch (error) {
84
+ throw new Error(`the manifest of ${reference} is not JSON`, { cause: error });
85
+ }
86
+ }
87
+
88
+ /** Whether a parsed manifest is an image index (OCI index or Docker manifest list). */
89
+ export const isImageIndex = (manifest) => INDEX_MEDIA_TYPES.includes(manifest?.mediaType);
@@ -7,7 +7,13 @@
7
7
  // `--release-workflow` identities; the statement of the nearest promotedCommit that is an ancestor of
8
8
  // (or equal to) `--commit` is the evidence (as xema-deploy's deploy verifier selects), and its source must be a
9
9
  // `ghcr.io/<owner>/dev/<name>` reference -> reused (evidence=prior-promotion)
10
- // - no image-promotion/v1 attestation (a legacy image) -> miss (reason=reuse-no-promotion): the caller builds
10
+ // - no image-promotion/v1 attestation (a legacy image) -> miss (reason=reuse-no-promotion): the caller builds,
11
+ // unless `--release-index-workflow` is given and the tag names release-arm64's index over main's amd64 digest A:
12
+ // the index is signed by that workflow on main, has exactly one linux/amd64 child A, and A passes this very
13
+ // reuse check with a statement promoting `--commit` itself (a rerun)
14
+ // -> reused (evidence=release-index, digest=A, index_digest=D,
15
+ // no predicate: the caller neither signs nor attests, tags :<sha>/main-<sha> to A and only the content tag (and
16
+ // latest) to D); any failed step keeps the miss
11
17
  // - a signature / identity failure, or a source outside dev/ -> error, never a miss
12
18
  // 2. else `<dev-ref>:<tag>` exists: prove it (legs a, b, c in
13
19
  // promote-image-provenance.mjs), copy it by digest (`imagetools create`
@@ -17,7 +23,7 @@
17
23
  // - provenance present but any leg fails -> error, never a miss
18
24
  // 3. else -> miss: the caller builds.
19
25
  //
20
- // On every non-miss result it writes the promotion predicate
26
+ // On every non-miss result except `evidence=release-index` it writes the promotion predicate
21
27
  // (`https://xema.dev/attestations/image-promotion/v1`) to `--promotion-predicate-out`, for the caller to
22
28
  // `cosign attest` under main's own identity. A copy states the facts it just proved; a reuse states this run's
23
29
  // release and commit and carries `source` and `developProvenance` from main's earlier promotion. It signs
@@ -33,7 +39,7 @@
33
39
  // No retries. Every refusal is an error, never a miss.
34
40
  //
35
41
  // Outputs: `result`, `digest`, `source`, `revision`, `evidence`,
36
- // `provenance_digest`, `promotion_predicate` and `promotion_predicate_type` appended to `$GITHUB_OUTPUT`
42
+ // `provenance_digest`, `promotion_predicate`, `promotion_predicate_type` (and `index_digest` on evidence=release-index) appended to `$GITHUB_OUTPUT`
37
43
  // (required; empty values where a fact does not apply), and one stdout line
38
44
  // `promote-image result=<r> digest=<d> src=<repository>@<d> revision=<sha> evidence=<e>`, with ` reason=<why>` appended
39
45
  // to a miss that has a reason.
@@ -43,13 +49,14 @@ import process from 'node:process';
43
49
  import { describeErrorChain } from '@xemahq/release-policy/registry';
44
50
  import { invokedDirectly } from './readiness.mjs';
45
51
  import { IMAGE_PROMOTION_PREDICATE_TYPE, promotionPredicate, reusedPromotionPredicate, writePromotionPredicate } from './promote-image-predicate.mjs';
46
- import { hasDevelopProvenance, reuseEarlierPromotion, verifyDevelopProvenance } from './promote-image-provenance.mjs';
52
+ import { hasDevelopProvenance, reuseEarlierPromotion, reuseThroughReleaseIndex, verifyDevelopProvenance } from './promote-image-provenance.mjs';
47
53
  import { copyByDigest, probeDigest, runTool } from './promote-image-registry.mjs';
48
54
 
49
55
  export const PromoteResult = Object.freeze({ REUSED: 'reused', COPIED: 'copied', MISS: 'miss' });
50
56
  export const PromoteEvidence = Object.freeze({
51
57
  PRIOR_PROMOTION: 'prior-promotion',
52
58
  DEVELOP_PROVENANCE: 'develop-provenance',
59
+ RELEASE_INDEX: 'release-index',
53
60
  NONE: 'none',
54
61
  });
55
62
  export const PromoteMissReason = Object.freeze({ NO_PROMOTION: 'reuse-no-promotion', NO_DEVELOP_PROVENANCE: 'copy-no-develop-provenance' });
@@ -78,11 +85,13 @@ export function assertCosignMajor(run = runTool) {
78
85
  const USAGE =
79
86
  'usage: xema-promote-image --service <name> --dev-ref <registry/repository> --release-ref <registry/repository> ' +
80
87
  '--tag <content tag> --commit <40-hex> --main-ref <git ref> --signer-repository <owner/repo> ' +
81
- '--develop-workflow <file.yaml> --release-workflow <file.yaml>... --promotion-predicate-out <path> -- <tag command>...';
88
+ '--develop-workflow <file.yaml> --release-workflow <file.yaml>... [--release-index-workflow <file.yaml>] ' +
89
+ '--promotion-predicate-out <path> -- <tag command>...';
82
90
 
83
91
  const SINGLE = Object.freeze(['service', 'dev-ref', 'release-ref', 'tag', 'commit', 'main-ref', 'signer-repository',
84
92
  'develop-workflow', 'promotion-predicate-out']);
85
93
  const REPEATABLE = Object.freeze(['release-workflow']);
94
+ const OPTIONAL = Object.freeze(['release-index-workflow']);
86
95
  // A repository reference with no tag and no digest (those are passed separately).
87
96
  const REPOSITORY_REF = /^[a-z0-9][a-z0-9.-]*(:[0-9]+)?(\/[a-z0-9]+(?:(?:[._]|__|-+)[a-z0-9]+)*)+$/;
88
97
  // The OCI distribution tag grammar.
@@ -104,7 +113,7 @@ export function parsePromoteArguments(argv) {
104
113
  const arg = flags[i];
105
114
  const name = arg.startsWith('--') ? arg.slice(2) : undefined;
106
115
  const repeatable = REPEATABLE.includes(name);
107
- if (name === undefined || (!SINGLE.includes(name) && !repeatable)) throw new Error(`unknown argument ${arg}\n${USAGE}`);
116
+ if (name === undefined || (!SINGLE.includes(name) && !OPTIONAL.includes(name) && !repeatable)) throw new Error(`unknown argument ${arg}\n${USAGE}`);
108
117
  if (!repeatable && options[name] !== undefined) throw new Error(`${arg} given twice`);
109
118
  const value = flags[i + 1];
110
119
  if (value === undefined || value.startsWith('--') || value === '') throw new Error(`${arg} needs a value`);
@@ -123,7 +132,8 @@ export function parsePromoteArguments(argv) {
123
132
  if (!COMMIT.test(options.commit)) refuse('commit', options.commit, 'a 40-hex commit');
124
133
  if (options['main-ref'].startsWith('-')) refuse('main-ref', options['main-ref'], 'a git ref');
125
134
  if (!GITHUB_REPOSITORY.test(options['signer-repository'])) refuse('signer-repository', options['signer-repository'], 'an owner/repo');
126
- for (const file of [options['develop-workflow'], ...options['release-workflow']]) {
135
+ const indexWorkflow = options['release-index-workflow'];
136
+ for (const file of [options['develop-workflow'], ...options['release-workflow'], ...(indexWorkflow === undefined ? [] : [indexWorkflow])]) {
127
137
  if (!WORKFLOW_FILE.test(file)) throw new Error(`workflow '${file}' is not a workflow file name under .github/workflows`);
128
138
  }
129
139
  return {
@@ -136,6 +146,7 @@ export function parsePromoteArguments(argv) {
136
146
  repository: options['signer-repository'],
137
147
  developWorkflow: options['develop-workflow'],
138
148
  releaseWorkflows: options['release-workflow'],
149
+ indexWorkflow,
139
150
  predicateOut: options['promotion-predicate-out'],
140
151
  tagCommand,
141
152
  };
@@ -145,16 +156,31 @@ const MISS = Object.freeze({ result: PromoteResult.MISS, digest: '', source: '',
145
156
 
146
157
  /**
147
158
  * Runs the steps. Returns `{ result, digest, source, evidence, predicate?, revision?, provenanceDigest?, reason? }`;
148
- * `predicate` is set on every non-miss result, `reason` on a miss that has one.
159
+ * `predicate` is set on every non-miss result except `evidence=release-index`, `reason` on a miss that has one.
149
160
  */
150
161
  export function promoteImage(options, run = runTool) {
151
- const { service, devRef, releaseRef, tag, commit, repository, releaseWorkflows } = options;
162
+ const { service, devRef, releaseRef, tag, commit, repository, releaseWorkflows, indexWorkflow } = options;
152
163
  assertCosignMajor(run);
153
164
  const releaseTagged = `${releaseRef}:${tag}`;
154
165
  const released = probeDigest(releaseTagged, run);
155
166
  if (released !== undefined) {
156
167
  const earlier = reuseEarlierPromotion({ releaseRef, tag, digest: released, repository, releaseWorkflows, commit }, run);
157
- if (earlier === undefined) return { ...MISS, reason: PromoteMissReason.NO_PROMOTION };
168
+ if (earlier === undefined) {
169
+ const viaIndex = indexWorkflow === undefined
170
+ ? undefined
171
+ : reuseThroughReleaseIndex({ releaseRef, tag, releaseDigest: released, repository, releaseWorkflows, indexWorkflow, commit }, run);
172
+ if (viaIndex === undefined) return { ...MISS, reason: PromoteMissReason.NO_PROMOTION };
173
+ // release-arm64 already merged A into the index the tag names: nothing to sign or attest, no predicate.
174
+ return {
175
+ result: PromoteResult.REUSED,
176
+ digest: viaIndex.digest,
177
+ source: viaIndex.source.reference,
178
+ evidence: PromoteEvidence.RELEASE_INDEX,
179
+ revision: viaIndex.source.revision,
180
+ provenanceDigest: viaIndex.developProvenance.payloadDigest,
181
+ indexDigest: viaIndex.index,
182
+ };
183
+ }
158
184
  return {
159
185
  result: PromoteResult.REUSED,
160
186
  digest: released,
@@ -196,7 +222,7 @@ export function main(argv, { githubOutput, run = runTool, write = (text) => proc
196
222
  if (existsSync(options.predicateOut)) {
197
223
  throw new Error(`--promotion-predicate-out ${options.predicateOut} already exists; refusing to leave a stale predicate to attest`);
198
224
  }
199
- const { result, digest, source, evidence, predicate, revision, provenanceDigest, reason } = promoteImage(options, run);
225
+ const { result, digest, source, evidence, predicate, revision, provenanceDigest, reason, indexDigest } = promoteImage(options, run);
200
226
  if (predicate !== undefined) writePromotionPredicate(options.predicateOut, predicate);
201
227
  const outputs = {
202
228
  result,
@@ -207,6 +233,7 @@ export function main(argv, { githubOutput, run = runTool, write = (text) => proc
207
233
  provenance_digest: provenanceDigest ?? '',
208
234
  promotion_predicate: predicate === undefined ? '' : options.predicateOut,
209
235
  promotion_predicate_type: predicate === undefined ? '' : IMAGE_PROMOTION_PREDICATE_TYPE,
236
+ ...(indexDigest === undefined ? {} : { index_digest: indexDigest }),
210
237
  };
211
238
  appendFileSync(githubOutput, Object.entries(outputs).map(([key, value]) => `${key}=${value}\n`).join(''));
212
239
  write(`promote-image result=${result} digest=${digest} src=${source} revision=${outputs.revision} evidence=${evidence}${reason === undefined ? '' : ` reason=${reason}`}\n`);
@@ -22,6 +22,7 @@ const PROMOTION_TYPE = 'https://xema.dev/attestations/image-promotion/v1';
22
22
  // A fake `docker`, `cosign` and `git` on PATH, and a fake tag command, all
23
23
  // answering from one JSON state file:
24
24
  // tags / errors / copyDigest `docker buildx imagetools inspect|create`
25
+ // raw[repository@digest] the manifest bytes `docker buildx imagetools inspect --raw` prints
25
26
  // signatures[subject] the identity `cosign verify` finds
26
27
  // downloadErrors[subject] what `cosign download attestation` fails with
27
28
  // attestations[subject] { identity, statements, type? } (type defaults to slsaprovenance1) `cosign verify-attestation` finds
@@ -45,6 +46,11 @@ if (tool === 'docker') {
45
46
  if (args[2] === 'inspect') {
46
47
  const ref = args[3];
47
48
  if (state.errors[ref]) fail(state.errors[ref]);
49
+ if (args[4] === '--raw') {
50
+ if (state.raw?.[ref] === undefined) fail('ERROR: ' + ref + ': not found\\n');
51
+ process.stdout.write(state.raw[ref]);
52
+ process.exit(0);
53
+ }
48
54
  if (!state.tags[ref]) fail('ERROR: ' + ref + ': not found\\n');
49
55
  process.stdout.write(JSON.stringify({ mediaType: 'application/vnd.oci.image.index.v1+json', digest: state.tags[ref] }));
50
56
  process.exit(0);
@@ -203,10 +209,10 @@ function promotionStatement({ subjectDigest = OTHER, predicateType = PROMOTION_T
203
209
 
204
210
  // A release digest main already promoted: its image-promotion/v1 attestation(s), signed by a main workflow.
205
211
  // `lineage` defaults to EARLIER_COMMIT being an ancestor of COMMIT.
206
- function promotedRelease({ source, identity = MAIN_IDENTITY, type = PROMOTION_TYPE, extra, statements, lineage = [[EARLIER_COMMIT, COMMIT]] } = {}) {
212
+ function promotedRelease({ source, identity = MAIN_IDENTITY, type = PROMOTION_TYPE, extra, statements, lineage = [[EARLIER_COMMIT, COMMIT]], promotedCommit } = {}) {
207
213
  return {
208
214
  tags: { [`${RELEASE}:${TAG}`]: OTHER },
209
- attestations: { [`${RELEASE}@${OTHER}`]: { identity, type, statements: statements ?? [promotionStatement({ source, extra })] } },
215
+ attestations: { [`${RELEASE}@${OTHER}`]: { identity, type, statements: statements ?? [promotionStatement({ source, extra, promotedCommit })] } },
210
216
  lineage,
211
217
  };
212
218
  }
@@ -610,3 +616,150 @@ test('an unparsable cosign version is refused', () => {
610
616
  const r = refused({ ...provenDevelop(), cosignVersion: 'GitVersion: devel\n' }, /no parsable GitVersion/);
611
617
  assert.deepEqual(r.state.calls, [['cosign', 'version']]);
612
618
  });
619
+
620
+ // ---- reuse through release-arm64's index: `:<tag>` names index D over main's amd64 digest A (= OTHER) ----
621
+ const ARM64_IDENTITY = 'https://github.com/acme/svc/.github/workflows/release-arm64.yaml@refs/heads/main';
622
+ const ARM64_CHILD = `sha256:${'a'.repeat(64)}`;
623
+ const child = (digest, architecture) => ({ mediaType: 'application/vnd.oci.image.manifest.v1+json', digest, size: 1, platform: { os: 'linux', architecture } });
624
+ const ATTESTATION_CHILD = {
625
+ mediaType: 'application/vnd.oci.image.manifest.v1+json', digest: `sha256:${'b'.repeat(64)}`, size: 1,
626
+ platform: { os: 'unknown', architecture: 'unknown' }, annotations: { 'vnd.docker.reference.type': 'attestation-manifest' },
627
+ };
628
+ const indexBytes = (manifests, mediaType = 'application/vnd.oci.image.index.v1+json') => JSON.stringify({ schemaVersion: 2, mediaType, manifests });
629
+
630
+ // `children` / `mediaType` shape D; `signer` signs D; `amd64` is A's state (promoted by main by default).
631
+ function releasedIndex({ children = [child(OTHER, 'amd64'), child(ARM64_CHILD, 'arm64'), ATTESTATION_CHILD], mediaType, signer = ARM64_IDENTITY, amd64 = promotedRelease({ promotedCommit: COMMIT }), raw } = {}) {
632
+ const bytes = indexBytes(children, mediaType);
633
+ const index = sha(bytes);
634
+ return {
635
+ index,
636
+ state: {
637
+ ...amd64,
638
+ tags: { [`${RELEASE}:${TAG}`]: index },
639
+ raw: { [`${RELEASE}@${index}`]: raw ?? `${bytes}\n` },
640
+ signatures: signer === null ? {} : { [`${RELEASE}@${index}`]: signer },
641
+ attestations: { ...amd64.attestations, [`${RELEASE}@${index}`]: { identity: ARM64_IDENTITY, statements: [provenance(index)] } },
642
+ },
643
+ };
644
+ }
645
+
646
+ function runIndexed(state) {
647
+ const { run, flags, cleanup } = scenario(state);
648
+ const at = flags.indexOf('--promotion-predicate-out');
649
+ try {
650
+ // spread now: it reads the predicate file before cleanup removes it
651
+ return { ...run([...flags.slice(0, at), '--release-index-workflow', 'release-arm64.yaml', ...flags.slice(at)]) };
652
+ } finally {
653
+ cleanup();
654
+ }
655
+ }
656
+
657
+ const MISS_LINE = 'promote-image result=miss digest= src= revision= evidence=none reason=reuse-no-promotion\n';
658
+ const assertIndexMiss = (r) => {
659
+ assert.equal(r.status, 0, r.stderr);
660
+ assert.equal(r.stdout, MISS_LINE);
661
+ assert.equal(r.outputs.result, 'miss');
662
+ assert.equal(r.predicate, undefined);
663
+ assert.deepEqual(creates(r.state), []);
664
+ };
665
+ const assertIndexRefused = (r, pattern) => {
666
+ assert.equal(r.status, 1, r.stdout);
667
+ assert.match(r.stderr, pattern);
668
+ assert.deepEqual(r.outputs, {});
669
+ assert.equal(r.predicate, undefined);
670
+ };
671
+
672
+ test('release index: the tag names release-arm64\'s index over main\'s promoted amd64 digest: reused at A, no predicate, nothing copied', () => {
673
+ const { index, state } = releasedIndex();
674
+ const r = runIndexed(state);
675
+ assert.equal(r.status, 0, r.stderr);
676
+ assert.equal(r.stdout, `promote-image result=reused digest=${OTHER} src=${DEV}@${OTHER} revision=${REVISION} evidence=release-index\n`);
677
+ assert.deepEqual(r.outputs, {
678
+ result: 'reused', digest: OTHER, source: `${DEV}@${OTHER}`, revision: REVISION, evidence: 'release-index',
679
+ provenance_digest: EARLIER_PROVENANCE.payloadDigest, promotion_predicate: '', promotion_predicate_type: '', index_digest: index,
680
+ });
681
+ assert.equal(r.predicate, undefined);
682
+ assert.deepEqual(creates(r.state), []);
683
+ assert.ok(r.state.calls.some((call) => call[0] === 'docker' && call[4] === `${RELEASE}@${index}` && call[5] === '--raw'));
684
+ assert.ok(r.state.calls.some((call) => call[0] === 'cosign' && call[1] === 'verify' && call.at(-1) === `${RELEASE}@${index}`));
685
+ assert.equal(r.state.calls.some((call) => call[0] === 'docker' && call[3] === `${DEV}:${TAG}`), false);
686
+ });
687
+
688
+ test('release index: without --release-index-workflow the index stays the existing miss', () => {
689
+ const { run, cleanup } = scenario(releasedIndex().state);
690
+ try {
691
+ const r = run();
692
+ assertIndexMiss(r);
693
+ assert.equal(r.state.calls.some((call) => call[5] === '--raw'), false);
694
+ } finally {
695
+ cleanup();
696
+ }
697
+ });
698
+
699
+ test('release index check 1: a tag naming a plain image without a promotion stays the existing miss', () => {
700
+ const bytes = JSON.stringify({ schemaVersion: 2, mediaType: 'application/vnd.oci.image.manifest.v1+json', layers: [] });
701
+ const plain = sha(bytes);
702
+ assertIndexMiss(runIndexed({
703
+ tags: { [`${RELEASE}:${TAG}`]: plain }, raw: { [`${RELEASE}@${plain}`]: bytes },
704
+ attestations: { [`${RELEASE}@${plain}`]: { identity: MAIN_IDENTITY, statements: [provenance(plain)] } },
705
+ }));
706
+ });
707
+
708
+ test('release index check 1: manifest bytes that do not hash to the tag digest are refused', () => {
709
+ const { state } = releasedIndex({ raw: indexBytes([child(DIGEST, 'amd64')]) });
710
+ assertIndexRefused(runIndexed(state), /does not hash to sha256:/);
711
+ });
712
+
713
+ test('release index: a tag naming a plain promoted image keeps the existing reuse unchanged (no index read)', () => {
714
+ const r = runIndexed(promotedRelease());
715
+ assert.equal(r.status, 0, r.stderr);
716
+ assert.equal(r.outputs.evidence, 'prior-promotion');
717
+ assert.equal(r.outputs.digest, OTHER);
718
+ assert.equal(r.predicate.promotedCommit, COMMIT);
719
+ assert.equal(r.state.calls.some((call) => call[5] === '--raw'), false);
720
+ });
721
+
722
+ test('release index check 2: an index signed by another main workflow is the existing miss', () => {
723
+ assertIndexMiss(runIndexed(releasedIndex({ signer: MAIN_IDENTITY }).state));
724
+ });
725
+
726
+ test('release index check 2: an unsigned index is the existing miss', () => {
727
+ assertIndexMiss(runIndexed(releasedIndex({ signer: null }).state));
728
+ });
729
+
730
+ test('release index check 2: an index signed by release-arm64 off main is the existing miss', () => {
731
+ assertIndexMiss(runIndexed(releasedIndex({ signer: ARM64_IDENTITY.replace('refs/heads/main', 'refs/heads/develop') }).state));
732
+ });
733
+
734
+ test('release index check 3: an index with no linux/amd64 child is the existing miss', () => {
735
+ assertIndexMiss(runIndexed(releasedIndex({ children: [child(ARM64_CHILD, 'arm64'), ATTESTATION_CHILD] }).state));
736
+ });
737
+
738
+ test('release index check 3: an index with two linux/amd64 children is refused as ambiguous', () => {
739
+ const { state } = releasedIndex({ children: [child(OTHER, 'amd64'), child(DIGEST, 'amd64'), child(ARM64_CHILD, 'arm64')] });
740
+ assertIndexRefused(runIndexed(state), /reuse via release index: ambiguous: .* has 2 linux\/amd64 children/);
741
+ });
742
+
743
+ test('release index check 4: an amd64 child with no promotion attestation is the existing miss', () => {
744
+ const amd64 = { attestations: { [`${RELEASE}@${OTHER}`]: { identity: MAIN_IDENTITY, statements: [provenance(OTHER)] } } };
745
+ assertIndexMiss(runIndexed(releasedIndex({ amd64 }).state));
746
+ });
747
+
748
+ test('release index check 4: an amd64 promotion stating another tag is refused', () => {
749
+ const amd64 = promotedRelease({ promotedCommit: COMMIT, extra: { release: { repository: RELEASE, tag: 'a'.repeat(64), digest: OTHER } } });
750
+ assertIndexRefused(runIndexed(releasedIndex({ amd64 }).state), /states release tag a+, not c+/);
751
+ });
752
+
753
+ test('release index check 4: an amd64 promotion signed off main is refused', () => {
754
+ assertIndexRefused(runIndexed(releasedIndex({ amd64: promotedRelease({ promotedCommit: COMMIT, identity: DEVELOP_IDENTITY }) }).state), /reuse: .* has no image-promotion\/v1 attestation signed by/);
755
+ });
756
+
757
+ test('release index check 4: an amd64 promotion of no ancestor commit is refused', () => {
758
+ assertIndexRefused(runIndexed(releasedIndex({ amd64: promotedRelease({ lineage: [] }) }).state), /reuse: no image-promotion\/v1 statement on .* promotes/);
759
+ });
760
+
761
+ test('release index check 4: an amd64 promotion of an ancestor commit (a later commit, same content) falls through to the existing miss', () => {
762
+ const r = runIndexed(releasedIndex({ amd64: promotedRelease() }).state);
763
+ assertIndexMiss(r);
764
+ assert.equal(r.outputs.index_digest, undefined);
765
+ });