@izykitten/ce 0.0.3-dev.0 → 0.0.3-dev.2

package/README.md

@@ -38,7 +38,7 @@ Tags and the `TAG_PREFIX`/`CACHE_PREFIX` values can use placeholders, filled fro
 | `{image}` | registry host and repository, lowercased, e.g. `code.izy.sh/library/gateway` |
 | `{ref}` | the branch name, `pr-<number>` for a pull request, or the tag name |
 | `{branch}`, `{pr}`, `{tag}` | the branch, PR number or tag, set only when building that kind of ref |
-| `{default}` | set only when building the default branch, never for a pull request |
+| `{default}` | set only when building the default branch, never for a pull request or tag |
 | `{revision}`, `{sha}` | the commit, in full and as its first 7 characters |
 | `{version}`, `{major}`, `{minor}`, `{patch}` | from a SemVer tag such as `v1.2.3` or `1.2.3-rc.1` |
 | `{stable}` | set only for a SemVer tag without a pre-release |
@@ -48,7 +48,7 @@ Tags and the `TAG_PREFIX`/`CACHE_PREFIX` values can use placeholders, filled fro
 
 Values are made safe for a tag: any ASCII character a tag cannot have becomes `-`, so branch `ci/modernize-build` gives `ci-modernize-build`; a non-ASCII character is an error. A value never adds a tag or changes the image name. Prefixes can use `{name}` but not guards.
 
-Guards decide from the run's metadata, so they need it too: outside a Forgejo Actions run, or with its event file missing `repository.default_branch`, `{?default}` fails like `{default}` would. Only `{env.NAME}` works everywhere.
+Guards decide from the run's metadata, so they need it too: outside a Forgejo Actions run, `{?default}` fails like `{default}` would. Only `{env.NAME}` works everywhere. To tell whether a branch is the default one, ce reads `repository.default_branch` from the event file, or `DEFAULT_BRANCH` when the event has none, as on a scheduled run; set `DEFAULT_BRANCH: main` in such a workflow. Pull requests and tags are never the default branch, so they need neither.
 
 Put templates in single quotes: `?` is a glob character in zsh.
 
@@ -64,6 +64,12 @@ A branch, PR and default-branch build:
     -t '{image}:{ref},{?default}latest,{?default}caddy-{env.CADDY_VERSION}' .
 ```
 
+With `CACHE_PREFIX='{ref}'`, each branch and pull request gets a gha cache of its own, so a new one starts cold. With `CACHE_FROM_DEFAULT=true`, or `--cache-from-default` before the engine, each gha `--cache-to` is also read from its scope on the default branch: ce expands the cache prefix as it would be there, so PR 3 reads `pr-3` and `main` and writes only `pr-3`, and with `scope=js` it reads `main-js`. The default branch comes from the event file or `DEFAULT_BRANCH`, as for `{default}`. A prefix with `{sha}` or `{revision}` cannot be worked out for it, and is an error. This only picks what is read: whether it hits depends on what the default branch last wrote.
+
+The same can be written by hand, since ce only prefixes a `--cache-from` whose scope matches a `--cache-to` and passes any other on unchanged: `--cache type=gha,mode=max --cache-from type=gha,scope=main`.
+
+The `--cache-from` that `--cache` makes leaves out a gha `mode=`, which only applies to export, and a `--cache-from` given twice, as the default branch's own scope is on the default branch, is passed once.
+
 A release on a SemVer tag, with the version aliases only for stable releases:
 
 ```sh
@@ -72,6 +78,15 @@ A release on a SemVer tag, with the version aliases only for stable releases:
 
 `{stable}` does not mean newest: rebuilding an older release moves its aliases back to it.
 
+### Checking a template
+
+`--dry-run` before the engine, or `CE_DRY_RUN=true`, prints the build summary and the engine command ce would run, and exits 0 without running it or waiting for the engine. A template that fails still exits 2, so it also works as a check before pushing a workflow:
+
+```sh
+GITHUB_REF=refs/tags/v1.2.3 GITHUB_SERVER_URL=https://code.example GITHUB_REPOSITORY=org/app \
+  ce --dry-run docker buildx build -t '{image}:{ref},{?stable}{major}.{minor}' .
+```
+
 ### OCI labels
 
 With `OCI_LABELS=true`, or `--oci-labels` before the engine (`ce --oci-labels docker buildx build …`, like the prefix options), builds get `org.opencontainers.image.source`, `revision`, `version`, `title` and `description` labels from the run. Labels the run cannot tell are left out. They override a Dockerfile `LABEL` with the same key, and a `--label` you pass wins over them. `created` is not set, so the labels do not change the image on every rebuild.
@@ -84,7 +99,7 @@ The package installs `docker-credential-ce-forgejo` next to `ce`. `ce` puts that
 
 ## Compatibility
 
-These stay the same from release to release, and [test/contract.test.ts](test/contract.test.ts) checks them by running the built package under Node and Bun:
+These behave the same under Node and Bun, and [test/contract.test.ts](test/contract.test.ts) checks them by running the built package under both. A release that changes one says so here:
 
 - commands, flags and the environment variables `login` reads;
 - the arguments the engine receives after tag and cache rewriting, and its exit code, signals and PID (`ce` replaces itself with the engine);
@@ -94,6 +109,8 @@ These stay the same from release to release, and [test/contract.test.ts](test/co
 
 The wording of messages is not part of that and may change.
 
+In 0.0.3, the `--cache-from` that `--cache` makes no longer has a gha `mode=`, which only applies to export, and a `--cache-from` given twice is passed once.
+
 Compared with the Python version:
 
 - The state file is locked with `ce-forgejo-credentials.json.pidlock`, holding the holder's PID and hostname, rather than `flock`, which Node does not have. A lock whose holder died is broken right away if the PID is gone on the same host, otherwise after two minutes. It does not exclude a Python `ce` using the same Docker config at the same moment.

package/dist/ce.js

@@ -298,7 +298,7 @@ class CiMetadata {
         const ref = this.#parsedRef();
         if (ref.kind !== "branch")
           return;
-        return ref.name === this.#defaultBranch() ? ref.name : undefined;
+        return ref.name === this.defaultBranch() ? ref.name : undefined;
       }
       case "revision":
         return this.#require("SHA", `{${name}}`);
@@ -359,11 +359,12 @@ class CiMetadata {
     const ref = this.#parsedRef();
     return ref.kind === "tag" ? SEMVER.exec(ref.name) : null;
   }
-  #defaultBranch() {
-    const repository = this.#payload().repository;
-    const branch = isObject(repository) ? repository.default_branch : undefined;
-    if (typeof branch !== "string" || !branch) {
-      throw new CeError("ce: {default} needs repository.default_branch in the event file, which it does not have");
+  defaultBranch() {
+    const repository = this.#ciEnv("EVENT_PATH") ? this.#payload().repository : undefined;
+    const fromEvent = isObject(repository) ? repository.default_branch : undefined;
+    const branch = typeof fromEvent === "string" && fromEvent || nonEmpty(this.#env.DEFAULT_BRANCH);
+    if (!branch) {
+      throw new CeError("ce: {default} needs repository.default_branch in the event file or DEFAULT_BRANCH, and neither is set");
     }
     return branch;
   }
@@ -383,6 +384,21 @@ class CiMetadata {
     return payload;
   }
 }
+var ON_DEFAULT_BRANCH = new Set(["ref", "branch", "default"]);
+var UNKNOWN_ON_DEFAULT_BRANCH = new Set(["revision", "sha"]);
+function onDefaultBranch(vars, branch) {
+  return {
+    known: (name) => vars.known(name),
+    get(name) {
+      if (ON_DEFAULT_BRANCH.has(name))
+        return branch;
+      if (UNKNOWN_ON_DEFAULT_BRANCH.has(name)) {
+        throw new CeError(`ce: the default branch's cache scope cannot use {${name}}: this run does not know its commit`);
+      }
+      return name === "image" || name.startsWith("env.") ? vars.get(name) : undefined;
+    }
+  };
+}
 
 // src/system.ts
 import { spawnSync } from "node:child_process";
@@ -505,6 +521,11 @@ function checkValue(template, vars, what) {
 }
 function expandValue(template, vars, encode, what) {
   checkValue(template, vars, what);
+  for (const part of parse(template, vars)) {
+    if ("name" in part && vars.get(part.name) === undefined) {
+      throw new CeError(`ce: {${part.name}} has no value here, in the ${what} ${JSON.stringify(template)}`);
+    }
+  }
   return expand(template, vars, encode);
 }
 
@@ -658,6 +679,7 @@ function ghaFields(value) {
   const type = fields?.findLast((field) => fieldKey(field) === "type");
   return type?.slice("type=".length) === "gha" ? fields : null;
 }
+var isGhaCache = (value) => ghaFields(value) !== null;
 var scopeIndex = (fields) => fields.findLastIndex((field) => fieldKey(field) === "scope");
 function cacheScopeName(value) {
   const fields = ghaFields(value);
@@ -691,6 +713,12 @@ function normalizeCacheScope(value, cachePrefix) {
     fields[i] = `scope=${scope}`;
   return csvJoin(fields);
 }
+function withoutCacheMode(value) {
+  const fields = ghaFields(value);
+  if (!fields?.some((field) => fieldKey(field) === "mode"))
+    return value;
+  return csvJoin(fields.filter((field) => fieldKey(field) !== "mode"));
+}
 
 // src/build-args.ts
 function consumeValue(argv, flag) {
@@ -710,7 +738,7 @@ var SUMMARIZED = {
   "--build-arg": "buildArgs",
   "--label": "labels"
 };
-function parseBuildArgs(argv, tagPrefix, cachePrefix, { vars = new CiMetadata, labels = {} } = {}) {
+function parseBuildArgs(argv, tagPrefix, cachePrefix, { vars = new CiMetadata, labels = {}, defaultBranch } = {}) {
   const args = [];
   const rawCacheFrom = [];
   const rawCacheTo = [];
@@ -764,7 +792,7 @@ function parseBuildArgs(argv, tagPrefix, cachePrefix, { vars = new CiMetadata, l
       rawCacheTo.push(attached ?? next());
     else if (name === "--cache") {
       const v = attached ?? next();
-      rawCacheFrom.push(v);
+      rawCacheFrom.push(withoutCacheMode(v));
       rawCacheTo.push(v);
     } else if (Object.hasOwn(SUMMARIZED, name)) {
       const flag = name;
@@ -792,11 +820,17 @@ function parseBuildArgs(argv, tagPrefix, cachePrefix, { vars = new CiMetadata, l
   const added = Object.entries(labels).filter(([key]) => !ownKeys.has(key)).map(([key, value]) => `${key}=${value}`);
   args.unshift(...added.map((label) => `--label=${label}`));
   summary.labels.unshift(...added);
+  const prefixTemplate = cachePrefix;
   if (rawCacheFrom.length || rawCacheTo.length)
     cachePrefix = expandPrefix(cachePrefix, vars, "cache prefix");
   const toNames = new Set(rawCacheTo.map(cacheScopeName));
-  for (const c of rawCacheFrom) {
-    const rewritten = toNames.has(cacheScopeName(c)) ? normalizeCacheScope(c, cachePrefix) : c;
+  const cacheFrom = rawCacheFrom.map((c) => toNames.has(cacheScopeName(c)) ? normalizeCacheScope(c, cachePrefix) : c);
+  const ghaTo = rawCacheTo.filter(isGhaCache);
+  if (defaultBranch && ghaTo.length) {
+    const prefix = expandPrefix(prefixTemplate, onDefaultBranch(vars, defaultBranch()), "cache prefix");
+    cacheFrom.push(...ghaTo.map((c) => normalizeCacheScope(withoutCacheMode(c), prefix)));
+  }
+  for (const rewritten of new Set(cacheFrom)) {
     args.push(`--cache-from=${rewritten}`);
     summary.cacheFrom.push(rewritten);
   }
@@ -1163,17 +1197,46 @@ function envFlag(name) {
     return true;
   throw new CeError(`ce: ${name} must be true or false, not ${JSON.stringify(process.env[name])}`);
 }
-async function runEngine(engine, argv, tagPrefix, cachePrefix, ociLabels) {
+function shellQuote(arg) {
+  return /^[A-Za-z0-9_@%+:,./-][A-Za-z0-9_@%+=:,./-]*$/u.test(arg) ? arg : `'${arg.replaceAll("'", `'\\''`)}'`;
+}
+function engineCommand(engine, args) {
+  if (engine === "docker" || !system.which("gosu"))
+    return [engine, ...args];
+  const user = process.env.USER ?? "";
+  if (!user) {
+    logError("ce: gosu is available but USER is not set; cannot drop privileges. Set USER or remove gosu from PATH.");
+    return null;
+  }
+  return ["gosu", user, engine, ...args];
+}
+async function runEngine(engine, argv, tagPrefix, cachePrefix, options = {}) {
+  const dryRun = options.dryRun ?? envFlag("CE_DRY_RUN");
   const command = findBuildCommand(engine, argv);
   let args = argv;
   let summary;
   if (command) {
     const vars = new CiMetadata;
-    const labels = !command.tagsOnly && (ociLabels ?? envFlag("OCI_LABELS")) ? vars.labels() : {};
-    const parsed = parseBuildArgs(argv.slice(command.start), tagPrefix, cachePrefix, { vars, labels });
+    const labels = !command.tagsOnly && (options.ociLabels ?? envFlag("OCI_LABELS")) ? vars.labels() : {};
+    const fromDefault = !command.tagsOnly && (options.cacheFromDefault ?? envFlag("CACHE_FROM_DEFAULT"));
+    const parsed = parseBuildArgs(argv.slice(command.start), tagPrefix, cachePrefix, {
+      vars,
+      labels,
+      defaultBranch: fromDefault ? () => vars.defaultBranch() : undefined
+    });
     args = [...argv.slice(0, command.start), ...parsed.args];
     summary = parsed.summary;
   }
+  if (dryRun) {
+    const run = engineCommand(engine, args);
+    if (!run)
+      return 1;
+    if (summary)
+      printBuildSummary(engine, args, summary);
+    io.stdout(`${run.map(shellQuote).join(" ")}
+`);
+    return 0;
+  }
   ensureHelperOnPath();
   if (!system.which(engine)) {
     logError(`Engine not found: ${engine}`);
@@ -1183,17 +1246,8 @@ async function runEngine(engine, argv, tagPrefix, cachePrefix, ociLabels) {
     return 1;
   if (summary)
     printBuildSummary(engine, args, summary);
-  if (engine === "docker")
-    return system.exec("docker", ["docker", ...args]);
-  if (system.which("gosu")) {
-    const user = process.env.USER ?? "";
-    if (!user) {
-      logError("ce: gosu is available but USER is not set; cannot drop privileges. Set USER or remove gosu from PATH.");
-      return 1;
-    }
-    return system.exec("gosu", ["gosu", user, engine, ...args]);
-  }
-  return system.exec(engine, [engine, ...args]);
+  const run = engineCommand(engine, args);
+  return run ? system.exec(run[0], run) : 1;
 }
 
 // src/login.ts
@@ -1203,14 +1257,18 @@ function registryGroups() {
   const suffixed = Object.keys(process.env).filter((name) => name.startsWith(prefix) && name.length > prefix.length).sort().map((name) => name.slice(prefix.length));
   return [...process.env.REGISTRY_HOST ? [""] : [], ...suffixed];
 }
+function unsetOrEmpty(name) {
+  return `${name} is ${process.env[name] === undefined ? "unset" : "empty"}`;
+}
 function oidcConfig() {
   const audience = process.env.AUD ?? "";
   if (!audience)
     return null;
   const requestUrl = process.env.ACTIONS_ID_TOKEN_REQUEST_URL ?? "";
   const requestToken = process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN ?? "";
-  if (!(requestUrl && requestToken)) {
-    return "Forgejo OIDC login config is incomplete: AUD requires ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN";
+  const missing = ["ACTIONS_ID_TOKEN_REQUEST_URL", "ACTIONS_ID_TOKEN_REQUEST_TOKEN"].filter((name) => !process.env[name]);
+  if (missing.length) {
+    return `Forgejo OIDC login config is incomplete: AUD is set, but ${missing.map(unsetOrEmpty).join("; ")}`;
   }
   try {
     assertAbsoluteHttpUrl(requestUrl, "must be an absolute HTTP(S) URL");
@@ -1257,36 +1315,46 @@ async function login() {
   let failed = false;
   for (const group of groups) {
     const suffix = group ? `__${group}` : "";
-    const host = process.env[`REGISTRY_HOST${suffix}`] ?? "";
-    const user = process.env[`REGISTRY_USER${suffix}`] ?? "";
-    const password = process.env[`REGISTRY_PASSWORD${suffix}`] ?? "";
+    const hostName = `REGISTRY_HOST${suffix}`;
+    const userName = `REGISTRY_USER${suffix}`;
+    const passwordName = `REGISTRY_PASSWORD${suffix}`;
+    const host = process.env[hostName] ?? "";
+    const user = process.env[userName] ?? "";
+    const password = process.env[passwordName] ?? "";
     const label = group || "default";
-    const incomplete = `Login config for '${label}' is incomplete: host/user/password not all set`;
-    if (!(host && user)) {
-      logError(incomplete);
-      failed = true;
-      continue;
-    }
-    for (const rawRegistry of host.split(/\s+/u).filter(Boolean)) {
-      let authority;
-      let registry;
+    const hosts = host.split(/\s+/u).filter(Boolean);
+    const registries = [];
+    for (const rawRegistry of hosts) {
       try {
-        authority = urlAuthority(rawRegistry);
-        registry = registryServer(rawRegistry);
+        const authority = urlAuthority(rawRegistry);
+        const registry = registryServer(rawRegistry);
+        registries.push({ registry, useOidc: oidc !== null && sameAuthority(authority, oidc.authority) });
       } catch (exc) {
         if (!(exc instanceof CeError))
           throw exc;
-        logError(`Invalid registry host for '${label}' (${rawRegistry}): ${exc.message}`);
+        logError(rawRegistry.includes("@") ? `Invalid registry host for '${label}' in ${hostName}, not shown as it has an "@" and may hold a password` : `Invalid registry host for '${label}' (${rawRegistry}): ${exc.message}`);
         failed = true;
-        continue;
-      }
-      const useOidc = oidc !== null && sameAuthority(authority, oidc.authority);
-      if (!useOidc && !password) {
-        logError(incomplete);
-        failed = true;
-        continue;
       }
-      targets.push({ registry, label, user, password, useOidc });
+    }
+    const missing = [];
+    if (!host)
+      missing.push(unsetOrEmpty(hostName));
+    else if (!hosts.length)
+      missing.push(`${hostName} has no registry in it`);
+    if (!user)
+      missing.push(unsetOrEmpty(userName));
+    const needPassword = registries.filter((r) => !r.useOidc).map((r) => r.registry);
+    if (!password && needPassword.length)
+      missing.push(`${unsetOrEmpty(passwordName)}, which ${needPassword.join(", ")} needs`);
+    if (missing.length) {
+      logError(`Login config for '${label}' is incomplete: ${missing.join("; ")}`);
+      failed = true;
+    }
+    if (!user)
+      continue;
+    for (const { registry, useOidc } of registries) {
+      if (useOidc || password)
+        targets.push({ registry, label, user, password, useOidc });
     }
   }
   let jwt = null;
@@ -1332,7 +1400,8 @@ async function login() {
 
 // src/cli.ts
 function helpText() {
-  return `Usage: ${io.argv0()} [--tag-prefix=<prefix>] [--cache-prefix=<prefix>] [--oci-labels] [command|engine] [args]
+  return `Usage: ${io.argv0()} [--tag-prefix=<prefix>] [--cache-prefix=<prefix>] [--oci-labels]
+          [--cache-from-default] [--dry-run] [command|engine] [args]
 
 Options:
     --tag-prefix=<prefix>
@@ -1348,10 +1417,21 @@ Options:
                       Also available via CACHE_PREFIX environment variable.
     --cache=<spec>    Shorthand for passing <spec> to both --cache-from and
                       --cache-to. Supports the same forms as --cache-from/--cache-to.
+                      A gha cache-from made this way leaves out mode=, which
+                      only applies to export. The same cache-from given twice
+                      is passed once.
+    --cache-from-default
+                      Also read each gha --cache-to from its scope on the
+                      default branch, from the cache prefix as it is there,
+                      so a new branch or PR starts from its cache.
+                      Also available via CACHE_FROM_DEFAULT=true.
     --oci-labels      Add org.opencontainers.image.* labels (source, revision,
                       version, title, description) from the Forgejo Actions
                       run to builds. A --label with the same key wins.
                       Also available via OCI_LABELS=true.
+    --dry-run         Print the build summary and the engine command, and
+                      exit without running it. Template errors still exit 2.
+                      Also available via CE_DRY_RUN=true.
 
 Placeholders, in build tags and in the tag and cache prefixes:
     {name}            The value of name; an error if it has none in this run.
@@ -1361,7 +1441,8 @@ Placeholders, in build tags and in the tag and cache prefixes:
     {ref}             Branch name, pr-<number>, or tag name
     {branch} {pr} {tag}
                       Set only when building that kind of ref
-    {default}         Set only when building the default branch
+    {default}         Set only when building the default branch, named by
+                      the event file or else DEFAULT_BRANCH
     {revision} {sha}  The commit, in full and as 7 characters
     {version} {major} {minor} {patch}
                       From a SemVer tag such as v1.2.3 (v optional)
@@ -1407,7 +1488,7 @@ Environment variables (used by login):
 `;
 }
 async function main(argv) {
-  let ociLabels;
+  const options = {};
   let tagPrefix = process.env.TAG_PREFIX ?? "";
   let cachePrefix = process.env.CACHE_PREFIX ?? process.env.TAG_PREFIX ?? "";
   for (;; ) {
@@ -1421,7 +1502,11 @@ async function main(argv) {
     else if (a?.startsWith("--cache-prefix="))
       [cachePrefix, argv] = [a.slice("--cache-prefix=".length), argv.slice(1)];
     else if (a === "--oci-labels")
-      [ociLabels, argv] = [true, argv.slice(1)];
+      [options.ociLabels, argv] = [true, argv.slice(1)];
+    else if (a === "--cache-from-default")
+      [options.cacheFromDefault, argv] = [true, argv.slice(1)];
+    else if (a === "--dry-run")
+      [options.dryRun, argv] = [true, argv.slice(1)];
     else
       break;
   }
@@ -1435,7 +1520,7 @@ async function main(argv) {
   if (command === "credential-helper")
     return credentialHelper(rest);
   if (isEngine(command))
-    return runEngine(command, rest, tagPrefix, cachePrefix, ociLabels);
+    return runEngine(command, rest, tagPrefix, cachePrefix, options);
   io.stderr(helpText());
   return 1;
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@izykitten/ce",
-  "version": "0.0.3-dev.0",
+  "version": "0.0.3-dev.2",
   "description": "Container engine wrapper: tag/cache rewriting for docker, podman and buildah, registry login, and a Forgejo OIDC Docker credential helper",
   "type": "module",
   "license": "BSD-3-Clause",