@izykitten/ce 0.0.3-dev.4 → 0.0.3-dev.5

package/README.md

@@ -51,20 +51,6 @@ Values are made safe for a tag: any ASCII character a tag cannot have becomes `-
 
 Prefixes can use guards too. A guard that does not pass leaves the prefix empty, since a prefix cannot be left out, and the prefix's other values are then not needed. `CACHE_PREFIX='{?pr}pr-{pr}'` gives each pull request a cache of its own (`pr-3`) and leaves every other build sharing the unprefixed one; with `CACHE_FROM_DEFAULT`, a PR also reads that shared one, as the prefix is empty on the default branch. The image name cannot use guards.
 
-`{arg.NAME}` is the value the build gets for the build arg `NAME`. That is `--build-arg NAME=value`, or the environment's `NAME` for `--build-arg NAME`, as the engine reads it, so a tag can name a value worked out on the command line:
-
-```sh
-ce docker build --build-arg VERSION="$(cat VERSION)" -t '{image}:{arg.VERSION}' .
-```
-
-The last `--build-arg NAME` wins. One whose variable is not set is passed over by Docker, which keeps an earlier value, while podman and buildah drop the value for the Containerfile's default. An empty value (`--build-arg NAME=`) is no value.
-
-Without a `--build-arg`, `{arg.NAME}` is the Containerfile's default, so `ARG CADDY=2.11.7` before `FROM caddy:$CADDY` can tag the image `caddy-2.11.7` with nothing on the command line. ce reads the Containerfile the engine would: the `-f` file if there is one (Docker's last), else the context's `Dockerfile` (or `dockerfile`), or with podman and buildah its `Containerfile`, then `Dockerfile`. The default is what the stage being built, `--target`'s or else the last, sets with its own `ARG NAME=…`, or the stage it is built from does; else the global `ARG NAME=…` before the first `FROM`, which a bare `ARG NAME`, or a `FROM` line, also gets. With no default anywhere, or an empty one, the build gets an empty `NAME`, so `{arg.NAME}` has no value and a guard leaves its tag out.
-
-`$` references in a default are worked out as the engine does. Docker's BuildKit reads them from the `ARG`s before them, the `--build-arg`s and the `ENV` lines in the stage. A stage built from another gets its values, then its `ENV`s again, which win over its `ARG`s, and runs its `ONBUILD ARG` and `ONBUILD ENV` lines first. podman and buildah expand an instruction from the values before it, so `ARG A=1 V=$A` reads an earlier `A`, and an `ENV`, the base image's too, wins over an `ARG`: inside a stage, ce only works out a reference to a name an `ENV` there sets. Their stage gets the `ARG`s of the stage its `FROM` names as written, so none through `FROM $BASE`. A builtin such as `TARGETPLATFORM` is the builder's, as is `SOURCE_DATE_EPOCH` and a proxy such as `HTTP_PROXY`, which the engine may set itself, unless a `--build-arg` sets it (podman and buildah set `TARGETPLATFORM` from the platform they build, whatever a `--build-arg` says); BuildKit also takes a builtin's own `ARG` default.
-
-The Containerfile is only read when a `{arg.NAME}` needs it. When ce cannot tell the default for certain, it fails with exit 2 and says why: when the Containerfile comes from stdin or a URL, the context is remote, podman or buildah are given several `-f` files or a `.in` one for `cpp`, or with Docker a `# syntax=` directive or `BUILDKIT_SYNTAX` picks a frontend other than `docker/dockerfile` (untagged, `:1`, or `:1.21` and later, which may be pinned by a digest; older ones work out some defaults differently); and when the default refers to a builtin, to a name it cannot know (it may be the base image's `ENV`), or uses a form like `${NAME#x}`; when an `ARG` or `ENV` name has something to expand, such as `ENV $NAME=x`, or the instruction has flags; when an instruction or stage name, or a name after a `$`, has letters other than ASCII's, or an `ARG`, `ENV` or `FROM`, or a heredoc's line, has space other than ASCII's; when a heredoc's delimiter is anything but a word, which may be quoted; when a line ends in more than one carriage return; and with podman or buildah, when the stage is built from one with an `ONBUILD ARG` or `ONBUILD ENV`, which they run only for some image formats; when a `--build-context` replaces a stage it is built from (with podman and buildah, what a `FROM` names); and when podman or buildah's `--env` or `--unsetenv`, which ce does not evaluate, may change a name a default refers to. The default branch's cache scope, with `CACHE_FROM_DEFAULT`, never uses the Containerfile, as that branch may have another one: there a `{arg.NAME}` in the cache prefix needs a `--build-arg`. With podman's `--build-arg-file`, which ce does not read, `{arg.NAME}` needs a `--build-arg NAME` of its own. ce does not see a base image's own `ONBUILD` lines, and Docker's legacy builder (`DOCKER_BUILDKIT=0`) is not modelled.
-
 Guards decide from the run's metadata, so they need it too: outside a Forgejo Actions run, `{?default}` fails like `{default}` would. Only `{env.NAME}` and `{arg.NAME}` work everywhere. A guard that does not pass decides on its own, though: a tag with one is left out, and a prefix with one is empty, even if another guard in it could not be worked out, so `{?default}{?env.X}` outside a run with `X` empty is simply left out. To tell whether a branch is the default one, ce reads `repository.default_branch` from the event file, or `DEFAULT_BRANCH` when the event has none, as on a scheduled run; set `DEFAULT_BRANCH: main` in such a workflow. Pull requests and tags are never the default branch, so they need neither.
 
 Put templates in single quotes: `?` is a glob character in zsh.
@@ -95,6 +81,50 @@ A release on a SemVer tag, with the version aliases only for stable releases:
 
 `{stable}` does not mean newest: rebuilding an older release moves its aliases back to it.
 
+### Build args and the Containerfile
+
+`{arg.NAME}` is the value the build gets for the build arg `NAME`, so a tag can carry a version from the command line or the Containerfile. The Containerfile is only read when a `{arg.NAME}` needs it; builds without one never open it.
+
+With `--build-arg`, it is that value:
+
+```sh
+ce docker build --build-arg VERSION="$(cat VERSION)" -t '{image}:{arg.VERSION}' .
+```
+
+The last `--build-arg NAME` wins, and `--build-arg NAME` alone takes the environment's `NAME`. If that is not set, Docker keeps an earlier value, while podman and buildah fall back to the Containerfile's default. An empty value (`--build-arg NAME=`) is no value.
+
+Without a `--build-arg`, it is the Containerfile's default:
+
+```dockerfile
+ARG CADDY_VERSION=2.11.7
+FROM caddy:builder
+ARG CADDY_VERSION
+RUN xcaddy build v${CADDY_VERSION}
+```
+
+`-t '{image}:caddy-{arg.CADDY_VERSION}'` then tags `caddy-2.11.7`, with nothing on the command line.
+
+**What ce reads**
+
+- **The file:** the `-f` file (with Docker, the last one), else the context's `Dockerfile` (or `dockerfile`); podman and buildah look for `Containerfile` first.
+- **The stage:** the one being built, `--target`'s or else the last. Other stages do not matter, unless this one is built `FROM` one of them, which it then starts from.
+- **The default:** the stage's own `ARG NAME=…`, the last one winning, or that of the stage it is built from; else the global `ARG NAME=…` before the first `FROM`, which a bare `ARG NAME` in the stage, or a `FROM` line, also gets. No default anywhere, or an empty one, is no value, and a guard leaves its tag out.
+- **`$` in a default:** worked out as the engine does, from the `ARG`s, `--build-arg`s and `ENV`s before it, with `$NAME`, `${NAME}`, `${NAME:-x}` and `${NAME:+x}` (and with Docker, `${NAME-x}` and `${NAME+x}`). With Docker, in a stage from `scratch`, a name nothing sets is empty, but for `PATH`.
+- **Builtins** such as `TARGETPLATFORM`, and `SOURCE_DATE_EPOCH` and proxies such as `HTTP_PROXY`, which the engine may set itself, are the builder's, unless a `--build-arg` sets them. Docker also takes a builtin's own `ARG` default, and works out a `SOURCE_DATE_EPOCH` that is not a plain number such as `1700000000` itself.
+- **podman and buildah** differ inside a stage: an `ENV`, the base image's too, wins over an `ARG`, so ce only works out a reference to a name an `ENV` there sets. A stage gets the `ARG`s of the stage its `FROM` names as written, so none through `FROM $BASE`. `TARGETPLATFORM` is always the platform they build.
+
+**What ce refuses**, with exit 2 and the reason:
+
+- **The file is not there to read:** `-f -`, a `-f` URL or a remote context; with podman or buildah, several `-f` files or a `.in` one for `cpp`.
+- **Another frontend:** a `# syntax=` directive or `BUILDKIT_SYNTAX` naming anything but `docker/dockerfile` untagged, `:1`, or `:1.21` and later, which may be pinned by a digest.
+- **A value ce cannot know:** a reference to a builtin, or to a name the base image's `ENV` may set; a form like `${NAME#x}`; with podman or buildah, `--env` or `--unsetenv` changing a name it refers to.
+- **Instructions ce does not read:** an `ARG` or `ENV` name with something to expand, such as `ENV $NAME=x`, or with flags.
+- **Characters other than ASCII's:** letters in an instruction, a stage name or a name after a `$`; space in an `ARG`, `ENV` or `FROM` line or a heredoc's line. Also a line ending in more than one carriage return.
+- **Heredocs ce cannot place:** a delimiter that is not a word, which may be quoted; on its line, a `${}` with space or quotes in its word outside double quotes, or with quotes, `$` or `<` in it anywhere.
+- **Stages that change under it:** a `--build-context` replacing a stage the built one comes from (with podman and buildah, what a `FROM` names); with podman or buildah, a stage built from one with an `ONBUILD ARG` or `ONBUILD ENV`, which they run only for some image formats.
+
+Two places always need a `--build-arg NAME`: a `{arg.NAME}` in the cache prefix with `CACHE_FROM_DEFAULT`, as the default branch may have another Containerfile; and with podman's `--build-arg-file`, which ce does not read. ce does not see a base image's own `ONBUILD` lines, and does not model Docker's legacy builder (`DOCKER_BUILDKIT=0`). Every rule here has a test in `test/containerfile.test.ts`.
+
 ### Checking a template
 
 `--dry-run` before the engine, or `CE_DRY_RUN=true`, prints the build summary and the engine command ce would run, and exits 0 without running it or waiting for the engine. A template that fails still exits 2, so it also works as a check before pushing a workflow. It checks the templates, not the engine: if the engine is not on `PATH`, it warns and still exits 0, so it works on a machine without one:

package/dist/ce.js

@@ -275,7 +275,7 @@ function heredocs(keyword, rest, line, buildkit) {
     } catch {}
   }
   const unsure = () => fail(`ce cannot tell which heredocs the << on line ${line} starts`);
-  if (UNUSUAL_SPACE.test(rest) || rest.replace(PLAIN_BRACES, "").includes("${"))
+  if (UNUSUAL_SPACE.test(rest) || !plainBraces(rest))
     unsure();
   const found = [];
   let split;
@@ -293,7 +293,29 @@ function heredocs(keyword, rest, line, buildkit) {
   }
   return found;
 }
-var PLAIN_BRACES = /\$\{[A-Za-z_][A-Za-z0-9_]*(?:(?::[-+?]|[-+?#%])[^\s"'\\`<${}]*)?\}/gu;
+var PLAIN_BRACES = /\$\{[A-Za-z_][A-Za-z0-9_]*(?:(?::[-+?]|[-+?#%])[^\s"'\\`<${}]*)?\}/uy;
+var QUOTED_BRACES = /\$\{[A-Za-z_][A-Za-z0-9_]*(?:(?::[-+?]|[-+?#%])[^\n"'\\`<${}]*)?\}/uy;
+function plainBraces(text) {
+  let quote = "";
+  for (let i = 0;i < text.length; i++) {
+    const ch = text[i];
+    if (ch === "\\" && quote !== "'")
+      i++;
+    else if (quote && ch === quote)
+      quote = "";
+    else if (!quote && (ch === '"' || ch === "'"))
+      quote = ch;
+    else if (quote !== "'" && text.startsWith("${", i)) {
+      const braces = quote ? QUOTED_BRACES : PLAIN_BRACES;
+      braces.lastIndex = i;
+      const match = braces.exec(text);
+      if (!match)
+        return false;
+      i += match[0].length - 1;
+    }
+  }
+  return true;
+}
 function jsonPrefix(text) {
   let depth = 0;
   for (let i = 0;i < text.length; i++) {
@@ -489,7 +511,10 @@ function expandWord(word, escape, model, lookup) {
   const buildkit = model === "buildkit";
   let i = 0;
   let depth = 0;
+  let unused = 0;
   const value = (name) => {
+    if (unused)
+      return;
     const v = lookup(name);
     if (v instanceof Unknown) {
       if (v.why !== "error")
@@ -528,13 +553,18 @@ function expandWord(word, escape, model, lookup) {
     if (!op)
       unsupported();
     i += op.length;
+    const v = value(name);
+    const set = op.startsWith(":") ? v !== undefined && v !== "" : v !== undefined;
+    const used = op.endsWith("-") ? !set : set;
     if (++depth > 100)
       fail("ce does not evaluate ${} nested this deeply");
+    if (!used)
+      unused++;
     const alternative = until("}");
+    if (!used)
+      unused--;
     depth--;
-    const v = value(name);
-    const set = op.startsWith(":") ? v !== undefined && v !== "" : v !== undefined;
-    return op.endsWith("-") ? set ? v : alternative : set ? alternative : "";
+    return used ? alternative : set ? v : "";
   };
   const until = (end) => {
     let out = "";
@@ -684,11 +714,13 @@ function argDefault(parsed, name, target, buildArg, model = "buildkit", { contex
         const spec = flag.slice(flag.indexOf(" ") + 1);
         const names = flag.startsWith("--unsetenv ") ? spec.includes('"') ? ["*"] : spec.split(",") : [spec.split("=")[0].replace(/-$/u, "")];
         const set = (k) => state.env.set(k, new Unknown("error", `${k} may be changed by ${flag}`));
-        for (const env of names)
+        for (const env of names) {
+          const prefix = flag.startsWith("--env ") && env.endsWith("*") ? env.slice(0, -1) : "";
           if (env.includes("*"))
-            [...state.env.keys()].forEach(set);
+            [...state.env.keys()].filter((k) => k.startsWith(prefix)).forEach(set);
           else
             set(env);
+        }
       }
       const written = findStage(stage.base, index);
       if (swapped !== undefined)
@@ -703,7 +735,7 @@ function argDefault(parsed, name, target, buildArg, model = "buildkit", { contex
         }
       }
     }
-    const see = (env) => (ref) => env.has(ref) ? env.get(ref) : BASE_ENV;
+    const see = (env) => (ref) => env.has(ref) ? env.get(ref) : !buildkit || state.opaque || ref === "PATH" ? BASE_ENV : undefined;
     const own = stage.instructions.filter((i) => !i.onbuild);
     for (const { kind, pairs, line } of [...triggers, ...own]) {
       const before = new Map(state.env);
@@ -1522,6 +1554,19 @@ var SHORT_BOOLS = {
   podman: "qD",
   buildah: "qD"
 };
+var BUILDX_VALUES = "add-host allow annotation attest build-arg build-context builder cache-from cache-to call cgroup-parent cpu-period cpu-quota cpu-shares cpuset-cpus cpuset-mems file iidfile isolation label memory memory-swap metadata-file network no-cache-filter output platform policy print progress provenance resource sbom secret security-opt shm-size ssh tag target ulimit";
+var BUILDAH_VALUES = "add-host annotation arch authfile blob-cache blob-info-cache-dir build-arg build-arg-file build-context cache-from cache-to cache-ttl cap-add cap-drop cdi-config-dir cert-dir cgroup-parent cgroupns compression-format compression-level cpp-flag cpu-period cpu-quota cpu-shares cpuset-cpus cpuset-mems creds cw decryption-key device dns dns-option dns-search env file format from group-add hooks-dir ignorefile iidfile iidfile-raw ipc isolation jobs label layer-label logfile loglevel manifest memory memory-swap metadata-file mount net network os os-feature os-version output override-arch override-os pid platform progress raw-iidfile retry retry-delay runtime-flag rusage-logfile sbom sbom-image-output sbom-image-purl-output sbom-merge-strategy sbom-output sbom-purl-output sbom-scanner-command sbom-scanner-image secret security-opt shm-size sign-by signature-policy source-date-epoch source-policy-file ssh tag target timestamp ulimit unsetannotation unsetenv unsetlabel userns userns-gid-map userns-gid-map-group userns-uid-map userns-uid-map-user uts variant volume";
+var VALUE_FLAGS = {
+  docker: new Set(BUILDX_VALUES.split(" ")),
+  podman: new Set(BUILDAH_VALUES.split(" ")),
+  buildah: new Set(BUILDAH_VALUES.split(" "))
+};
+var SHORT_VALUES = {
+  docker: "cfmot",
+  podman: "cfmotv",
+  buildah: "cfmotv"
+};
+var takesValue = (arg, engine) => arg.startsWith("--") ? VALUE_FLAGS[engine].has(arg.slice(2)) : arg.length === 2 && SHORT_VALUES[engine].includes(arg[1]);
 function ungroup(argv, engine) {
   const end = argv.indexOf("--");
   return argv.flatMap((arg, at) => {
@@ -1579,6 +1624,9 @@ function parseBuildArgs(given, tagPrefix, cachePrefix, {
         buildArgs.set(value, null);
       if (engine !== "docker" && buildArgs.has("TARGETPLATFORM"))
         buildArgs.set("TARGETPLATFORM", null);
+      const epoch = buildArgs.get("SOURCE_DATE_EPOCH");
+      if (engine === "docker" && typeof epoch === "string" && !(/^(?:0|-?[1-9][0-9]*)$/u.test(epoch) && BigInt(epoch) >= -(2n ** 63n) && BigInt(epoch) < 2n ** 63n))
+        buildArgs.set("SOURCE_DATE_EPOCH", null);
     }
     if (field === "dockerfile")
       files.push(value);
@@ -1645,6 +1693,8 @@ function parseBuildArgs(given, tagPrefix, cachePrefix, {
       parts.push(arg);
       summary.dockerfile = short();
       files.push(summary.dockerfile);
+    } else if (takesValue(arg, engine) && i + 1 < argv.length) {
+      parts.push(arg, argv[++i]);
     } else {
       parts.push(arg);
       if (!arg.startsWith("-")) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@izykitten/ce",
-  "version": "0.0.3-dev.4",
+  "version": "0.0.3-dev.5",
   "description": "Container engine wrapper: tag/cache rewriting for docker, podman and buildah, registry login, and a Forgejo OIDC Docker credential helper",
   "type": "module",
   "license": "BSD-3-Clause",