@arc-e-tect/api-only-publisher 0.16.0 → 0.16.2

package/README.adoc

@@ -6,7 +6,7 @@
 :source-highlighter: rouge
 // The released version of this component: its snippets use it. The release
 // workflow updates it; do not change it by hand.
-:api-only-publisher-version: 0.16.0
+:api-only-publisher-version: 0.16.2
 
 [NOTE]
 .About this documentation
@@ -700,9 +700,13 @@ A run that would publish nothing at all fails instead of succeeding quietly.
 [#idempotent]
 === Publishing the same version again
 
-`publish` is idempotent: publishing a version again with the same content changes nothing, on every channel.
+`publish` is idempotent: publishing a version again with the same documents changes nothing, on every channel.
 
-That rests on `pack` being reproducible.
+A published version is its documents: the names and SHA-256 hashes that `manifest.json` lists under `files`, which are exactly what a subscriber records in its lockfile.
+A remote channel compares those, not the bytes of what it holds.
+The manifest also records the commit a version was packed from, and that commit's time, so the same documents packed from a later commit are another archive, and the same version.
+
+Packing the same commit again gives the same bytes, because `pack` is reproducible.
 The same documents always pack to the same archive, byte for byte: the archive is written with a fixed mode, no owner and a fixed timestamp, and `manifest.json`'s `producedAt` is not the time `pack` ran but the time of the commit being packed.
 Set `SOURCE_DATE_EPOCH` (seconds since 1970, the https://reproducible-builds.org/specs/source-date-epoch/[reproducible-builds convention ↗]) to choose another; outside a git repository, where there is no commit, it is the only way to make two packs identical, and without it `producedAt` is the current time.
 
@@ -716,11 +720,15 @@ What happens when the destination already has the version:
 |Written again. With the same content the bytes are the same; with different content the version is replaced, and every subscriber whose lockfile recorded the old bytes fails its build until the change is published as a new version.
 
 |`maven` and `nuget` to a URL, `npm`, `github-release`, `gitlab-package`
-|The same bytes: nothing is sent, and the log says `already published, identical`. Different bytes: `publish` fails, naming the version -- a published version is never replaced, so publish the change as a new version.
+|The same documents: nothing is sent, and the log says `already published, identical`, or `with the same documents` when the copy came from another commit.
+Other documents: `publish` fails, naming the version -- a published version is never replaced, so publish the change as a new version.
+A copy that cannot be read, so there is nothing to compare, fails as one with other documents would, saying why it `cannot be compared`.
 `maven` and `gitlab-package` read back every file before uploading any, so a run interrupted half-way is completed by the next one; a GitLab registry that allows duplicate packages, as it does by default, would otherwise replace a version.
-`github-release` compares the asset it would upload with the one the release has, and never overwrites one.
-`npm` compares the integrity hash the registry records, or the tarball's SHA-1 where it records only that, as GitLab's does; its files are packed with fixed modes, so the package is the same bytes whatever the umask of the machine that publishes it.
-A NuGet feed that cannot return the package it has, so there is nothing to compare, fails as a different one would.
+A version completed beside an archive from another commit gets that archive's `manifest.json`, so the two agree on where the version came from; a POM carries no documents, and one already there is left as it is.
+`github-release` reads the manifest in the asset the release has, and never overwrites one.
+`npm` takes the same integrity hash the registry records, or the same tarball SHA-1 where it records only that, as GitLab's does, for the same package; otherwise it downloads the registry's copy and compares its documents.
+Its files are packed with fixed modes, so the package is the same bytes whatever the umask of the machine that publishes it.
+`nuget` reads the manifest in the package the feed returns.
 |===
 
 [#remote-switch]
@@ -832,7 +840,7 @@ A .NET build uses it with `dotnet nuget add source <repository>`, or a `nuget.co
 *A remote feed* is configured by its service index, the URL a feed's settings show and that ends in `index.json`, such as `https://api.nuget.org/v3/index.json`.
 The Publisher reads the service index for its `PackagePublish/2.0.0` endpoint, then pushes the package there with one `PUT`, as `multipart/form-data`, with the key from the environment variable `tokenEnv` names in the `X-NuGet-ApiKey` header.
 The `PUT` goes to that endpoint with a trailing `/`, as `dotnet nuget push` sends it; a GitLab feed recognises the upload only there.
-Before the push, the Publisher reads the feed's copy of the version back (its `PackageBaseAddress/3.0.0` resource): the same package succeeds without a push, a different one fails, and a copy that cannot be read, other than one the feed does not have, fails rather than being pushed over (<<idempotent,Publishing the same version again §>>).
+Before the push, the Publisher reads the feed's copy of the version back (its `PackageBaseAddress/3.0.0` resource): a package with the same documents succeeds without a push, one with other documents fails, and a copy that cannot be read, other than one the feed does not have, fails rather than being pushed over (<<idempotent,Publishing the same version again §>>).
 The check comes first because a feed may accept a different package under a version it already has: GitLab does, by default, and serves the new one from then on.
 A feed that answers the push with 409 -- the version appeared in the meantime -- is read back and compared the same way.
 Without the variable set, the push fails before anything is sent, naming it.
@@ -903,7 +911,7 @@ A deploy token is the least privilege for packages alone: it reaches one project
 
 *The same version again.*
 GitLab allows duplicate packages by default: it accepts a different file under a version it already has, and serves that one from then on.
-So the channel reads every file back before uploading any: an identical file is left alone, a missing one is uploaded, and a different one fails `publish` (<<idempotent,Publishing the same version again §>>).
+So the channel reads every file back before uploading any: a file with the same documents is left alone, a missing one is uploaded, and one with other documents fails `publish` (<<idempotent,Publishing the same version again §>>).
 Turning *Allow duplicates* off for generic packages, in the group's *Settings › Packages and registries*, makes GitLab refuse a replaced file too, from every client; the Publisher keeps a version immutable either way.
 
 *`release: true`* also creates the tag's GitLab release, tagging `$CI_COMMIT_SHA` or, outside CI, the checkout's `HEAD`, and links both files as packages.
@@ -932,7 +940,7 @@ channels:
 *The token* goes as a bearer token, `Authorization: Bearer <token>`, unless `tokenHeader` names another header, which then carries it as it is.
 GitLab's Maven registry takes a job token as `Job-Token`, a personal or project access token as `Private-Token`, and a deploy token as `Deploy-Token`; a GitLab Maven repository URL needs the project's numeric ID, since Gradle reads `%2F` in a path as `/`.
 
-*Every file is read back* before any is uploaded: identical is left alone, missing is uploaded, different fails `publish`, naming the file.
+*Every file is read back* before any is uploaded: the artifact and the manifest with the same documents are left alone, a missing file is uploaded, and other documents fail `publish`, naming the file; the POM carries no documents, and one already there is left alone.
 
 *`maven-metadata.xml`* lists an artifact's versions, which is how Gradle resolves a version range and how Renovate and Dependabot see a new version.
 Nexus, Artifactory and GitHub Packages keep it themselves; GitLab does not for files uploaded this way.
@@ -1309,7 +1317,7 @@ A service subscribes to its own contract to implement it, and to the contracts o
 
 * Versions are per target, so a change to one service's fragments releases only that service's contract.
 A change to a fragment several targets share changes all of them, and `changed` names which, as <<closure,Independent versioning, and the dependency closure §>> describes.
-* Publishing every target on every run is safe: a version that is already published, with the same bytes, is left alone (<<idempotent,Publishing the same version again §>>).
+* Publishing every target on every run is safe: a version that is already published, with the same documents, is left alone, whichever commit it was packed from (<<idempotent,Publishing the same version again §>>).
 What fails is a changed contract whose version was not changed, which is the mistake `changed` exists to catch first.
 * `landscape` is optional: a whole-landscape view for reading, generated on every build and never published, as <<aggregates,Aggregates §>> describes.
 
@@ -2035,7 +2043,7 @@ include::pipelines/github-actions.yml[]
 
 What the two steps lose, and how the pipeline makes up for it:
 
-* *Idempotence.* A channel's remote mode leaves a version the registry has alone when it is identical, and refuses it when it differs; `npm publish` or `dotnet nuget push` of an existing version fails instead.
+* *Idempotence.* A channel's remote mode leaves a version the registry has alone when it holds the same documents, and refuses it when it does not; `npm publish` or `dotnet nuget push` of an existing version fails instead.
 Each publishing step checks for the version first, compares the bytes, and publishes only what is missing, so a re-run is safe and a real conflict still fails.
 * *Maven checksums and metadata.* The local Maven layout holds the artifact, its POM and its manifest, and no `.md5`, `.sha1` or `maven-metadata.xml`.
 `mvn deploy:deploy-file` writes those when it uploads, so the repository ends up as a remote `maven` channel would leave it, and Gradle resolves it.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@arc-e-tect/api-only-publisher",
-  "version": "0.16.0",
+  "version": "0.16.2",
   "description": "Builds, packs and publishes API description documents from a library of reusable fragments.",
   "license": "MIT",
   "author": "Arc-E-Tect",

package/src/channels.js

@@ -17,6 +17,7 @@ const { ChannelError } = require("./channel-error");
 const { publishNuget } = require("./nuget");
 const { publishGitlabPackage } = require("./gitlab");
 const { refuseRemote } = require("./remote");
+const { judgeHeld } = require("./held-documents");
 
 /** What every package and release says it is, unless `description` says otherwise. */
 const DEFAULT_DESCRIPTION = "API description documents for {target}.";
@@ -175,29 +176,29 @@ function publishNpm(archive, manifest, options, log) {
             return { location, distTag, name, files: [location] };
         }
 
-        // Idempotent: a version the registry already has is left alone when it is the
-        // same package, and refused when it is not -- npm never replaces a version.
+        // Idempotent: a version the registry already has is left alone when it holds the
+        // same documents, and refused when it does not -- npm never replaces a version.
         const registry = options.registry ? ["--registry", options.registry] : [];
         const [local] = JSON.parse(execFileSync("npm", ["pack", "--dry-run", "--json"],
             { cwd: workDir, encoding: "utf8", stdio: "pipe" }));
         const published = npmPublishedDist(name, version, registry);
         if (published) {
             // The SHA-512 integrity when the registry records one; the tarball's SHA-1
-            // otherwise, as GitLab's npm registry records only that.
+            // otherwise, as GitLab's npm registry records only that. The same hash is the
+            // same package; any other answer is settled by the documents in the registry's copy.
             const [held, mine] = published.integrity ? [published.integrity, local.integrity]
                 : published.shasum ? [published.shasum, local.shasum] : [null, null];
-            if (!held) {
-                throw new ChannelError(
-                    `npm already has ${name}@${version}, and it cannot be compared: the registry records neither ` +
-                    "an integrity nor a shasum for it. A published version is never replaced; publish the change as a new version.");
-            }
-            if (held === mine) {
+            if (held && held === mine) {
                 log(`-- npm ${name}@${version} is already published, identical; nothing to do`);
                 return { location: name + "@" + version, distTag, name, alreadyPublished: true };
             }
-            throw new ChannelError(
-                `npm already has ${name}@${version}, and it is not this package (${held}, not ${mine}). ` +
+            const refused = (why) => new ChannelError(`npm already has ${name}@${version}, and it ${why}. ` +
                 "A published version is never replaced; publish the change as a new version.");
+            const judged = judgeHeld(npmPublishedPackage(name, version, registry), "tgz", manifest);
+            if (judged.verdict === "unreadable") throw refused(`cannot be compared: ${judged.reason}`);
+            if (judged.verdict === "different") throw refused(held ? `is not this package (${held}, not ${mine})` : "is not this package");
+            log(`-- npm ${name}@${version} is already published, with the same documents; nothing to do`);
+            return { location: name + "@" + version, distTag, name, alreadyPublished: true };
         }
 
         const args = ["publish", "--tag", distTag, ...registry];
@@ -228,6 +229,24 @@ function npmPublishedDist(name, version, registry) {
     }
 }
 
+/** The package npm holds as `name@version`, downloaded with the registry's own credentials. */
+function npmPublishedPackage(name, version, registry) {
+    const dir = fs.mkdtempSync(path.join(os.tmpdir(), "api-only-npm-held-"));
+    try {
+        let packed;
+        try {
+            packed = JSON.parse(execFileSync("npm", ["pack", `${name}@${version}`, "--pack-destination", dir, "--json", ...registry],
+                { encoding: "utf8", stdio: "pipe" }));
+        } catch (error) {
+            const output = `${error.stdout || ""}${error.stderr || ""}`.trim();
+            throw new ChannelError(`could not read ${name}@${version} back from npm to compare: ${output || error.message}`);
+        }
+        return fs.readFileSync(path.join(dir, packed[0].filename));
+    } finally {
+        fs.rmSync(dir, { recursive: true, force: true });
+    }
+}
+
 /**
  * `github-release` -- attach the archive to a GitHub release.
  *
@@ -265,26 +284,29 @@ function publishGithubRelease(archive, manifest, options, log) {
     }
 
     // Idempotent, as the registries are: an asset the release already has is left alone
-    // when it is the same bytes, and refused when it is not. Never --clobber: that would
-    // replace a published contract under the same version.
+    // when it holds the same documents, and refused when it does not. Never --clobber: that
+    // would replace a published contract under the same version.
     const asset = path.basename(archive);
     const assets = exists
         ? gh(["release", "view", tag, "--repo", repository, "--json", "assets", "--jq", ".assets[].name"]).split("\n").map((a) => a.trim())
         : [];
     if (assets.includes(asset)) {
         const dir = fs.mkdtempSync(path.join(os.tmpdir(), "api-only-gh-"));
+        let held;
         try {
             gh(["release", "download", tag, "--repo", repository, "--pattern", asset, "--dir", dir]);
-            if (fs.readFileSync(path.join(dir, asset)).equals(fs.readFileSync(archive))) {
-                log(`-- ${repository} release ${tag} already has ${asset}, identical; nothing to do`);
-                return { location: `${repository}@${tag}`, tag, alreadyPublished: true };
-            }
+            held = fs.readFileSync(path.join(dir, asset));
         } finally {
             fs.rmSync(dir, { recursive: true, force: true });
         }
-        throw new ChannelError(
-            `${repository} release ${tag} already has ${asset}, and it is not this archive. ` +
+        const refused = (why) => new ChannelError(`${repository} release ${tag} already has ${asset}, and it ${why}. ` +
             "A published version is never replaced; publish the change as a new version.");
+        const judged = judgeHeld(held, "tgz", manifest);
+        if (judged.verdict === "unreadable") throw refused(`cannot be compared: ${judged.reason}`);
+        if (judged.verdict === "different") throw refused("is not this archive");
+        const identical = held.equals(fs.readFileSync(archive));
+        log(`-- ${repository} release ${tag} already has ${asset}, ${identical ? "identical" : "with the same documents"}; nothing to do`);
+        return { location: `${repository}@${tag}`, tag, alreadyPublished: true };
     }
     gh(["release", "upload", tag, archive, "--repo", repository]);
     log(`-- Published ${manifest.target} ${manifest.version} to ${repository} release ${tag}`);
@@ -408,41 +430,57 @@ function publishMavenRemote(archive, manifest, options, log) {
     const base = `${artifactDir}/${version}/${artifactId}-${version}`;
 
     const work = (async () => {
-        // Idempotent, file by file: each file the repository already has is left alone when
-        // it is the same bytes, and refuses the whole version when it is not. Every file is
-        // read back before any is uploaded, so a run interrupted after the artifact is
-        // completed by the next one, and a version that differs anywhere is not touched.
+        // Idempotent, by documents: each file the repository already has that carries them --
+        // the artifact and the manifest -- is left alone when it holds the same documents, and
+        // refuses the whole version when it does not; the POM carries none, and is left alone.
+        // Every file is read back before any is uploaded, so a run interrupted after the
+        // artifact is completed by the next one, and a version that differs anywhere is not
+        // touched.
         const files = [
-            { url: `${base}.${extension}`, bytes: fs.readFileSync(archive), type: "application/octet-stream" },
+            { url: `${base}.${extension}`, bytes: fs.readFileSync(archive), type: "application/octet-stream", kind: "tgz" },
             { url: `${base}.pom`, type: "application/xml",
               bytes: Buffer.from(pom(options.groupId, artifactId, version, extension, pomDescription(options, manifest)), "utf8") },
-            { url: `${base}-manifest.json`, type: "application/json",
+            { url: `${base}-manifest.json`, type: "application/json", kind: "json",
               bytes: Buffer.from(JSON.stringify(manifest, null, 2) + "\n", "utf8") },
         ];
+        const refused = (file, why) => new ChannelError(
+            `${options.repository} already has ${options.groupId}:${artifactId}:${version}, ` +
+            `and its ${file.url.split("/").pop()} ${why}. ` +
+            "A published version is never replaced; publish the change as a new version.");
         const missing = [];
+        let identical = true;
+        let heldArchiveManifest = null;
         for (const file of files) {
             const existing = await getFile(file.url, auth);
             if (!existing) {
                 missing.push(file);
-            } else if (!existing.equals(file.bytes)) {
-                throw new ChannelError(
-                    `${options.repository} already has ${options.groupId}:${artifactId}:${version}, ` +
-                    `and its ${file.url.split("/").pop()} is not this one. ` +
-                    "A published version is never replaced; publish the change as a new version.");
+                continue;
             }
+            identical = identical && existing.equals(file.bytes);
+            if (!file.kind) continue;
+            const judged = judgeHeld(existing, file.kind, manifest);
+            if (judged.verdict === "unreadable") throw refused(file, `cannot be compared: ${judged.reason}`);
+            if (judged.verdict === "different") throw refused(file, "is not this one");
+            if (file.kind === "tgz") heldArchiveManifest = judged.held.bytes;
+        }
+        for (const file of missing) {
+            // A version completed beside an artifact from another commit gets that artifact's
+            // manifest, so the two agree on where the version came from.
+            const bytes = file.kind === "json" && heldArchiveManifest ? heldArchiveManifest : file.bytes;
+            await putFile(file.url, bytes, auth, file.type);
         }
-        for (const file of missing) await putFile(file.url, file.bytes, auth, file.type);
         if (options.metadata !== false) {
             await updateMetadata(`${artifactDir}/maven-metadata.xml`, options.groupId, artifactId, manifest, auth, log);
         }
-        return missing.length === 0;
+        return { alreadyPublished: missing.length === 0, identical };
     })();
 
     // The CLI is synchronous throughout; surfacing the promise here would make
     // every caller async for one channel's benefit.
-    return work.then((alreadyPublished) => {
+    return work.then(({ alreadyPublished, identical }) => {
         log(alreadyPublished
-            ? `-- ${options.groupId}:${artifactId}:${version} is already in ${options.repository}, identical; nothing to do`
+            ? `-- ${options.groupId}:${artifactId}:${version} is already in ${options.repository}, ` +
+              `${identical ? "identical" : "with the same documents"}; nothing to do`
             : `-- Deployed ${options.groupId}:${artifactId}:${version} to ${options.repository}`);
         return {
             location: `${base}.${extension}`, coordinates: `${options.groupId}:${artifactId}:${version}@${extension}`,

package/src/gitlab.js

@@ -17,6 +17,7 @@ const { execFileSync } = require("child_process");
 
 const { isPrerelease } = require("./version-policy");
 const { ChannelError } = require("./channel-error");
+const { judgeHeld } = require("./held-documents");
 
 const GITLAB_COM = "https://gitlab.com/api/v4";
 const TOKEN_HEADERS = { private: "PRIVATE-TOKEN", deploy: "DEPLOY-TOKEN", job: "JOB-TOKEN" };
@@ -84,26 +85,33 @@ async function readBack(fileUrl, name, headers) {
     return Buffer.from(await response.arrayBuffer());
 }
 
-function notThisOne(name, manifest) {
-    return new ChannelError(`GitLab already has ${name} in ${manifest.target} ${manifest.version}, and it is not this one; ` +
+function refused(name, manifest, why) {
+    return new ChannelError(`GitLab already has ${name} in ${manifest.target} ${manifest.version}, and it ${why}; ` +
         "a published version is never replaced, so publish the change as a new version");
 }
 
-/** Uploads one file unless GitLab has it: true when it was already there, identical. */
-async function upload(fileUrl, name, bytes, manifest, headers) {
-    const held = await readBack(fileUrl, name, headers);
-    if (held) {
-        if (held.equals(bytes)) return true;
-        throw notThisOne(name, manifest);
-    }
-    const response = await fetch(fileUrl, { method: "PUT", headers, body: bytes });
-    if (response.ok) return false;
-    const refused = `${await answered(response)}`;
+/** The manifest in GitLab's copy of a file, which must describe the same documents as `manifest`. */
+function sameAs(held, file, manifest) {
+    const judged = judgeHeld(held, file.kind, manifest);
+    if (judged.verdict === "unreadable") throw refused(file.name, manifest, `cannot be compared: ${judged.reason}`);
+    if (judged.verdict === "different") throw refused(file.name, manifest, "is not this one");
+    return judged.held;
+}
+
+/**
+ * Uploads one file unless GitLab has it: GitLab's copy when it was already there with the
+ * same documents, or null when it was uploaded now.
+ */
+async function upload(file, manifest, headers) {
+    const held = await readBack(file.url, file.name, headers);
+    if (held) return { held, found: sameAs(held, file, manifest) };
+    const response = await fetch(file.url, { method: "PUT", headers, body: file.bytes });
+    if (response.ok) return null;
+    const answer = `${await answered(response)}`;
     // Refused, perhaps as a duplicate another run published meanwhile: compare that one.
-    const now = await readBack(fileUrl, name, headers).catch(() => null);
-    if (now && now.equals(bytes)) return true;
-    if (now) throw notThisOne(name, manifest);
-    throw new ChannelError(`GitLab refused ${name}: ${refused}.` +
+    const now = await readBack(file.url, file.name, headers).catch(() => null);
+    if (now) return { held: now, found: sameAs(now, file, manifest) };
+    throw new ChannelError(`GitLab refused ${file.name}: ${answer}.` +
         (response.status === 401 || response.status === 403 ? TOKEN_HINT : ""));
 }
 
@@ -159,18 +167,28 @@ function publishGitlabPackage(archive, manifest, options, log) {
     const base = `${settings.url}/projects/${encodeURIComponent(settings.project)}/packages/generic/` +
         `${encodeURIComponent(packageName)}/${encodeURIComponent(manifest.version)}`;
     const files = [
-        { name: path.basename(archive), bytes: require("fs").readFileSync(archive) },
-        { name: "manifest.json", bytes: Buffer.from(JSON.stringify(manifest, null, 2) + "\n") },
+        { name: path.basename(archive), bytes: require("fs").readFileSync(archive), kind: "tgz" },
+        { name: "manifest.json", bytes: Buffer.from(JSON.stringify(manifest, null, 2) + "\n"), kind: "json" },
     ].map((file) => ({ ...file, url: `${base}/${encodeURIComponent(file.name)}` }));
 
     return (async () => {
-        let identical = 0;
+        // Idempotent, by documents: a file GitLab has is left alone when it holds the same
+        // documents, and refused when it does not.
+        let held = 0;
+        let identical = true;
         for (const file of files) {
-            if (await upload(file.url, file.name, file.bytes, manifest, settings.headers)) identical += 1;
+            const copy = await upload(file, manifest, settings.headers);
+            if (!copy) continue;
+            held += 1;
+            identical = identical && copy.held.equals(file.bytes);
+            // A version completed beside an archive from another commit gets that archive's
+            // manifest, so the two agree on where the version came from.
+            if (file.kind === "tgz") files[1].bytes = copy.found.bytes;
         }
-        const alreadyPublished = identical === files.length;
+        const alreadyPublished = held === files.length;
         log(alreadyPublished
-            ? `-- ${packageName} ${manifest.version} is already in GitLab project ${settings.project}, identical; nothing to do`
+            ? `-- ${packageName} ${manifest.version} is already in GitLab project ${settings.project}, ` +
+              `${identical ? "identical" : "with the same documents"}; nothing to do`
             : `-- Published ${packageName} ${manifest.version} to GitLab project ${settings.project}`);
 
         if (options.release === true && !wantsRelease) {

package/src/held-documents.js

@@ -0,0 +1,91 @@
+"use strict";
+
+// What a destination already holds under a version, told apart by its documents.
+//
+// A published version is its documents: the manifest's `files`, each document's name and
+// SHA-256, which is exactly what a subscriber records in its lockfile. Comparing bytes
+// instead says too much: the manifest also records the commit it was packed from and that
+// commit's time, so the same documents packed from a later commit are another archive,
+// and a channel comparing bytes would refuse a version that has not changed.
+
+const { untgz } = require("./tar");
+const { unzip } = require("./zip");
+
+class HeldError extends Error {}
+
+// Where each kind of package keeps the manifest: the packed archive at its root, an npm
+// package below package/, a NuGet package in its content folder.
+const MANIFEST_ENTRIES = ["manifest.json", "package/manifest.json", "contracts/manifest.json"];
+
+function kindOf(bytes) {
+    if (bytes.length >= 2 && bytes[0] === 0x1f && bytes[1] === 0x8b) return "tgz";
+    if (bytes.length >= 4 && bytes.readUInt32LE(0) === 0x04034b50) return "zip";
+    if (/^\s*\{/.test(bytes.toString("utf8", 0, 64))) return "json";
+    throw new HeldError("it is neither a manifest nor a package");
+}
+
+/**
+ * The manifest a destination holds, from the manifest itself or from the package that
+ * carries it: its parsed contents and its bytes, as written.
+ *
+ * @param {Buffer} bytes a manifest.json, a packed archive, an npm package or a .nupkg
+ * @param {"tgz"|"zip"|"json"} [kind] what `bytes` should be; recognised when omitted
+ * @returns {{manifest: object, bytes: Buffer}}
+ */
+function heldManifest(bytes, kind = kindOf(bytes)) {
+    let raw = bytes;
+    if (kind !== "json") {
+        let entries;
+        try {
+            entries = kind === "zip" ? unzip(bytes) : untgz(bytes);
+        } catch (error) {
+            // A truncated package can also end a read out of bounds; it is unreadable either way.
+            throw new HeldError(error.message);
+        }
+        const entry = entries.find((e) => MANIFEST_ENTRIES.includes(e.name));
+        if (!entry) throw new HeldError("it holds no manifest.json");
+        raw = entry.data;
+    }
+    let manifest;
+    try {
+        manifest = JSON.parse(raw.toString("utf8"));
+    } catch (error) {
+        throw new HeldError(`its manifest.json is not JSON: ${error.message}`);
+    }
+    if (!manifest || !Array.isArray(manifest.files)) throw new HeldError("its manifest.json lists no files");
+    return { manifest, bytes: raw };
+}
+
+/** The manifest in `bytes`; see heldManifest. */
+function manifestIn(bytes, kind) {
+    return heldManifest(bytes, kind).manifest;
+}
+
+/**
+ * Whether `held` describes the same documents as `mine`: the same target and version, and
+ * the same document names with the same hashes, in any order. Provenance is not compared.
+ */
+function sameDocuments(held, mine) {
+    if (held.target !== mine.target || String(held.version) !== String(mine.version)) return false;
+    const listing = (manifest) => manifest.files.map((f) => `${f.path}\0${f.sha256}`).sort().join("\n");
+    return listing(held) === listing(mine);
+}
+
+/**
+ * What a destination's copy of a version is, against the manifest being published:
+ * `same` with the manifest it carries, `different`, or `unreadable` with the reason --
+ * something that cannot be compared is never taken for the same version.
+ *
+ * @returns {{verdict: "same", held: {manifest: object, bytes: Buffer}} | {verdict: "different"} | {verdict: "unreadable", reason: string}}
+ */
+function judgeHeld(bytes, kind, manifest) {
+    let held;
+    try {
+        held = heldManifest(bytes, kind);
+    } catch (error) {
+        return { verdict: "unreadable", reason: error.message };
+    }
+    return sameDocuments(held.manifest, manifest) ? { verdict: "same", held } : { verdict: "different" };
+}
+
+module.exports = { heldManifest, manifestIn, sameDocuments, judgeHeld, HeldError };

package/src/nuget.js

@@ -25,6 +25,7 @@ const { execFileSync } = require("child_process");
 const { parse, isPrerelease } = require("./version-policy");
 const { zip } = require("./zip");
 const { ChannelError } = require("./channel-error");
+const { judgeHeld } = require("./held-documents");
 
 /** Where the documents and manifest.json sit inside the package. */
 const CONTENT_FOLDER = "contracts";
@@ -191,15 +192,27 @@ function contentUrl(content, id, version) {
     return `${content.replace(/\/?$/, "/")}${lowerId}/${lowerVersion}/${lowerId}.${lowerVersion}.nupkg`;
 }
 
-/**
- * Whether the feed's copy of this version is these bytes: true or false, or null when the
- * feed offers no way to read it back.
- */
-async function sameAsPublished(content, id, version, bytes, key) {
+/** The feed's copy of this version, or null when the feed offers no way to read it back. */
+async function readableCopy(content, id, version, key) {
     if (!content) return null;
     const response = await fetch(contentUrl(content, id, version), { headers: { "X-NuGet-ApiKey": key } });
     if (!response.ok) return null;
-    return Buffer.from(await response.arrayBuffer()).equals(bytes);
+    return Buffer.from(await response.arrayBuffer());
+}
+
+/**
+ * Whether the feed's copy `held` is this version, by its documents: "identical" for the
+ * same bytes, "same" for the same documents in another package, as from another commit;
+ * anything else is refused, as a published version is never replaced.
+ */
+function judge(held, bytes, id, manifest) {
+    if (held.equals(bytes)) return "identical";
+    const refused = (why) => new ChannelError(`the NuGet feed already has ${id} ${manifest.version}, and it ${why}; ` +
+        "a published version is never replaced, so publish the change as a new version");
+    const judged = judgeHeld(held, "zip", manifest);
+    if (judged.verdict === "unreadable") throw refused(`cannot be compared: ${judged.reason}`);
+    if (judged.verdict === "different") throw refused("is not this package");
+    return "same";
 }
 
 /**
@@ -245,19 +258,17 @@ function publishRemote(archive, manifest, options, id, log) {
         // Idempotent before the push, not only after a 409: a feed that allows duplicate
         // versions -- GitLab's default -- accepts a different package under a version it
         // has, and serves that one from then on.
+        const already = (verdict) => {
+            log(`-- NuGet ${id} ${manifest.version} is already published, ` +
+                `${verdict === "identical" ? "identical" : "with the same documents"}; nothing to do`);
+            return {
+                location: endpoint, id, version: manifest.version, prerelease: isPrerelease(manifest.version),
+                alreadyPublished: true,
+            };
+        };
         if (content) {
             const held = await publishedCopy(content, id, manifest.version, key);
-            if (held && held.equals(bytes)) {
-                log(`-- NuGet ${id} ${manifest.version} is already published, identical; nothing to do`);
-                return {
-                    location: endpoint, id, version: manifest.version, prerelease: isPrerelease(manifest.version),
-                    alreadyPublished: true,
-                };
-            }
-            if (held) {
-                throw new ChannelError(`the NuGet feed already has ${id} ${manifest.version}, and it is not this package; ` +
-                    "a published version is never replaced, so publish the change as a new version");
-            }
+            if (held) return already(judge(held, bytes, id, manifest));
         }
         const boundary = `api-only-${crypto.createHash("sha256").update(bytes).digest("hex").slice(0, 32)}`;
         const body = Buffer.concat([
@@ -276,18 +287,11 @@ function publishRemote(archive, manifest, options, id, log) {
             body,
         });
         if (response.status === 409) {
-            // Idempotent: the same package again is success, a different one is not.
-            const same = await sameAsPublished(content, id, manifest.version, bytes, key);
-            if (same) {
-                log(`-- NuGet ${id} ${manifest.version} is already published, identical; nothing to do`);
-                return {
-                    location: endpoint, id, version: manifest.version, prerelease: isPrerelease(manifest.version),
-                    alreadyPublished: true,
-                };
-            }
-            throw new ChannelError(`the NuGet feed already has ${id} ${manifest.version}` +
-                (same === false ? ", and it is not this package" : ", and it cannot be read back to compare") +
-                "; a published version is never replaced, so publish the change as a new version");
+            // Idempotent: the same documents again are success, others are not.
+            const held = await readableCopy(content, id, manifest.version, key);
+            if (held) return already(judge(held, bytes, id, manifest));
+            throw new ChannelError(`the NuGet feed already has ${id} ${manifest.version}, and it cannot be read back to compare; ` +
+                "a published version is never replaced, so publish the change as a new version");
         }
         if (!response.ok) {
             throw new ChannelError(`the NuGet feed refused the package: ${response.status} ${response.statusText}. ` +

package/src/tar.js

@@ -63,4 +63,40 @@ function tgz(entries, mtime) {
     return gzipped;
 }
 
-module.exports = { tgz, TarError };
+function text(block, start, length) {
+    const end = block.indexOf(0, start);
+    return block.toString("utf8", start, end < 0 || end > start + length ? start + length : end);
+}
+
+/**
+ * The regular files in a gzipped tar archive, in order: what `tgz` writes, and what npm
+ * packs, with a ustar prefix joined to the name. Directories, links and pax headers are
+ * skipped.
+ *
+ * @param {Buffer} bytes
+ * @returns {Array<{name: string, data: Buffer}>}
+ */
+function untgz(bytes) {
+    let tar;
+    try {
+        tar = zlib.gunzipSync(bytes);
+    } catch (error) {
+        throw new TarError(`not a gzipped tar archive: ${error.message}`);
+    }
+    const entries = [];
+    for (let offset = 0; offset + BLOCK <= tar.length;) {
+        const block = tar.subarray(offset, offset + BLOCK);
+        if (block.every((byte) => byte === 0)) break;
+        const prefix = block.toString("ascii", 257, 262) === "ustar" ? text(block, 345, 155) : "";
+        const name = prefix ? `${prefix}/${text(block, 0, 100)}` : text(block, 0, 100);
+        const size = parseInt(text(block, 124, 12).trim() || "0", 8);
+        const start = offset + BLOCK;
+        if (start + size > tar.length) throw new TarError(`the archive ends inside ${name}`);
+        const type = String.fromCharCode(block[156]);
+        if (type === "0" || type === "\0") entries.push({ name, data: Buffer.from(tar.subarray(start, start + size)) });
+        offset = start + Math.ceil(size / BLOCK) * BLOCK;
+    }
+    return entries;
+}
+
+module.exports = { tgz, untgz, TarError };

package/src/zip.js

@@ -100,4 +100,45 @@ function zip(entries) {
     return Buffer.concat([...locals, directory, end]);
 }
 
-module.exports = { zip, crc32, ZipError };
+/**
+ * The entries of a ZIP archive, from its central directory, each inflated and checked
+ * against its CRC-32: what `zip` writes, and what a NuGet feed returns of it.
+ *
+ * @param {Buffer} bytes
+ * @returns {Array<{name: string, data: Buffer}>}
+ */
+function unzip(bytes) {
+    const end = bytes.lastIndexOf(Buffer.from([0x50, 0x4b, 0x05, 0x06]));
+    if (end < 0 || end + 22 > bytes.length) throw new ZipError("not a ZIP archive: it has no end of central directory record");
+    const entries = [];
+    let offset = bytes.readUInt32LE(end + 16);
+    for (let i = 0; i < bytes.readUInt16LE(end + 10); i++) {
+        if (offset + 46 > bytes.length || bytes.readUInt32LE(offset) !== 0x02014b50) {
+            throw new ZipError(`not a ZIP archive: no central directory entry at ${offset}`);
+        }
+        const method = bytes.readUInt16LE(offset + 10);
+        const crc = bytes.readUInt32LE(offset + 16);
+        const compressed = bytes.readUInt32LE(offset + 20);
+        const nameLength = bytes.readUInt16LE(offset + 28);
+        const local = bytes.readUInt32LE(offset + 42);
+        const name = bytes.toString("utf8", offset + 46, offset + 46 + nameLength);
+        if (local + 30 > bytes.length || bytes.readUInt32LE(local) !== 0x04034b50) {
+            throw new ZipError(`not a ZIP archive: no local header for ${name}`);
+        }
+        const start = local + 30 + bytes.readUInt16LE(local + 26) + bytes.readUInt16LE(local + 28);
+        const raw = bytes.subarray(start, start + compressed);
+        let data;
+        try {
+            data = method === 8 ? zlib.inflateRawSync(raw) : method === 0 ? Buffer.from(raw) : null;
+        } catch (error) {
+            throw new ZipError(`${name} cannot be inflated: ${error.message}`);
+        }
+        if (data === null) throw new ZipError(`${name} is compressed with method ${method}, which is not supported`);
+        if (crc32(data) !== crc) throw new ZipError(`${name} does not match its CRC-32`);
+        entries.push({ name, data });
+        offset += 46 + nameLength + bytes.readUInt16LE(offset + 30) + bytes.readUInt16LE(offset + 32);
+    }
+    return entries;
+}
+
+module.exports = { zip, unzip, crc32, ZipError };