@arc-e-tect/api-only-publisher 0.3.0 → 0.4.0

package/README.adoc

@@ -190,6 +190,7 @@ A `{{token}}` is replaced with the contents of `<token>.md`, found by searching
 Indentation is preserved, so multi-line Markdown stays valid inside an indented YAML scalar.
 
 . **Bundle.** `@redocly/cli bundle` or `@asyncapi/cli bundle` resolves every `$ref` into one flat, self-contained document under `<build.dist>/<target>/`.
+An OpenAPI document's components each carry the path of the fragment they came from, as <<fragment-paths,`x-fragment-path` §>> describes.
 
 . **Stamp the version** of each published target on its *finished* document: the version its version file declares, with any `--pre-release` identifiers appended.
 A `publish: false` target keeps the version its source declares.
@@ -376,6 +377,7 @@ defaults:
   openapi:
     lint: .redocly.yaml
     outputName: openapi.yaml
+    fragmentPaths: true  # x-fragment-path on every component; the default
   asyncapi:
     outputName: asyncapi.yaml
   placeholders:
@@ -463,6 +465,69 @@ It is deliberately not a re-serialisation: re-emitting a document reformats ever
 Splicing keeps every other byte exactly as the bundler wrote it.
 A document with no `info` block, or no version inside it, is an error.
 
+[#fragment-paths]
+=== Every component says which fragment it came from
+
+Bundling flattens the library's directories into one `components` namespace, and the bundler names each component after its file.
+Two fragments with the same file name in different directories become `UserV1` and `UserV1-2`, and nothing in the document says which is which.
+
+So every component the bundler builds from a fragment carries that fragment's path, relative to `sources.root`, as its first key:
+
+[source,yaml]
+----
+components:
+  schemas:
+    UsernameV1:
+      x-fragment-path: openapi/components/common/schemas/UsernameV1.yaml
+      type: string
+----
+
+The path always uses forward slashes, and it names a file in the library.
+Other OpenAPI tooling ignores the key, as it ignores any `x-` extension.
+It is how the API-Only TranscriberJ ties generated code to the fragment it came from, instead of to a component name the bundler picked.
+
+It is on by default, for OpenAPI documents only.
+`defaults.openapi.fragmentPaths: false` turns it off.
+
+How it is done::
+The bundler is not told anything; it is shown stamped fragments, in copies of the staged tree under `<build.staging>/fragment-paths/<target>/`.
+The first bundle is built with every fragment stamped, to learn which fragments become components.
+The second, the one that is published, is built with only those stamped.
+So a document differs from one built with `fragmentPaths: false` by exactly one line per component, and in no other byte.
+The cost is a second bundler run per OpenAPI target.
+
+What stops the build::
+A fragment that writes `x-fragment-path` itself: the Publisher sets that key and nothing else may.
+A component's fragment that is also inlined somewhere else in the same document, since the inlined copy would carry the key as well; the error names where.
+
+A component written directly in a bundle root, rather than `$ref`'d from a fragment, has no fragment and carries no path.
+
+[#fragment-path-versions]
+==== What a change to `x-fragment-path` means for a contract's version
+
+Semantic versioning describes the contract, so the version follows from what a change does to the contract's promises:
+
+[cols="2,1,3", options="header"]
+|===
+|Change |Version |Why
+
+|The key appears for the first time
+|MINOR
+|A new feature of the published document.
+
+|A fragment moves to another directory, changing the value
+|PATCH
+|The contract promises the same; only its provenance metadata differs.
+
+|The key is removed, by `fragmentPaths: false` after it was published
+|MAJOR
+|Consumers relying on it lose provenance and collision disambiguation, and may stop generating code at all.
+|===
+
+Upgrading to a Publisher that stamps the key changes every OpenAPI document that has components built from fragments.
+`changed` does not see that, because a target's closure hash covers the fragments, not the Publisher.
+Bump each such target's MINOR version in the upgrade's commit, or set `fragmentPaths: false` until you do: publishing a changed document under a version already published is refused by every Subscriber that locked it (see <<versions,Versions §>>).
+
 [#distribution]
 === `distribution` is transitional
 
@@ -492,6 +557,9 @@ In order of what usually needs changing:
 |The lint rules
 |`defaults.openapi.lint`
 
+|Whether components carry `x-fragment-path`
+|`defaults.openapi.fragmentPaths`, defaulting to `true`
+
 |Where lint reports go
 |`reports.lint`, defaulting to `build/reports/lint/<target>/<kind>.txt`
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@arc-e-tect/api-only-publisher",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Builds, packs and publishes API description documents from a library of reusable fragments.",
   "license": "MIT",
   "author": "Arc-E-Tect",

package/src/config.js

@@ -153,8 +153,30 @@ function load(configPath) {
         isPublished(target) {
             return this.targets[target].publish !== false;
         },
-        bundlePath(target, kind) {
-            return path.join(this.stagingDir(kind), this.targets[target][kind].bundle);
+        // A target's bundle root in a staged tree: the build's own, unless another
+        // staged copy is named.
+        bundlePath(target, kind, stagingRoot = this.stagingRoot(kind)) {
+            return path.join(
+                stagingRoot,
+                requireString(this.sources[kind], `sources.${kind}`),
+                this.targets[target][kind].bundle);
+        },
+        // Whether built documents of this kind carry x-fragment-path on each
+        // component. On unless turned off; OpenAPI only.
+        fragmentPaths(kind) {
+            if (kind !== "openapi") return false;
+            const configured = (this.defaults.openapi || {}).fragmentPaths;
+            if (configured === undefined) return true;
+            if (typeof configured !== "boolean") {
+                throw new ConfigError(
+                    `defaults.openapi.fragmentPaths must be true or false, not ${JSON.stringify(configured)}`);
+            }
+            return configured;
+        },
+        // Where one target's stamped copies of the staged tree are built.
+        fragmentPathStaging(target) {
+            const staging = this.build.staging || "build/staging";
+            return path.resolve(this.root, staging, "fragment-paths", target);
         },
         // The file a target's version is read from: the target's own `versionFile`,
         // relative to this configuration, or else <target>.bundle.properties beside

package/src/fragment-paths.js

@@ -0,0 +1,107 @@
+"use strict";
+
+// x-fragment-path: which fragment a bundled component came from.
+//
+// Bundling flattens the library's directory tree into one components namespace,
+// renaming on collision (`UserV1`, `UserV1-2`). The extension puts the provenance
+// back, so a consumer -- the TranscriberJ above all -- never has to read meaning
+// into a bundler's key.
+//
+// The bundler is not told anything; it is shown stamped fragments. It copies a
+// fragment's top-level keys into whatever it produces from that fragment, so a
+// stamp at the top of a file arrives on the component built from it. But the same
+// is true of every fragment the bundler inlines -- paths, info, tags -- so the
+// build stamps in two passes: every fragment first, to learn which ones became
+// components, then only those. See pipeline.bundleWithFragmentPaths.
+
+const fs = require("fs");
+const path = require("path");
+const YAML = require("yaml");
+
+const KEY = "x-fragment-path";
+
+class FragmentPathError extends Error {}
+
+/**
+ * Stamp YAML fragments under `root`, in place, with their path from `root`.
+ *
+ * The path uses forward slashes on every platform. A file whose root is not a
+ * mapping -- a list of servers, say -- has nowhere to carry the key and is left
+ * alone; nothing that becomes a component looks like that.
+ *
+ * @param {Set<string>|null} only the root-relative paths to stamp; all when null
+ * @returns {string[]} the root-relative paths stamped, sorted
+ */
+function stampFiles(root, { only = null } = {}) {
+    const stamped = [];
+    const walk = (dir) => {
+        for (const entry of fs.readdirSync(dir, { withFileTypes: true }).sort((a, b) => (a.name < b.name ? -1 : 1))) {
+            const full = path.join(dir, entry.name);
+            if (entry.isDirectory()) {
+                walk(full);
+                continue;
+            }
+            if (!/\.ya?ml$/.test(entry.name)) continue;
+            const rel = path.relative(root, full).split(path.sep).join("/");
+            if (only && !only.has(rel)) continue;
+
+            const doc = YAML.parseDocument(fs.readFileSync(full, "utf8"));
+            if (!YAML.isMap(doc.contents)) continue;
+            if (doc.contents.has(KEY)) {
+                throw new FragmentPathError(`${rel} declares ${KEY}; the Publisher sets that key and a fragment may not`);
+            }
+            doc.contents.items.unshift(doc.createPair(KEY, rel));
+            fs.writeFileSync(full, doc.toString());
+            stamped.push(rel);
+        }
+    };
+    walk(root);
+    return stamped;
+}
+
+/**
+ * The stamp on every component of a parsed document, keyed `<type>/<name>`, in
+ * document order. A component with no stamp -- one written inline in the bundle
+ * root, say -- maps to null.
+ *
+ * @returns {Map<string, string|null>}
+ */
+function componentPaths(document) {
+    const result = new Map();
+    for (const [type, entries] of Object.entries(document.components || {})) {
+        for (const [name, value] of Object.entries(entries || {})) {
+            const fragment = value && typeof value === "object" ? value[KEY] : undefined;
+            result.set(`${type}/${name}`, fragment === undefined ? null : fragment);
+        }
+    }
+    return result;
+}
+
+/**
+ * Every stamp in a parsed document that is not directly on a component, with the
+ * JSON pointer of the object carrying it.
+ *
+ * @returns {Array<{at: string, path: string}>}
+ */
+function strayPaths(document) {
+    const stray = [];
+    const visit = (node, pointer) => {
+        if (Array.isArray(node)) {
+            node.forEach((item, i) => visit(item, `${pointer}/${i}`));
+            return;
+        }
+        if (!node || typeof node !== "object") return;
+        const onComponent = /^\/components\/[^/]+\/[^/]+$/.test(pointer);
+        if (Object.hasOwn(node, KEY) && !onComponent) {
+            stray.push({ at: pointer || "/", path: node[KEY] });
+        }
+        for (const [key, value] of Object.entries(node)) {
+            if (key === KEY) continue;
+            visit(value, `${pointer}/${key.replace(/~/g, "~0").replace(/\//g, "~1")}`);
+        }
+    };
+    visit(document, "");
+    return stray;
+}
+
+module.exports = { stampFiles, componentPaths, strayPaths, FragmentPathError, KEY };

package/src/index.js

@@ -16,6 +16,7 @@ module.exports = {
     ...require("./closure"),
     ...require("./unreferenced"),
     ...require("./aggregate"),
+    ...require("./fragment-paths"),
     ...require("./pipeline"),
     ...require("./pack"),
     ...require("./changed"),

package/src/pipeline.js

@@ -5,10 +5,12 @@
 const fs = require("fs");
 const path = require("path");
 const { execFileSync } = require("child_process");
+const YAML = require("yaml");
 
 const { substituteFile } = require("./placeholders");
 const { stampFile } = require("./version");
 const { generateAsyncApi, isAggregate } = require("./aggregate");
+const { stampFiles, componentPaths, strayPaths, FragmentPathError, KEY } = require("./fragment-paths");
 
 class BuildError extends Error {}
 
@@ -82,8 +84,8 @@ function substituteTree(config, kind, log) {
     log(`-- Substituted ${tokens} placeholder(s) across ${files} file(s)`);
 }
 
-function bundle(config, target, kind, outFile, log) {
-    const source = config.bundlePath(target, kind);
+function bundle(config, target, kind, outFile, log, { stagingRoot } = {}) {
+    const source = config.bundlePath(target, kind, stagingRoot);
     if (!fs.existsSync(source)) {
         throw new BuildError(`target '${target}': bundle root not found at ${source}`);
     }
@@ -96,6 +98,58 @@ function bundle(config, target, kind, outFile, log) {
     }
 }
 
+/**
+ * Bundle a target with x-fragment-path on every component built from a fragment.
+ *
+ * Two passes over two fresh copies of the staged tree, so the build's own staged
+ * tree -- which every other target bundles from -- is never stamped. The first
+ * stamps every fragment and learns which of them the bundler made components of.
+ * The second stamps only those, so the document differs from an unstamped one by
+ * exactly one line per such component.
+ *
+ * That holds as long as no component's fragment is also inlined somewhere else;
+ * if one is, the inlined copy would carry the stamp too, and the build fails
+ * rather than publish it.
+ */
+function bundleWithFragmentPaths(config, target, kind, outFile, log) {
+    const scratch = config.fragmentPathStaging(target);
+    fs.rmSync(scratch, { recursive: true, force: true });
+
+    const pass = (name, only) => {
+        const root = path.join(scratch, name);
+        fs.cpSync(config.stagingRoot(kind), root, { recursive: true });
+        try {
+            stampFiles(root, { only });
+        } catch (error) {
+            if (error instanceof FragmentPathError) throw new BuildError(`target '${target}': ${error.message}`);
+            throw error;
+        }
+        const file = path.join(scratch, `${name}.yaml`);
+        bundle(config, target, kind, file, () => {}, { stagingRoot: root });
+        return YAML.parse(fs.readFileSync(file, "utf8"));
+    };
+
+    log(`-- Bundling ${path.basename(config.bundlePath(target, kind))} with ${KEY}`);
+    const discovered = componentPaths(pass("discover", null));
+    const fragments = new Set([...discovered.values()].filter(Boolean));
+    const document = pass("stamp", fragments);
+
+    const components = componentPaths(document);
+    if ([...components.keys()].join("\n") !== [...discovered.keys()].join("\n")) {
+        throw new BuildError(`target '${target}': stamping ${KEY} changed the bundler's component names`);
+    }
+    const stray = strayPaths(document);
+    if (stray.length > 0) {
+        throw new BuildError(
+            `target '${target}': a component's fragment is also inlined elsewhere, so ${KEY} would appear ` +
+            `off its component: ${stray.map((s) => `${s.at} (${s.path})`).join(", ")}`);
+    }
+
+    fs.mkdirSync(path.dirname(outFile), { recursive: true });
+    fs.copyFileSync(path.join(scratch, "stamp.yaml"), outFile);
+    log(`-- Stamped ${KEY} on ${[...components.values()].filter(Boolean).length} component(s)`);
+}
+
 function lint(config, kind, file, log, { report = false, reportFile = null } = {}) {
     const tool = kind === "openapi" ? config.tool("redocly") : config.tool("asyncapi");
     const args = kind === "openapi"
@@ -180,7 +234,11 @@ function build(config, { targets, versionOf = () => null, kinds = ["openapi", "a
                 generateAsyncApi(config, target, { log });
             }
             const outFile = path.join(config.distDir(target), config.outputName(kind));
-            bundle(config, target, kind, outFile, log);
+            if (config.fragmentPaths(kind)) {
+                bundleWithFragmentPaths(config, target, kind, outFile, log);
+            } else {
+                bundle(config, target, kind, outFile, log);
+            }
             const version = versionOf(target);
             if (version) {
                 log(`-- Stamping version '${version}'`);
@@ -200,4 +258,4 @@ function build(config, { targets, versionOf = () => null, kinds = ["openapi", "a
     return results;
 }
 
-module.exports = { build, prepare, stage, substituteTree, bundle, lint, distribute, BuildError };
+module.exports = { build, prepare, stage, substituteTree, bundle, bundleWithFragmentPaths, lint, distribute, BuildError };