@team-supercharge/oasg 11.0.0 → 12.0.1

package/README.md

@@ -70,11 +70,7 @@ For detailed configuration instructions please refer to the [Configuration](#con
 
 ## Configure Linter
 
-Create a `.spectral.yaml` or `.spectral.js` file in the repo's root directory with the following minimal content:
-
-```yaml
-extends: '@team-supercharge/oasg/ruleset'
-```
+Create a `.spectral.js` file in the repo's root directory with the following minimal content:
 
 ```js
 import oasgRuleset from '@team-supercharge/oasg/ruleset';
@@ -84,9 +80,9 @@ export default {
 };
 ```
 
-The file is used for configuring [Spectral](https://stoplight.io/open-source/spectral/) on a per-project basis and will enable usage of custom _OASg_ rulesets.
+The file is used for configuring [Spectral](https://stoplight.io/open-source/spectral/) on a per-project basis and will enable usage of the base _OASg_ ruleset and enables to add project-specific rules as well. For detailed instructions on how to use custom rules refer to the [Linter rules](#linter-rules) section below.
 
-For detailed instructions on how to use custom rules refer to the [Linter rules](#linter-rules) section below.
+> ⚠️ The YAML ruleset format is highly discouraged to be used, as it won't work seamlessly [when being used with Docker](#use-with-docker).
 
 ## Use with Docker
 
@@ -160,17 +156,43 @@ $ npx oasg publish [target]
 
 _OASg_ makes use of the awesome [Spectral](https://stoplight.io/open-source/spectral/) project to provide linting capabilities and comes with a set of more strict default rules we call the _OASg Ruleset_ which is defined in the file [ruleset/ruleset.js](ruleset/ruleset.js) and exported as `@team-supercharge/oasg/ruleset` for the outside world.
 
-The project's `.spectral.yaml` or `.spectral.js` file in the API repository should be always configured to extend the default _OASg Ruleset_ using the methods described in the [Configure linter](#configure-linter) section.
+The project's `.spectral.js` file in the API repository should be always configured to extend the default _OASg Ruleset_ using the methods described in the [Configure linter](#configure-linter) section.
 
 Rules defined in the ruleset can be split into two categories:
 
   * **default rules** - rules that are enabled by default, can be disabled with the `rule-name: off` syntax
   * **opt-in rules** - rules that need to be explicitly enabled with the `rule-name: true` syntax
 
-You can further customize the ruleset in `.spectral.yaml` according to the [documentation](https://meta.stoplight.io/docs/spectral/docs/guides/4-custom-rulesets.md) using the `rules:` property.
+You can further customize the ruleset in `.spectral.js` according to the [documentation](https://meta.stoplight.io/docs/spectral/docs/guides/4-custom-rulesets.md) using the `rules:` property.
 
 An example configuration can be found here:
 
+```js
+import oasgRuleset from '@team-supercharge/oasg/ruleset';
+
+import { truthy } from '@stoplight/spectral-functions';
+
+export default {
+  extends: oasgRuleset,
+  rules: {
+    'oasg-object-names-pascal-case': false,          // disables a default rule - don't do this in general :)
+    'oasg-object-names-api-model-suffix': true,      // enables a non-default rule
+
+    'oasg-path-casing-kebab': false,                 // switches a default rule to another one
+    'oasg-path-casing-pascal': true,
+
+    'my-rule-name': {                                // adds a fully custom rule which applies to only your project
+      description: 'Tags must have a description.',
+      given: '$.tags[*]',
+      severity: 'error',
+      then: {
+        field: 'description',
+        function: truthy
+      }
+    }
+```
+
+
 ```yaml
 extends: '@team-supercharge/oasg/ruleset'
 
@@ -285,8 +307,51 @@ Common source parameters
 |-|-|-|-|
 | id | Unique identifier | Y | - |
 | type | Source type: `simple` / `merged` | N | `simple` |
+| bundle | Bundle specification into a single file | N | `true` |
+| sortSchemas | Sort `components.schemas` alphabetically | N | `true` |
+| decorators | Array of files for decorator functions | N | `[]` |
+| cleanup | Cleans specification from unused paths, tags and schemas | N | `true` |
 | overrides | Override properties of the OpenApi file | N | - |
 
+The source specification is by default bundled into a single file (resolving external dependencies) using the [@redocly/cli package's](https://www.npmjs.com/package/@redocly/cli) `bundle` command.
+
+> ⚠️ It is higly recommended to keep `bundle` on if you plan to use the `openapi` target type, as external refs won't be part of the target artifact.
+
+Decorator files must export a `decorate(document)` function which transforms the parsed OpenAPI document and returns it at the end of the function.
+
+Full example with decorators:
+
+```json
+{
+  "id": "platform-api",
+  "type": "merged",
+  "inputs": [
+    "api/*.openapi.yaml"
+  ],
+  "bundle": false,
+  "sortSchemas": false,
+  "decorators": ["decorators/one", "decorators/two"],
+  "cleanup": false
+}
+```
+
+```js
+// decorators/one.js
+
+function decorate(document) {
+  document.info.title = 'My Custom Decorated Title';
+
+  return document;
+}
+
+exports.decorate = decorate;
+```
+
+After the decorators have run, the specification is by default cleaned up (can be turned off by setting the `"cleanup": false` option):
+  - `paths` with no methods in them are removed
+  - `tags` with no endpoints using them are removed
+  - `components` that are unused are removed (only if `bundle` option is also enabled)
+
 ### Source Types
 
 #### Simple
@@ -688,74 +753,51 @@ describe('Auth', function () {
 |Parameter| Description| Required | Default |
 |-|-|-|-|
 | fileName | Name of the generated file | N | `openapi.yaml` |
-| bundle | Bundle specification into a single file | N | `true` |
-| sortSchemas | Sort `components.schemas` alphabetically | N | `true` |
-| decorators | Array of files for decorator functions | N | `[]` |
-| cleanup | Cleans specification from unused paths, tags and schemas | N | `true` |
-
-The target specification is by default bundled into a single file (resolving external dependencies) using the [@redocly/cli package's](https://www.npmjs.com/package/@redocly/cli) `bundle` command.
-
-> ⚠️ It is higly recommended to keep `bundle` on, as external refs won't be part of the target artifact.
-
-Decorator files must export a `decorate(document)` function which transforms the parsed OpenAPI document and returns it at the end of the function.
-
-Full example with decorators:
 
-```json
-{
-  "id": "api-docs",
-  "type": "openapi",
-  "source": "source-simple",
-  "fileName": "my-openapi-file.yaml",
-  "bundle": false,
-  "sortSchemas": false,
-  "decorators": ["decorators/one", "decorators/two"],
-  "cleanup": false
-}
-```
-
-```js
-// decorators/one.js
+---
 
-function decorate(document) {
-  document.info.title = 'My Custom Decorated Title';
+# Migration Guide
 
-  return document;
-}
+This section covers the breaking changes and their migrations across major version upgrades.
 
-exports.decorate = decorate;
-```
+## From `11.x.x` to `12.0.0`
 
-After the decorators have run, the specification is by default cleaned up (can be turned off by setting the `"cleanup": false` option):
-  - `paths` with no methods in them are removed
-  - `tags` with no endpoints using them are removed
-  - `components` that are unused are removed (only if `bundle` option is also enabled)
+The following options from the `openapi` target type has been moved to the [Source](#source) configuration.
 
----
+* `bundle`
+* `sortSchemas`
+* `decorators`
+* `cleanup`
 
-# Migration Guide
+Please update your `config.json` accordingly: move these properties - if they exist - to the respective `source` configuration.
 
-This section covers the breaking changes and their migrations across major version upgrades.
+> ⚠️ As the `bundle`, `sortSchemas` and `cleanup` flags are **enabled by default**, even without using custom decorators some normalization steps are applied to the source specification before generating the targets. If this causes any problems in your project (although highly unlikely it will) consider **disable** these flags.
 
 ## From `10.x.x` to `11.0.0`
 
 ### Linting
 
-If no [Custom Functions](#custom-functions) were used in `.spectral.yaml`, the YAML format can be kept, just update the reference in the `extends` statement:
+The linter configuration previously stored in `.spectral.yaml` needs to be updated to the JS format:
+
+1. rename the file to `.spectral.js` (don't forget to update `spectral.rulesetFile` in `.vscode/settings.json` if used!)
+2. follow the [instructions](#configure-linter) for extending the base _OASg_ ruleset in JS
+3. migrate project-specific custom [rules](#custom-rules) and [functions](#custom-functions) to JS format
+
+The typical minimal configuration from this in YAML
 
   ```yaml
-  # old one
   extends: '@team-supercharge/oasg/rules/default.yaml'
-
-  # new one
-  extends: '@team-supercharge/oasg/ruleset'
   ```
 
-If there were any custom functions used in `.spectral.yaml`, it needs to be updated to the JS format:
+must become this in JS (if no custom rules or functions are used):
 
-1. rename the file to `.spectral.js` (don't forget to update `spectral.rulesetFile` in `.vscode/settings.json`!)
-2. follow the [instructions](#configure-linter) for extending the base _OASg_ ruleset in JS
-3. migrate project-specific custom [rules](#custom-rules) and [functions](#custom-functions) to JS format
+  ```js
+  import oasgRuleset from '@team-supercharge/oasg/ruleset';
+
+  export default {
+    extends: oasgRuleset
+  }
+  ```
 
 ## From `9.x.x` to `10.0.0`
 

package/bin/merger.js

@@ -26,6 +26,8 @@ async function merge(source) {
 
   dump(document, outFile);
 
+  console.log(`merged input files =>\n  ${inputFiles.join('\n  ')}\ninto\n  ${outFile}\n`);
+
   return outFile;
 }
 
@@ -41,7 +43,7 @@ async function getInputFilesWithGlob(inputs) {
   const inputFiles = [];
   for (const input of inputs) {
     const files = await glob(input);
-    inputFiles.push(...files);
+    inputFiles.push(...files.sort());
   }
   const uniqueInputFiles = inputFiles.filter((value, index, self) => self.indexOf(value) === index);
   return uniqueInputFiles;

package/bin/oasg

@@ -18,6 +18,7 @@ const { exec } = require(`${__dirname}/exec.js`);
 const { merge } = require(`${__dirname}/merger.js`);
 const { applyOverrides } = require(`${__dirname}/overrider.js`);
 const { openApiTarget } = require(`${__dirname}/openapi-target.js`);
+const { processSource } = require(`${__dirname}/process-source.js`);
 const { globSync } = require('glob');
 
 const projectPackageJson = JSON.parse(fs.readFileSync('package.json'));
@@ -215,6 +216,25 @@ async function parseConfig() {
     config.sources = [DEFAULT_SOURCE];
   }
 
+  // set default source values
+  config.sources.forEach(s => {
+    if (s.bundle === undefined) {
+      s.bundle = true;
+    }
+
+    if (s.sortSchemas === undefined) {
+      s.sortSchemas = true;
+    }
+
+    if (s.decorators === undefined) {
+      s.decorators = [];
+    }
+
+    if (s.cleanup === undefined) {
+      s.cleanup = true;
+    }
+  });
+
   // set default source to targets with undefined
   config.targets.forEach(t => {
     if (!t.source) {
@@ -313,11 +333,19 @@ async function parseConfig() {
   return config;
 }
 
-async function buildSources() {
+async function buildSources(sourceIds) {
   const sources = {};
 
   for (var i = 0; i < config.sources.length; i++) {
     const source = config.sources[i];
+
+    // only build sources that are defined
+    if (!sourceIds.includes(source.id)) {
+      continue;
+    }
+
+    console.log(`\n=====\n  id:\t\t${source.id}\n  type:\t\t${source.type}\n  bundle:\t${source.bundle}\n  sortSchemas:\t${source.sortSchemas}\n  decorators: \t${JSON.stringify(source.decorators)}\n  cleanup:\t${source.cleanup}\n---\n`);
+
     let file;
 
     switch (source.type) {
@@ -330,6 +358,7 @@ async function buildSources() {
         break;
     }
 
+    file = processSource(source, file, VERSION);
     file = await applyOverrides(source, file);
 
     sources[source.id] = file;
@@ -368,7 +397,7 @@ async function serve(argv) {
   checkSourceId(sourceId);
 
   // handle input
-  const sources = await buildSources();
+  const sources = await buildSources([sourceId]);
   const input = sources[sourceId];
   const document = YAML.load(input);
 
@@ -415,7 +444,7 @@ async function proxy(argv) {
   checkSourceId(sourceId);
 
   // handle input
-  const sources = await buildSources();
+  const sources = await buildSources([sourceId]);
   const input = sources[sourceId];
   const document = YAML.load(input);
 
@@ -446,7 +475,7 @@ async function generate(argv) {
   VERSION = determineArtifactVersion(argv);
   const preRelease = determinePreRelease(argv);
 
-  const sources = await buildSources();
+  const sources = await buildSources(sourceIdsFromTargetIds(targetIds));
 
   console.log(`generate targets: ${targetIds}`);
 
@@ -455,7 +484,7 @@ async function generate(argv) {
 
     // handle docs target
     if (target.type === 'openapi') {
-      openApiTarget(target, sources[target.source], VERSION);
+      openApiTarget(target, sources[target.source]);
       return;
     }
 
@@ -478,7 +507,7 @@ async function generate(argv) {
 // publish
 async function publish(argv) {
   const targetIds = determineTargetIds(argv);
-  const sources = await buildSources();
+  const sources = await buildSources(sourceIdsFromTargetIds(targetIds));
   const preRelease = determinePreRelease(argv);
 
   console.log(`publish targets: ${targetIds}`);
@@ -630,6 +659,17 @@ function determineTargetIds(argv) {
   return targetIds;
 }
 
+function sourceIdsFromTargetIds(targetIds) {
+  const sourceIds = [];
+  targetIds.forEach(targetId => {
+    const target = config.targets.find(t => t.id === targetId);
+    if (target.source) {
+      sourceIds.push(target.source);
+    }
+  });
+  return sourceIds;
+}
+
 function validTargetIds() {
   return config.targets.map(t => t.id);
 }

package/bin/openapi-target.js

@@ -2,128 +2,24 @@ const fs = require('fs');
 const SwaggerParser = require("@apidevtools/swagger-parser");
 
 const { dump } = require(`${__dirname}/dump.js`);
-const { exec } = require(`${__dirname}/exec.js`);
 
-function bundleSpec(source, target, remove) {
-  exec(`npx redocly bundle ${source} --keep-url-references${remove ?  ' --remove-unused-components' : ''} --output ${target}`, 'pipe');
-}
-
-async function openApiTarget(target, source, version) {
+async function openApiTarget(target, source) {
   const outDir = `out/${target.id}`;
   let outFile = target.fileName || `openapi.yaml`;
   outFile = `${outDir}/${outFile}`;
 
   if (fs.existsSync(outDir)) {
-    fs.rmdirSync(outDir, { recursive: true })
+    fs.rmSync(outDir, { recursive: true })
   }
   fs.mkdirSync(outDir, { recursive: true });
 
-  // bundle spec to a single file by default
-  const bundle = target.bundle === undefined || target.bundle;
-  if (bundle) {
-    bundleSpec(source, outFile, false);
-    console.log(`bundled specification`);
-  }
-
   // parse document
-  let document = await SwaggerParser.parse(bundle ? outFile : source);
-
-  // set version of target document
-  document.info.version = version;
-
-  // sort schemas alphabeticaly
-  const sortSchemas = target.sortSchemas === undefined || target.sortSchemas;
-  if (sortSchemas) {
-    document.components.schemas = Object.keys(document.components.schemas)
-      .sort()
-      .reduce((obj, key) => { obj[key] = document.components.schemas[key]; return obj; }, {});
-  }
-
-  // apply custom decorators
-  const decorators = target.decorators || [];
-  decorators.forEach(d => {
-    const decoratorFile = `${d}.js`;
-    const decoratorPath = `${process.cwd()}/${decoratorFile}`;
-
-    if (!fs.existsSync(decoratorPath)) {
-      console.error(`skipped decorator '${decoratorFile}' - file does not exist`);
-      return;
-    }
-
-    const decorator = require(decoratorPath);
-
-    if (!decorator.decorate) {
-      console.error(`skipped decorator '${decoratorFile}' - must export a 'decorate(document)' function`);
-      return;
-    }
-
-    document = decorator.decorate(document);
+  let document = await SwaggerParser.parse(source);
 
-    console.log(`applied decorator '${decoratorFile}'`);
-  });
-
-  // clean up unused things
-  const cleanup = target.cleanup === undefined || target.cleanup;
-  if (cleanup) {
-    const removedPaths = [];
-    const removedTags = [];
-    const usedTags = [];
-
-    // remove empty paths & gather tags
-    const paths = document.paths;
-    for (const pathKey in paths) {
-      const path = document.paths[pathKey];
-
-      if (Object.keys(path).length === 0) {
-        removedPaths.push(pathKey);
-        delete paths[pathKey];
-      }
-
-      for (const methodKey in path) {
-        const operation = path[methodKey];
-
-        operation.tags.forEach(tag => {
-          if (!usedTags.includes(tag)) {
-            usedTags.push(tag);
-          }
-        });
-      }
-    }
-
-    if (removedPaths.length !== 0) {
-      console.log(`cleaned up unused paths:\n  ${removedPaths.join('\n  ')}`);
-    }
-
-    // remouve unused tags
-    const tags = [...document.tags];
-    tags.forEach(tag => {
-      if (!usedTags.includes(tag.name)) {
-        removedTags.push(tag.name);
-        document.tags.splice(document.tags.indexOf(tag), 1);
-      }
-    });
-
-    if (removedTags.length !== 0) {
-      console.log(`cleaned up unused tags:\n  ${removedTags.join('\n  ')}`);
-    }
-  }
+  // target-specific functions, e.g. write as JSON
 
   // write file
   dump(document, outFile);
-
-  // re-bundle to remove unused components
-  if (cleanup && bundle) {
-    console.log(`re-bundling specification to remove unused components`);
-    let currentSize;
-    let bundledSize;
-    do {
-      currentSize = fs.statSync(outFile).size;
-      bundleSpec(outFile, outFile, true);
-      console.log('.');
-      bundledSize = fs.statSync(outFile).size;
-    }
-    while (bundledSize !== currentSize)
-  }
 }
 
 exports.openApiTarget = openApiTarget;

package/bin/process-source.js

@@ -0,0 +1,138 @@
+const crypto = require('crypto');
+const fs = require('fs');
+const SwaggerParser = require("@apidevtools/swagger-parser");
+
+const { dump } = require(`${__dirname}/dump.js`);
+const { exec } = require(`${__dirname}/exec.js`);
+
+function bundleSpec(source, target, remove) {
+  exec(`npx redocly bundle ${source} --keep-url-references${remove ?  ' --remove-unused-components' : ''} --output ${target}`, 'pipe');
+}
+
+async function processSource(source, sourceFile, version) {
+  const outDir = `./out/.tmp`;
+  let outFile;
+
+  if (!sourceFile.startsWith(outDir)) {
+    if (!fs.existsSync(outDir)) {
+      fs.mkdirSync(outDir, { recursive: true });
+    }
+
+    outFile = `${outDir}/${source.id}-${crypto.randomBytes(4).readUInt32LE(0)}.yaml`;
+  }
+  else {
+    outFile = sourceFile;
+  }
+
+  // bundle spec to a single file by default
+  if (source.bundle) {
+    bundleSpec(sourceFile, outFile, false);
+    console.log(`bundled specification`);
+  }
+
+  // parse document
+  let document = await SwaggerParser.parse(source.bundle ? outFile : sourceFile);
+
+  // set version of target document
+  document.info.version = version;
+
+  // sort schemas alphabeticaly
+  if (source.sortSchemas && document.components && document.components.schemas) {
+    document.components.schemas = Object.keys(document.components.schemas)
+      .sort()
+      .reduce((obj, key) => { obj[key] = document.components.schemas[key]; return obj; }, {});
+  }
+
+  // apply custom decorators
+  const decorators = source.decorators || [];
+  decorators.forEach(d => {
+    const decoratorFile = `${d}.js`;
+    const decoratorPath = `${process.cwd()}/${decoratorFile}`;
+
+    if (!fs.existsSync(decoratorPath)) {
+      console.error(`skipped decorator '${decoratorFile}' - file does not exist`);
+      return;
+    }
+
+    const decorator = require(decoratorPath);
+
+    if (!decorator.decorate) {
+      console.error(`skipped decorator '${decoratorFile}' - must export a 'decorate(document)' function`);
+      return;
+    }
+
+    document = decorator.decorate(document);
+
+    console.log(`applied decorator '${decoratorFile}'`);
+  });
+
+  // clean up unused things
+  if (source.cleanup) {
+    const removedPaths = [];
+    const removedTags = [];
+    const usedTags = [];
+
+    // remove empty paths & gather tags
+    const paths = document.paths || {};
+    for (const pathKey in paths) {
+      const path = document.paths[pathKey];
+
+      if (Object.keys(path).length === 0) {
+        removedPaths.push(pathKey);
+        delete paths[pathKey];
+      }
+
+      for (const methodKey in path) {
+        const maybeOperation = path[methodKey];
+
+        if (!maybeOperation.tags) {
+          continue;
+        }
+
+        maybeOperation.tags.forEach(tag => {
+          if (!usedTags.includes(tag)) {
+            usedTags.push(tag);
+          }
+        });
+      }
+    }
+
+    if (removedPaths.length !== 0) {
+      console.log(`cleaned up unused paths:\n  ${removedPaths.join('\n  ')}`);
+    }
+
+    // remouve unused tags
+    const tags = [...document.tags];
+    tags.forEach(tag => {
+      if (!usedTags.includes(tag.name)) {
+        removedTags.push(tag.name);
+        document.tags.splice(document.tags.indexOf(tag), 1);
+      }
+    });
+
+    if (removedTags.length !== 0) {
+      console.log(`cleaned up unused tags:\n  ${removedTags.join('\n  ')}`);
+    }
+  }
+
+  // write file
+  dump(document, outFile);
+
+  // re-bundle to remove unused components
+  if (source.cleanup && source.bundle) {
+    console.log(`re-bundling specification to remove unused components`);
+    let currentSize;
+    let bundledSize;
+    do {
+      currentSize = fs.statSync(outFile).size;
+      bundleSpec(outFile, outFile, true);
+      console.log('.');
+      bundledSize = fs.statSync(outFile).size;
+    }
+    while (bundledSize !== currentSize)
+  }
+
+  return outFile;
+}
+
+exports.processSource = processSource;

package/config.schema.yml

@@ -37,6 +37,16 @@ sources:
         pattern: '^[a-zA-Z0-9]+([-_][a-zA-Z0-9]+)*$' # only kebab-case or snake_case IDs
       type:
         type: string
+      bundle:
+        type: boolean
+      sortSchemas:
+        type: boolean
+      decorators:
+        type: array
+        items:
+          type: string
+      cleanup:
+        type: boolean
       overrides:
         $ref: '#/definitions/SourceOverrides'
     required:
@@ -320,16 +330,6 @@ targets:
             pattern: "^openapi$"
           fileName:
             type: string
-          bundle:
-            type: boolean
-          sortSchemas:
-            type: boolean
-          decorators:
-            type: array
-            items:
-              type: string
-          cleanup:
-            type: boolean
 
 definitions:
   # default

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@team-supercharge/oasg",
-  "version": "11.0.0",
+  "version": "12.0.1",
   "description": "Node-based tool to lint OpenAPI documents and generate clients, servers and documentation from them",
   "author": "Supercharge",
   "license": "MIT",