@nasebanal/cli 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +78 -3
- package/dist/auth.js +5 -0
- package/dist/http.js +5 -0
- package/dist/index.js +2 -0
- package/dist/output.js +24 -0
- package/dist/reports/ci.js +31 -0
- package/dist/reports/upload.js +221 -25
- package/dist/reports/zip.js +128 -0
- package/dist/specs/command.js +44 -0
- package/dist/specs/push.js +188 -0
- package/package.json +1 -1
- package/specs/assurance.yaml +1814 -158
package/README.md
CHANGED
|
@@ -64,6 +64,10 @@ nb assurance report upload --file specmatic/junit/TEST-junit-jupiter.xml --kind
|
|
|
64
64
|
# One repository, several API versions: label the result (the version is part of the suite's identity, so v1 and v2 of the same name sit side by side)
|
|
65
65
|
nb assurance report upload --file specmatic/v2/junit.xml --kind contract --tool specmatic --api-version v2
|
|
66
66
|
# unit tests hold for every version: leave --api-version out
|
|
67
|
+
# set NB_API_VERSION once (a CI variable per job or branch) instead of repeating the flag; the flag wins, an empty variable means none
|
|
68
|
+
NB_API_VERSION=v2 nb assurance report upload --file specmatic/v2/junit.xml --kind contract --tool specmatic
|
|
69
|
+
# a project may require a version for some kinds of result (Assurance > Projects > the project's settings): a result of such a kind
|
|
70
|
+
# without one is refused with 422 API_VERSION_REQUIRED (the message says to pass --api-version or set NB_API_VERSION)
|
|
67
71
|
|
|
68
72
|
# Extra numbers for the suite (repeatable), e.g. ZAP's findings by severity - what the Assurance security gate reads
|
|
69
73
|
nb assurance report upload --file zap/report/baseline-junit.xml --kind security --tool zap --metric high=0 --metric medium=3 --metric low=2
|
|
@@ -74,12 +78,54 @@ nb assurance report upload --dir locust/logs/20261001_090000
|
|
|
74
78
|
|
|
75
79
|
- **Project:** found by name, created when missing (`--no-create-project` to fail
|
|
76
80
|
instead). In CI the name defaults to the repository (`GITHUB_REPOSITORY` /
|
|
77
|
-
`CI_PROJECT_PATH`); elsewhere pass `--project`.
|
|
81
|
+
`CI_PROJECT_PATH`); elsewhere pass `--project`. That is the zero-configuration
|
|
82
|
+
start; once the project exists, **pin it by id** (`--project-id <id>` or
|
|
83
|
+
`NB_ASSURANCE_PROJECT_ID`): the upload then goes straight to that project, nothing
|
|
84
|
+
is looked up by name or created, so renaming the project, a typo, or a renamed
|
|
85
|
+
repository can no longer send results elsewhere. An unknown id is an error, never a
|
|
86
|
+
new project. The id is printed after every upload (and, in GitHub Actions, written
|
|
87
|
+
to `$GITHUB_OUTPUT` as `project_id`). Keep one id per environment (`--env`), and
|
|
88
|
+
take care when copying a workflow to another repository: its id goes with it.
|
|
89
|
+
- **Link to the report in the repository:** the CLI also sends the CI `--provider`
|
|
90
|
+
(`github` | `gitlab` | `other`, detected) and, when the report file is **committed**
|
|
91
|
+
(tracked, unchanged since `HEAD`, and `HEAD` is the reported commit), its path
|
|
92
|
+
relative to the repository root, so Assurance can link to the file at that commit.
|
|
93
|
+
A report that CI has just generated is not in the commit, so no path is sent;
|
|
94
|
+
override with `--report-path`.
|
|
95
|
+
- **Tie the project to its API in the catalog (`--spec-id`, `--spec-environment`):**
|
|
96
|
+
`--spec-id <id>` links the project to a spec of the API catalog (its id in Assurance /
|
|
97
|
+
Specs), `--spec-environment production` says which environment of it the project covers
|
|
98
|
+
(needs `--spec-id`). A new project is created already linked; an existing one is
|
|
99
|
+
changed **only when it differs**, so a pipeline that runs on every commit writes nothing
|
|
100
|
+
once the link is right. Pointing it at another spec clears the old environment unless one
|
|
101
|
+
is given too. The id is not looked up in the catalog (it is another service): a typo shows
|
|
102
|
+
as "no spec" on the project. The link is set after the result is in, so a failure there
|
|
103
|
+
leaves the result reported and says so. `--spec-file <openapi.yaml>` does the whole chain in one command (see below).
|
|
104
|
+
- **Register the spec file and tie the project to it in one command (`--spec-file`):**
|
|
105
|
+
`nb assurance report upload ... --spec-file openapi.yaml [--spec-name N] [--spec-group G]
|
|
106
|
+
[--spec-org ID] [--spec-environment production]` first runs `nb specs push` on the file,
|
|
107
|
+
then creates / links the project to the spec it returns, then reports the result. The
|
|
108
|
+
catalog's answer never costs the result: if registering the spec fails (a plan without the
|
|
109
|
+
catalog, the limit of specs), the result is still reported, without the link, and the command
|
|
110
|
+
ends with the reason and a non-zero exit. Exclusive with `--spec-id`.
|
|
78
111
|
- **Run:** every job of one pipeline run shares a run key (CI: `GITHUB_RUN_ID` +
|
|
79
112
|
attempt, `CI_PIPELINE_ID`; locally `local-<sha>-<day>`), so a unit job and an e2e
|
|
80
113
|
job land in the same run, and re-running a job replaces its own result.
|
|
81
114
|
Commit, branch and CI URL are detected the same way; override with `--run-key`,
|
|
82
115
|
`--commit`, `--branch`, `--ci-url`.
|
|
116
|
+
- **Keep the tool's own report (`--attach <file|folder>`):** a CI runner is thrown away
|
|
117
|
+
when the job ends, and the summary above is not the tool's report (Playwright's HTML
|
|
118
|
+
report, Specmatic's report, a JUnit file with everything in it). `--attach` sends the
|
|
119
|
+
original with the result: a file as it is, a folder as a zip made by the CLI (regular
|
|
120
|
+
files only, symbolic links are never followed). Up to 20 MiB per result and 200 MiB per
|
|
121
|
+
project; the CLI checks the size before anything is sent. It is **opt-in twice** - nothing is
|
|
122
|
+
attached unless you pass it, and the server refuses it (403 `ATTACHMENTS_DISABLED`) until
|
|
123
|
+
the project has turned on "Attaching original reports" in its settings in Assurance (off by
|
|
124
|
+
default). Make sure the report holds no secrets or personal data.
|
|
125
|
+
The server only stores it (next to the result's cases, deleted with the result and by
|
|
126
|
+
the project's retention) and offers it as a **download** to the project's members in
|
|
127
|
+
Assurance; it is never opened, unzipped or shown in the browser. A failed attachment
|
|
128
|
+
leaves the result reported and the command exits non-zero with the reason.
|
|
83
129
|
- **Pass / fail:** JUnit - any failed or errored test fails the suite. Locust - an
|
|
84
130
|
unhandled task exception, or a failure ratio above `--max-fail-ratio` (default
|
|
85
131
|
`$LOCUST_MAX_FAIL_RATIO`, else 0), fails it - the same rule as nb-quickstarts'
|
|
@@ -91,6 +137,30 @@ nb assurance report upload --dir locust/logs/20261001_090000
|
|
|
91
137
|
is sent in total (names and outcomes always stay). Locust reports have no cases.
|
|
92
138
|
- `--dry-run` prints what would be sent and calls nothing.
|
|
93
139
|
|
|
140
|
+
## The API catalog from a pipeline (`nb specs push`)
|
|
141
|
+
|
|
142
|
+
`nb specs push --file openapi.yaml` registers an OpenAPI document in the catalog (your own
|
|
143
|
+
specs, Evolution): a new spec, or a new version of the one that has the same name (or `--id`).
|
|
144
|
+
It is made to run on **every commit**:
|
|
145
|
+
|
|
146
|
+
- **Nothing is uploaded when the file is already the latest version.** The CLI compares the
|
|
147
|
+
document's SHA-256 with the `content_hash` the catalog keeps for the latest version before it
|
|
148
|
+
sends anything; the same bytes print `unchanged (already vN: nothing uploaded)`. (The catalog
|
|
149
|
+
refuses an identical upload itself, `409 UNCHANGED`, which is also taken as "unchanged", for two
|
|
150
|
+
jobs racing.) The comparison is of bytes: a generator that writes a timestamp into the
|
|
151
|
+
document makes a new version every run, so keep the output stable.
|
|
152
|
+
- **No group has to exist.** A spec is registered in no group unless `--group <name>` is given
|
|
153
|
+
(the group is made when missing); an existing spec is never moved by a push (the catalog
|
|
154
|
+
moves it, by drag and drop or the select on its page).
|
|
155
|
+
- The name is `--name`, else the document's `info.title`, else the file's name. A new version
|
|
156
|
+
carries a note made from the commit, the branch and the CI run (`--note` replaces it), so
|
|
157
|
+
"which commit made v7" has an answer.
|
|
158
|
+
- `--org <id|personal>` (or `$NB_SPECS_ORG`) says whose catalog it is: the catalog is per
|
|
159
|
+
owner and needs `X-Active-Organization-Id`; `personal` (the default) is your own, an
|
|
160
|
+
organization's needs its Enterprise plan. `--dry-run` reads the catalog and says what would
|
|
161
|
+
happen. `NB_SPECS_BASE_URL` points it at another host (local: `http://localhost:8792`).
|
|
162
|
+
- In GitHub Actions the spec's id is written to `$GITHUB_OUTPUT` as `spec_id`.
|
|
163
|
+
|
|
94
164
|
## Installation
|
|
95
165
|
|
|
96
166
|
The CLI installs the `nb` command. Node ≥ 20 is required.
|
|
@@ -204,7 +274,10 @@ Notes:
|
|
|
204
274
|
|
|
205
275
|
### From source (contributors)
|
|
206
276
|
|
|
207
|
-
|
|
277
|
+
No registry token and no sibling checkout needed: the build fetches the pinned,
|
|
278
|
+
public API contracts from `api-specs.nasebanal.com` (see
|
|
279
|
+
[Local Development](#local-development)). Contributing: see
|
|
280
|
+
[CONTRIBUTING.md](CONTRIBUTING.md).
|
|
208
281
|
|
|
209
282
|
```bash
|
|
210
283
|
git clone https://github.com/nasebanal/nb-cli.git
|
|
@@ -359,7 +432,9 @@ npm run typecheck # tsc --noEmit
|
|
|
359
432
|
`sync-specs` resolves each spec from, in order:
|
|
360
433
|
|
|
361
434
|
1. `node_modules/@nasebanal/api-specs-<api>/spec.yaml` (installed package), then
|
|
362
|
-
2. `../nb-api-specs/packages/<api>/spec.yaml` (local monorepo checkout)
|
|
435
|
+
2. `../nb-api-specs/packages/<api>/spec.yaml` (local monorepo checkout), then
|
|
436
|
+
3. the version pinned in `spec-versions.json`, fetched from the public specs host
|
|
437
|
+
(the only source under CI, and what an outside contributor gets).
|
|
363
438
|
|
|
364
439
|
In production the published `@nasebanal/api-specs-*` packages are pinned as
|
|
365
440
|
dependencies — the consumer side of CDD. In local dev we copy straight from the
|
package/dist/auth.js
CHANGED
|
@@ -11,6 +11,7 @@ import { Command } from "commander";
|
|
|
11
11
|
import { createInterface } from "node:readline";
|
|
12
12
|
import { loadConfig, saveConfig, resolveToken, resolveEnv, configFilePath } from "./config.js";
|
|
13
13
|
import { login } from "./oauth.js";
|
|
14
|
+
import { envTokenHint } from "./output.js";
|
|
14
15
|
function readSecretLine(promptText) {
|
|
15
16
|
const rl = createInterface({ input: process.stdin, output: process.stdout, terminal: false });
|
|
16
17
|
process.stdout.write(promptText);
|
|
@@ -60,6 +61,10 @@ export function buildAuthCommand() {
|
|
|
60
61
|
const expiry = tokens.expires_at ? new Date(tokens.expires_at).toLocaleString() : "n/a";
|
|
61
62
|
process.stdout.write(`\n✓ Logged in. Credentials saved to ${configFilePath()}\n`);
|
|
62
63
|
process.stdout.write(` Access token expires: ${expiry}${tokens.refresh_token ? " (auto-refreshes)" : ""}\n`);
|
|
64
|
+
// The login is saved, but a token exported in this shell still wins: say so now, not after a confusing 401 later.
|
|
65
|
+
const shadowed = envTokenHint();
|
|
66
|
+
if (shadowed)
|
|
67
|
+
process.stderr.write(`\n! ${shadowed.replace("takes priority over `nb auth login`.", "takes priority over this login.")}\n`);
|
|
63
68
|
});
|
|
64
69
|
auth
|
|
65
70
|
.command("logout")
|
package/dist/http.js
CHANGED
|
@@ -26,11 +26,16 @@ export async function request(opts) {
|
|
|
26
26
|
const headers = { Accept: "application/json" };
|
|
27
27
|
if (opts.token)
|
|
28
28
|
headers.Authorization = `Bearer ${opts.token}`;
|
|
29
|
+
Object.assign(headers, opts.headers);
|
|
29
30
|
const init = { method: opts.method.toUpperCase(), headers };
|
|
30
31
|
if (opts.body !== undefined) {
|
|
31
32
|
headers["Content-Type"] = "application/json";
|
|
32
33
|
init.body = JSON.stringify(opts.body);
|
|
33
34
|
}
|
|
35
|
+
else if (opts.rawBody !== undefined) {
|
|
36
|
+
headers["Content-Type"] = opts.contentType ?? "application/octet-stream";
|
|
37
|
+
init.body = opts.rawBody;
|
|
38
|
+
}
|
|
34
39
|
let res;
|
|
35
40
|
try {
|
|
36
41
|
res = await fetch(url, init);
|
package/dist/index.js
CHANGED
|
@@ -14,6 +14,7 @@ import { buildAuthCommand } from "./auth.js";
|
|
|
14
14
|
import { specExists } from "./spec/loader.js";
|
|
15
15
|
import { printError } from "./output.js";
|
|
16
16
|
import { buildReportUploadCommand } from "./reports/upload.js";
|
|
17
|
+
import { buildSpecsCommand } from "./specs/command.js";
|
|
17
18
|
import { buildUpgradeCommand, startUpdateNotifier } from "./update.js";
|
|
18
19
|
// Single source of truth for the version: package.json (dist/ and src/ both sit
|
|
19
20
|
// next to it). A hard-coded string here drifts from the release tag.
|
|
@@ -28,6 +29,7 @@ program
|
|
|
28
29
|
.showHelpAfterError();
|
|
29
30
|
program.addCommand(buildAuthCommand());
|
|
30
31
|
program.addCommand(buildUpgradeCommand(version));
|
|
32
|
+
program.addCommand(buildSpecsCommand());
|
|
31
33
|
for (const api of API_CATALOG) {
|
|
32
34
|
if (!specExists(api.spec)) {
|
|
33
35
|
// Spec not synced yet — register a stub that explains how to fix it rather
|
package/dist/output.js
CHANGED
|
@@ -2,7 +2,28 @@
|
|
|
2
2
|
* Output helpers. Default output is pretty JSON to stdout; errors go to stderr.
|
|
3
3
|
* A future iteration can add `--format table` once command metadata is richer.
|
|
4
4
|
*/
|
|
5
|
+
import { appendFileSync } from "node:fs";
|
|
5
6
|
import { ApiError } from "./http.js";
|
|
7
|
+
/** In GitHub Actions, make a value available to later steps as `steps.<id>.outputs.<key>`. A missing output file never fails a command that succeeded. */
|
|
8
|
+
export function exportGithubOutput(key, value) {
|
|
9
|
+
const file = process.env.GITHUB_OUTPUT;
|
|
10
|
+
if (!file)
|
|
11
|
+
return;
|
|
12
|
+
try {
|
|
13
|
+
appendFileSync(file, `${key}=${value}\n`);
|
|
14
|
+
}
|
|
15
|
+
catch {
|
|
16
|
+
// The command succeeded; its output file is a convenience.
|
|
17
|
+
}
|
|
18
|
+
}
|
|
19
|
+
/**
|
|
20
|
+
* Why a 401 may come from the shell and not from the login: `NB_TOKEN` in the environment always wins over `nb auth login`
|
|
21
|
+
* (the CI path), so a token left exported - typically a PAT made for production - is what gets sent, and a local server has
|
|
22
|
+
* never heard of it. Null when `NB_TOKEN` is not set (then the hint to log in is the whole story).
|
|
23
|
+
*/
|
|
24
|
+
export function envTokenHint() {
|
|
25
|
+
return process.env.NB_TOKEN ? "NB_TOKEN is set in this shell and takes priority over `nb auth login`. If it is not valid for this server (a PAT made for production is unknown to a local one), run `unset NB_TOKEN` and try again." : null;
|
|
26
|
+
}
|
|
6
27
|
export function printJson(data) {
|
|
7
28
|
if (typeof data === "string") {
|
|
8
29
|
process.stdout.write(data + "\n");
|
|
@@ -19,6 +40,9 @@ export function printError(err) {
|
|
|
19
40
|
}
|
|
20
41
|
if (err.status === 401) {
|
|
21
42
|
process.stderr.write("\nHint: run `nb auth login` or set NB_TOKEN to authenticate.\n");
|
|
43
|
+
const envHint = envTokenHint();
|
|
44
|
+
if (envHint)
|
|
45
|
+
process.stderr.write(`${envHint}\n`);
|
|
22
46
|
}
|
|
23
47
|
return;
|
|
24
48
|
}
|
package/dist/reports/ci.js
CHANGED
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
* piles up). Every value can still be overridden on the command line.
|
|
6
6
|
*/
|
|
7
7
|
import { execFileSync } from "node:child_process";
|
|
8
|
+
import { basename, dirname, resolve } from "node:path";
|
|
8
9
|
export function readGit() {
|
|
9
10
|
const git = (...args) => {
|
|
10
11
|
try {
|
|
@@ -16,6 +17,36 @@ export function readGit() {
|
|
|
16
17
|
};
|
|
17
18
|
return { sha: git("rev-parse", "HEAD"), branch: git("branch", "--show-current") };
|
|
18
19
|
}
|
|
20
|
+
const runGit = (args, cwd) => {
|
|
21
|
+
try {
|
|
22
|
+
return execFileSync("git", args, { cwd, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim();
|
|
23
|
+
}
|
|
24
|
+
catch {
|
|
25
|
+
return null;
|
|
26
|
+
}
|
|
27
|
+
};
|
|
28
|
+
/**
|
|
29
|
+
* The path of a report file inside the repository, relative to its root (`reports/junit.xml`), for a
|
|
30
|
+
* link to the file at the run's commit. Only a file that is COMMITTED qualifies: tracked, identical to
|
|
31
|
+
* HEAD, and HEAD is the commit being reported. A report that CI just generated is none of those, and a
|
|
32
|
+
* link to a file that is not in the commit would be a dead one, so the answer is then null.
|
|
33
|
+
*/
|
|
34
|
+
export function readReportPath(file, commitSha, git = runGit) {
|
|
35
|
+
const abs = resolve(file);
|
|
36
|
+
const cwd = dirname(abs);
|
|
37
|
+
const name = basename(abs);
|
|
38
|
+
const path = git(["ls-files", "--full-name", "--", name], cwd);
|
|
39
|
+
if (!path)
|
|
40
|
+
return null;
|
|
41
|
+
if (git(["diff", "--quiet", "HEAD", "--", name], cwd) === null)
|
|
42
|
+
return null;
|
|
43
|
+
if (commitSha) {
|
|
44
|
+
const head = git(["rev-parse", "HEAD"], cwd);
|
|
45
|
+
if (!head || !(head === commitSha || head.startsWith(commitSha) || commitSha.startsWith(head)))
|
|
46
|
+
return null;
|
|
47
|
+
}
|
|
48
|
+
return path.replace(/\\/g, "/");
|
|
49
|
+
}
|
|
19
50
|
const nonEmpty = (v) => (v && v.trim() !== "" ? v : null);
|
|
20
51
|
export function detectCi(env, git = readGit, now = new Date()) {
|
|
21
52
|
if (env.GITHUB_ACTIONS === "true") {
|
package/dist/reports/upload.js
CHANGED
|
@@ -12,17 +12,62 @@
|
|
|
12
12
|
* instead of a megabyte of report.
|
|
13
13
|
*/
|
|
14
14
|
import { Command } from "commander";
|
|
15
|
-
import { existsSync, readFileSync, readdirSync } from "node:fs";
|
|
15
|
+
import { existsSync, lstatSync, readFileSync, readdirSync } from "node:fs";
|
|
16
16
|
import { basename, dirname, join } from "node:path";
|
|
17
17
|
import { resolveAccessToken, resolveEnv } from "../config.js";
|
|
18
18
|
import { request, resolveBaseUrl, ApiError } from "../http.js";
|
|
19
19
|
import { loadSpec } from "../spec/loader.js";
|
|
20
|
-
import { printJson } from "../output.js";
|
|
21
|
-
import { detectCi } from "./ci.js";
|
|
20
|
+
import { exportGithubOutput, printJson } from "../output.js";
|
|
21
|
+
import { detectCi, readReportPath } from "./ci.js";
|
|
22
22
|
import { parseJunit } from "./junit.js";
|
|
23
23
|
import { summariseLocust } from "./locust.js";
|
|
24
24
|
import { parseSpecmaticCoverage } from "./specmatic.js";
|
|
25
|
+
import { MAX_ATTACHMENT_BYTES, zipDirectory } from "./zip.js";
|
|
26
|
+
import { checkOrg, ciNote, pushSpec, readSpecFile, specNameFor, hashText, specsBaseUrl } from "../specs/push.js";
|
|
25
27
|
export const SUITE_KINDS = ["unit", "e2e", "load", "contract", "security"];
|
|
28
|
+
export const RUN_PROVIDERS = ["github", "gitlab", "other"];
|
|
29
|
+
/** What the API's ids look like (12 letters and digits); anything else is a typo, not a project. */
|
|
30
|
+
const PROJECT_ID_PATTERN = /^[A-Za-z0-9_-]{1,64}$/;
|
|
31
|
+
/** A spec's id in the API catalog (nb-specs-api): letters, digits, `_` and `-`, up to 64. */
|
|
32
|
+
const SPEC_ID_PATTERN = /^[A-Za-z0-9_-]{1,64}$/;
|
|
33
|
+
const MAX_SPEC_ENVIRONMENT = 40;
|
|
34
|
+
/** `--spec-id`: trimmed; it is never looked up here (the catalog is another service; a typo shows as "no spec" in the project). */
|
|
35
|
+
export function checkSpecId(raw) {
|
|
36
|
+
const v = raw.trim();
|
|
37
|
+
if (!SPEC_ID_PATTERN.test(v))
|
|
38
|
+
throw new Error(`--spec-id '${raw}' is not a catalog spec id (letters, digits, '-' and '_', up to 64)`);
|
|
39
|
+
return v;
|
|
40
|
+
}
|
|
41
|
+
/** `--spec-environment`: which environment of the API the project covers (production, staging, ...). */
|
|
42
|
+
export function checkSpecEnvironment(raw) {
|
|
43
|
+
const v = raw.trim();
|
|
44
|
+
if (v === "" || v.length > MAX_SPEC_ENVIRONMENT)
|
|
45
|
+
throw new Error(`--spec-environment must be 1 to ${MAX_SPEC_ENVIRONMENT} characters`);
|
|
46
|
+
return v;
|
|
47
|
+
}
|
|
48
|
+
/** `--project-id` / `NB_ASSURANCE_PROJECT_ID`: trimmed; empty is none. */
|
|
49
|
+
export function checkProjectId(raw) {
|
|
50
|
+
const v = raw?.trim();
|
|
51
|
+
if (!v)
|
|
52
|
+
return undefined;
|
|
53
|
+
if (!PROJECT_ID_PATTERN.test(v))
|
|
54
|
+
throw new Error(`--project-id '${raw}' is not a project id (letters, digits, '-' and '_'): copy it from the project's page or from a previous upload's output`);
|
|
55
|
+
return v;
|
|
56
|
+
}
|
|
57
|
+
export function checkProvider(raw) {
|
|
58
|
+
const v = raw.trim().toLowerCase();
|
|
59
|
+
if (!RUN_PROVIDERS.includes(v))
|
|
60
|
+
throw new Error(`--provider must be one of ${RUN_PROVIDERS.join(" | ")}`);
|
|
61
|
+
return v;
|
|
62
|
+
}
|
|
63
|
+
/** `--report-path`: a path relative to the repository root, as the API accepts it. */
|
|
64
|
+
export function checkReportPath(raw) {
|
|
65
|
+
const v = raw.trim();
|
|
66
|
+
if (v === "" || v.length > 512 || v.startsWith("/") || v.includes("\\") || v.split("/").some((s) => s === ".." || s === "")) {
|
|
67
|
+
throw new Error("--report-path must be a path relative to the repository root, e.g. reports/junit.xml");
|
|
68
|
+
}
|
|
69
|
+
return v;
|
|
70
|
+
}
|
|
26
71
|
/**
|
|
27
72
|
* The tool is a label (nb-assurance-api `TOOL_PATTERN`): what a result means is carried by its kind and its
|
|
28
73
|
* numbers, so any scanner or runner can be reported without a change here, as long as its report is
|
|
@@ -72,7 +117,7 @@ export function parseMetrics(pairs) {
|
|
|
72
117
|
export function checkApiVersion(label) {
|
|
73
118
|
const v = label.trim();
|
|
74
119
|
if (!API_VERSION_PATTERN.test(v)) {
|
|
75
|
-
throw new Error(`--api-version '${label}' is not a version label: use letters, digits, '.', '_' and '-' (up to 32), e.g. v1 or 2026-10`);
|
|
120
|
+
throw new Error(`--api-version / NB_API_VERSION '${label}' is not a version label: use letters, digits, '.', '_' and '-' (up to 32), e.g. v1 or 2026-10`);
|
|
76
121
|
}
|
|
77
122
|
return v;
|
|
78
123
|
}
|
|
@@ -168,15 +213,16 @@ export function parseReport(opts) {
|
|
|
168
213
|
};
|
|
169
214
|
}
|
|
170
215
|
/** Find the project by name, or create it (unless told not to). Returns its id. */
|
|
171
|
-
export async function ensureProject(conn, name, repo, create) {
|
|
216
|
+
export async function ensureProject(conn, name, repo, create, spec) {
|
|
172
217
|
const find = async () => {
|
|
173
218
|
const res = await request({ ...conn, method: "GET", path: "/api/v1/projects", query: { name } });
|
|
174
219
|
const list = res.data.data.data;
|
|
175
|
-
|
|
220
|
+
const hit = list[0];
|
|
221
|
+
return hit ? { id: hit.id, created: false, spec: { id: hit.spec_id ?? null, environment: hit.spec_environment ?? null } } : null;
|
|
176
222
|
};
|
|
177
223
|
const existing = await find();
|
|
178
224
|
if (existing)
|
|
179
|
-
return
|
|
225
|
+
return existing;
|
|
180
226
|
if (!create) {
|
|
181
227
|
throw new Error(`No project named '${name}' (create it first, or drop --no-create-project).`);
|
|
182
228
|
}
|
|
@@ -185,30 +231,78 @@ export async function ensureProject(conn, name, repo, create) {
|
|
|
185
231
|
...conn,
|
|
186
232
|
method: "POST",
|
|
187
233
|
path: "/api/v1/projects",
|
|
188
|
-
|
|
234
|
+
// A new project is born linked: nothing to compare, nothing to patch afterwards.
|
|
235
|
+
body: { name, ...(repo ? { repo } : {}), ...(spec ? { spec_id: spec.id, ...(spec.environment ? { spec_environment: spec.environment } : {}) } : {}) },
|
|
189
236
|
});
|
|
190
|
-
return { id: res.data.data.id, created: true };
|
|
237
|
+
return { id: res.data.data.id, created: true, spec: spec ? { id: spec.id, environment: spec.environment ?? null } : undefined, specLinked: !!spec };
|
|
191
238
|
}
|
|
192
239
|
catch (err) {
|
|
193
240
|
// Another job of the same pipeline created it between our lookup and our create.
|
|
194
241
|
if (err instanceof ApiError && err.status === 409) {
|
|
195
242
|
const raced = await find();
|
|
196
243
|
if (raced)
|
|
197
|
-
return
|
|
244
|
+
return raced;
|
|
198
245
|
}
|
|
199
246
|
throw err;
|
|
200
247
|
}
|
|
201
248
|
}
|
|
249
|
+
/**
|
|
250
|
+
* Bring the project's link to the catalog in line with what the CI says. Only what differs is written, so a pipeline that
|
|
251
|
+
* runs on every commit leaves the project untouched once it is linked. Pointing at another spec clears the environment
|
|
252
|
+
* unless one is given too (the API's rule: the old environment belonged to the old spec).
|
|
253
|
+
*/
|
|
254
|
+
async function syncSpecLink(conn, project, spec) {
|
|
255
|
+
const want = spec.environment ?? null;
|
|
256
|
+
if (project.specLinked)
|
|
257
|
+
return { id: spec.id, environment: want, changed: true, was: null };
|
|
258
|
+
let current = project.spec;
|
|
259
|
+
if (!current) {
|
|
260
|
+
const res = await request({ ...conn, method: "GET", path: `/api/v1/projects/${encodeURIComponent(project.id)}` });
|
|
261
|
+
const row = res.data.data;
|
|
262
|
+
current = { id: row.spec_id ?? null, environment: row.spec_environment ?? null };
|
|
263
|
+
}
|
|
264
|
+
const sameId = current.id === spec.id;
|
|
265
|
+
const patch = {};
|
|
266
|
+
if (!sameId)
|
|
267
|
+
patch.spec_id = spec.id;
|
|
268
|
+
if (spec.environment !== undefined && (!sameId || current.environment !== want))
|
|
269
|
+
patch.spec_environment = spec.environment;
|
|
270
|
+
if (Object.keys(patch).length === 0)
|
|
271
|
+
return { id: spec.id, environment: current.environment, changed: false, was: current.id };
|
|
272
|
+
await request({ ...conn, method: "PATCH", path: `/api/v1/projects/${encodeURIComponent(project.id)}`, body: patch });
|
|
273
|
+
return { id: spec.id, environment: spec.environment ?? (sameId ? current.environment : null), changed: true, was: current.id };
|
|
274
|
+
}
|
|
202
275
|
export async function uploadReport(p) {
|
|
203
|
-
const project = await ensureProject(p.conn, p.project, p.repo, p.createProject);
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
276
|
+
const project = p.projectId ? { id: p.projectId, created: false } : await ensureProject(p.conn, p.project, p.repo, p.createProject, p.spec);
|
|
277
|
+
let res;
|
|
278
|
+
try {
|
|
279
|
+
res = await request({
|
|
280
|
+
...p.conn,
|
|
281
|
+
method: "POST",
|
|
282
|
+
path: `/api/v1/projects/${encodeURIComponent(project.id)}/results`,
|
|
283
|
+
body: p.payload,
|
|
284
|
+
});
|
|
285
|
+
}
|
|
286
|
+
catch (err) {
|
|
287
|
+
// A pinned id that is not found (or not yours) is a wrong id, not a project to create.
|
|
288
|
+
if (p.projectId && err instanceof ApiError && err.status === 404) {
|
|
289
|
+
throw new Error(`No project with id '${p.projectId}' (check --project-id / NB_ASSURANCE_PROJECT_ID, and --env).`);
|
|
290
|
+
}
|
|
291
|
+
throw err;
|
|
292
|
+
}
|
|
210
293
|
const data = res.data.data;
|
|
211
|
-
|
|
294
|
+
const done = { project_id: project.id, project_created: project.created, ...data };
|
|
295
|
+
if (p.spec) {
|
|
296
|
+
// The result is in; a link that fails now leaves it as it is and says so.
|
|
297
|
+
try {
|
|
298
|
+
done.spec = await syncSpecLink(p.conn, project, p.spec);
|
|
299
|
+
}
|
|
300
|
+
catch (err) {
|
|
301
|
+
const detail = err instanceof ApiError ? (err.data?.error?.message ?? err.message) : err.message;
|
|
302
|
+
throw new Error(`The result was reported, but the project could not be linked to the catalog: ${detail}`);
|
|
303
|
+
}
|
|
304
|
+
}
|
|
305
|
+
return done;
|
|
212
306
|
}
|
|
213
307
|
function fail(message) {
|
|
214
308
|
throw new Error(message);
|
|
@@ -275,6 +369,40 @@ function resolveFormat(opts, file) {
|
|
|
275
369
|
return "junit";
|
|
276
370
|
return fail("Cannot tell the report format from the file name; pass --format junit|locust");
|
|
277
371
|
}
|
|
372
|
+
/** `--attach`: a file as it is, or a folder as a zip. Refuses before anything is sent: missing, empty, or too big for the server. */
|
|
373
|
+
export function prepareAttachment(path, limit = MAX_ATTACHMENT_BYTES) {
|
|
374
|
+
if (!existsSync(path))
|
|
375
|
+
throw new Error(`--attach: not found: ${path}`);
|
|
376
|
+
const stat = lstatSync(path);
|
|
377
|
+
if (stat.isSymbolicLink())
|
|
378
|
+
throw new Error(`--attach: ${path} is a symbolic link; pass the real path`);
|
|
379
|
+
if (stat.isDirectory())
|
|
380
|
+
return { name: `${basename(path.replace(/[\\/]+$/, "")) || "report"}.zip`, bytes: zipDirectory(path, limit) };
|
|
381
|
+
if (!stat.isFile())
|
|
382
|
+
throw new Error(`--attach: ${path} is not a file or a folder`);
|
|
383
|
+
if (stat.size === 0)
|
|
384
|
+
throw new Error(`--attach: ${path} is empty`);
|
|
385
|
+
if (stat.size > limit)
|
|
386
|
+
throw new Error(`--attach: ${path} is ${(stat.size / 1048576).toFixed(1)} MiB, more than the ${limit / 1048576} MiB the server takes`);
|
|
387
|
+
return { name: basename(path), bytes: readFileSync(path) };
|
|
388
|
+
}
|
|
389
|
+
/** Keep the original report with a result that was just reported. A failure here leaves the result as it is. */
|
|
390
|
+
export async function uploadAttachment(conn, ids, a) {
|
|
391
|
+
try {
|
|
392
|
+
await request({
|
|
393
|
+
...conn,
|
|
394
|
+
method: "PUT",
|
|
395
|
+
path: `/api/v1/projects/${encodeURIComponent(ids.project_id)}/runs/${encodeURIComponent(ids.run_id)}/suites/${encodeURIComponent(ids.suite_id)}/attachment`,
|
|
396
|
+
query: { name: a.name },
|
|
397
|
+
rawBody: a.bytes,
|
|
398
|
+
contentType: a.name.endsWith(".zip") ? "application/zip" : "application/octet-stream",
|
|
399
|
+
});
|
|
400
|
+
}
|
|
401
|
+
catch (err) {
|
|
402
|
+
const detail = err instanceof ApiError ? (err.data?.error?.message ?? err.message) : err.message;
|
|
403
|
+
throw new Error(`The result was reported, but the attached report was not kept: ${detail}`);
|
|
404
|
+
}
|
|
405
|
+
}
|
|
278
406
|
export function buildReportUploadCommand() {
|
|
279
407
|
return new Command("upload")
|
|
280
408
|
.description("Report a test result file (JUnit XML or Locust CSV) to an Assurance project")
|
|
@@ -288,9 +416,19 @@ export function buildReportUploadCommand() {
|
|
|
288
416
|
.option("--label <name>", "tells apart results that are otherwise the same, such as two scans of one tool on the same components (e.g. --label baseline, --label full-scan); part of the result's identity, so the same label again replaces the result; optional")
|
|
289
417
|
.option("--tool-version <version>", "the version of the tool that produced the report (e.g. 9.1.0): informational, shown next to the tool, handy when a tool's new version changes its layout; optional")
|
|
290
418
|
.option("--metric <key=value>", "an extra number for the suite, repeatable (e.g. --metric high=0 --metric medium=3; for security: critical, high, medium, low, info)", (v, all = []) => [...all, v])
|
|
291
|
-
.option("--api-version <label>", "the API version the result is for (e.g. v1, v2), part of the result's identity next to its kind and components; leave it out for results that hold for every version, such as unit tests")
|
|
419
|
+
.option("--api-version <label>", "the API version the result is for (e.g. v1, v2), part of the result's identity next to its kind and components; leave it out for results that hold for every version, such as unit tests. Default: the NB_API_VERSION environment variable (set it once per CI job or branch). A project can require a version for some kinds of result, and then refuses one without it")
|
|
292
420
|
.option("--project <name>", "project name (CI: the repository); found, or created if missing")
|
|
421
|
+
.option("--attach <path>", "keep the tool's own report with the result: a file as it is, or a folder as a zip (up to 20 MiB). The server only stores it and offers it for download; nothing opens it. Nothing is attached unless you pass this, so check the report holds no secrets")
|
|
422
|
+
.option("--spec-id <id>", "tie the project to this API of the catalog (the spec's id in Assurance / Specs): set when it differs, so a pipeline that runs on every commit leaves it alone once it is linked. A new project is created already linked. The id is not looked up here")
|
|
423
|
+
.option("--spec-file <path>", "register this OpenAPI file in the API catalog first (a new spec, or a new version when its content changed; nothing is uploaded when it is already the latest) and tie the project to it: the whole chain in one command. Needs the catalog (Evolution) access of the account. Exclusive with --spec-id")
|
|
424
|
+
.option("--spec-name <name>", "with --spec-file: the spec's name in the catalog (default: the document's info.title)")
|
|
425
|
+
.option("--spec-group <name>", "with --spec-file: file a NEW spec in this group (made when missing); default none. An existing spec is never moved")
|
|
426
|
+
.option("--spec-org <id>", "with --spec-file: the organization whose catalog it is (an id), or 'personal' (default; or $NB_SPECS_ORG)")
|
|
427
|
+
.option("--spec-environment <name>", "which environment of that API the project covers (production, staging, ...); needs --spec-id")
|
|
428
|
+
.option("--project-id <id>", "send to this project by id (or $NB_ASSURANCE_PROJECT_ID): no lookup by name and nothing is created, so a rename never moves the results; recommended for a pipeline once the project exists")
|
|
293
429
|
.option("--repo <repo>", "repository recorded on a newly created project (CI: detected)")
|
|
430
|
+
.option("--provider <provider>", `the CI provider: ${RUN_PROVIDERS.join(" | ")} (CI: detected)`)
|
|
431
|
+
.option("--report-path <path>", "the report file's path in the repository, relative to its root (default: detected, and only for a file committed at the reported commit)")
|
|
294
432
|
.option("--run-key <key>", "identity of the pipeline run (CI: detected); all jobs of a run share it")
|
|
295
433
|
.option("--commit <sha>", "commit SHA (CI: detected)")
|
|
296
434
|
.option("--branch <name>", "branch (CI: detected)")
|
|
@@ -318,7 +456,9 @@ export function buildReportUploadCommand() {
|
|
|
318
456
|
const maxFailRatio = Number(ratioRaw);
|
|
319
457
|
if (!Number.isFinite(maxFailRatio) || maxFailRatio < 0 || maxFailRatio > 1)
|
|
320
458
|
fail("--max-fail-ratio must be a number between 0 and 1");
|
|
321
|
-
|
|
459
|
+
// The flag wins; NB_API_VERSION (a CI variable) is the default, and an empty one (a CI that declares it but does not set it) is none.
|
|
460
|
+
const versionRaw = opts.apiVersion ?? (process.env.NB_API_VERSION || undefined);
|
|
461
|
+
const apiVersion = versionRaw === undefined ? undefined : checkApiVersion(versionRaw);
|
|
322
462
|
const parsed = parseReport({
|
|
323
463
|
format,
|
|
324
464
|
kind,
|
|
@@ -351,22 +491,44 @@ export function buildReportUploadCommand() {
|
|
|
351
491
|
}
|
|
352
492
|
const ci = detectCi(process.env);
|
|
353
493
|
const runKey = opts.runKey ?? ci.runKey ?? fail("Cannot determine the run key; pass --run-key");
|
|
354
|
-
const
|
|
494
|
+
const attachment = opts.attach !== undefined ? prepareAttachment(opts.attach) : undefined;
|
|
495
|
+
if (opts.specId !== undefined && opts.specFile !== undefined)
|
|
496
|
+
fail("--spec-id and --spec-file are exclusive: one names a spec of the catalog, the other registers a file");
|
|
497
|
+
if (opts.specEnvironment !== undefined && opts.specId === undefined && opts.specFile === undefined)
|
|
498
|
+
fail("--spec-environment needs --spec-id or --spec-file");
|
|
499
|
+
for (const [flag, value] of [["--spec-name", opts.specName], ["--spec-group", opts.specGroup], ["--spec-org", opts.specOrg]]) {
|
|
500
|
+
if (value !== undefined && opts.specFile === undefined)
|
|
501
|
+
fail(`${flag} needs --spec-file`);
|
|
502
|
+
}
|
|
503
|
+
const specEnvironment = opts.specEnvironment !== undefined ? checkSpecEnvironment(opts.specEnvironment) : undefined;
|
|
504
|
+
let spec = opts.specId !== undefined ? { id: checkSpecId(opts.specId), ...(specEnvironment !== undefined ? { environment: specEnvironment } : {}) } : undefined;
|
|
505
|
+
// The file is read and its name settled before anything is sent, so a bad path fails the command while nothing has happened yet.
|
|
506
|
+
const specText = opts.specFile !== undefined ? readSpecFile(opts.specFile) : undefined;
|
|
507
|
+
const specOrg = opts.specFile !== undefined ? checkOrg(opts.specOrg ?? process.env.NB_SPECS_ORG) : undefined;
|
|
508
|
+
const projectId = checkProjectId(opts.projectId ?? process.env.NB_ASSURANCE_PROJECT_ID);
|
|
509
|
+
const project = opts.project ?? ci.project ?? projectId ?? fail("Cannot determine the project name; pass --project <name> or --project-id <id>");
|
|
510
|
+
if (projectId && opts.project)
|
|
511
|
+
process.stderr.write("--project is ignored: --project-id decides where the result goes\n");
|
|
355
512
|
const commit = opts.commit ?? ci.commitSha;
|
|
356
513
|
const branch = opts.branch ?? ci.branch;
|
|
357
514
|
const ciUrl = opts.ciUrl ?? ci.ciUrl;
|
|
515
|
+
const provider = opts.provider !== undefined ? checkProvider(opts.provider) : ci.provider === "local" ? undefined : ci.provider;
|
|
516
|
+
const reportPath = opts.reportPath !== undefined ? checkReportPath(opts.reportPath) : readReportPath(file, commit);
|
|
517
|
+
if (reportPath)
|
|
518
|
+
parsed.suite.report_path = reportPath;
|
|
358
519
|
const payload = {
|
|
359
520
|
run: {
|
|
360
521
|
key: runKey,
|
|
361
522
|
...(commit ? { commit_sha: commit } : {}),
|
|
362
523
|
...(branch ? { branch } : {}),
|
|
363
524
|
...(ciUrl ? { ci_url: ciUrl } : {}),
|
|
525
|
+
...(provider ? { provider } : {}),
|
|
364
526
|
},
|
|
365
527
|
suite: parsed.suite,
|
|
366
528
|
};
|
|
367
529
|
const repo = opts.repo ?? ci.repo;
|
|
368
530
|
if (opts.dryRun) {
|
|
369
|
-
printJson({ project, repo, payload });
|
|
531
|
+
printJson({ project, ...(projectId ? { project_id: projectId } : {}), repo, payload, ...(spec ? { spec } : {}), ...(specText !== undefined && opts.specFile !== undefined ? { spec_file: { file: opts.specFile, name: specNameFor(specText, opts.specFile, opts.specName), content_hash: hashText(specText), ...(opts.specGroup ? { group: opts.specGroup } : {}) } } : {}), ...(attachment ? { attachment: { name: attachment.name, size_bytes: attachment.bytes.length } } : {}) });
|
|
370
532
|
process.stderr.write(`(dry run) ${basename(dirname(file)) || "."}/${basename(file)}: ${parsed.note}\n`);
|
|
371
533
|
return;
|
|
372
534
|
}
|
|
@@ -374,15 +536,49 @@ export function buildReportUploadCommand() {
|
|
|
374
536
|
const override = process.env.NB_BASE_URL?.replace(/\/$/, "");
|
|
375
537
|
const baseUrl = override ?? resolveBaseUrl(loadSpec("assurance.yaml").servers, env);
|
|
376
538
|
const token = opts.token || (await resolveAccessToken());
|
|
539
|
+
// The catalog comes first: the project is tied to the spec it returns. It must not cost the result: a failure here is
|
|
540
|
+
// kept, the result is reported without the link, and the command says so at the end.
|
|
541
|
+
let pushed;
|
|
542
|
+
let specError;
|
|
543
|
+
if (opts.specFile !== undefined && specOrg !== undefined) {
|
|
544
|
+
try {
|
|
545
|
+
pushed = await pushSpec({
|
|
546
|
+
conn: { baseUrl: specsBaseUrl(env), token, org: specOrg },
|
|
547
|
+
file: opts.specFile,
|
|
548
|
+
name: opts.specName,
|
|
549
|
+
group: opts.specGroup,
|
|
550
|
+
note: ciNote(ci, commit),
|
|
551
|
+
});
|
|
552
|
+
spec = { id: pushed.id, ...(specEnvironment !== undefined ? { environment: specEnvironment } : {}) };
|
|
553
|
+
}
|
|
554
|
+
catch (err) {
|
|
555
|
+
specError = err;
|
|
556
|
+
}
|
|
557
|
+
}
|
|
377
558
|
const result = await uploadReport({
|
|
378
559
|
conn: { baseUrl, token },
|
|
379
560
|
project,
|
|
561
|
+
projectId,
|
|
380
562
|
repo,
|
|
381
563
|
createProject: opts.createProject,
|
|
382
564
|
payload,
|
|
565
|
+
spec,
|
|
383
566
|
});
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
567
|
+
if (attachment)
|
|
568
|
+
await uploadAttachment({ baseUrl, token }, result, attachment);
|
|
569
|
+
printJson({ ...result, ...(attachment ? { attachment: { name: attachment.name, size_bytes: attachment.bytes.length } } : {}), ...(pushed ? { catalog: pushed } : {}) });
|
|
570
|
+
exportGithubOutput("project_id", result.project_id);
|
|
571
|
+
if (pushed)
|
|
572
|
+
exportGithubOutput("spec_id", pushed.id);
|
|
573
|
+
process.stderr.write(`✓ ${kind}/${checkedTool} ${payload.suite.status}: ${parsed.note} -> project '${project}' (id ${result.project_id})` +
|
|
574
|
+
`${result.project_created ? " (created)" : ""}, run '${runKey}'${result.replaced ? " (replaced)" : ""}` +
|
|
575
|
+
`${attachment ? `, attached ${attachment.name} (${(attachment.bytes.length / 1024).toFixed(0)} KiB)` : ""}` +
|
|
576
|
+
`${pushed ? `, spec ${pushed.name} v${pushed.version} (${pushed.action.replace("_", " ")})` : ""}` +
|
|
577
|
+
`${result.spec ? `, catalog ${result.spec.id}${result.spec.environment ? ` (${result.spec.environment})` : ""} ${result.spec.changed ? (result.spec.was ? `(changed from ${result.spec.was})` : "(linked)") : "(already linked)"}` : ""}\n`);
|
|
578
|
+
if (!projectId) {
|
|
579
|
+
process.stderr.write(` Pin it so a rename or a typo cannot send results elsewhere: --project-id ${result.project_id} (or NB_ASSURANCE_PROJECT_ID=${result.project_id})\n`);
|
|
580
|
+
}
|
|
581
|
+
if (specError)
|
|
582
|
+
throw new Error(`The result was reported, but the spec could not be registered in the catalog (the project is not tied to it): ${specError.message}`);
|
|
387
583
|
});
|
|
388
584
|
}
|