@sagentix-inc/evidence-collector 0.4.0 → 0.4.1

package/README.md

@@ -30,6 +30,7 @@ Node 24 or later. The source is TypeScript run by Node's type stripping, so ther
 cd collector
 npm ci            # ajv 8.20.0 and ajv-formats 3.0.1 (v1 SDR validation); typescript 5.9.3, @types/node 24.13.6
 npm test          # node:test, fakes only, no network
+npm run coverage  # the same, with coverage floors
 ```
 
 The OPA sub-source needs `opa` on PATH (`brew install opa`, or the `open-policy-agent/setup-opa` action) and OpenTofu 1.8 or later to produce the state.
@@ -91,6 +92,38 @@ node src/cli.ts run --source cloudflare \
 
 Exit codes: 0 sent (or dry run); 1 error (unreadable key, network failure, or a source that failed: its checks went as `skipped` and everything else was still pushed and reported); 2 usage; 3 nothing to send, because every check was skipped; 4 sos-app refused the push, with its message printed (401 bad signature or revoked key, 409 replay, 400 unknown check or stale `issuedAt`); 75 `agent` only, after it wrote a signed update (below). `run` never exits 75.
 
+### Check the setup first: `collector doctor` (0.4.1)
+
+Run the doctor before the first real run, and again whenever a source starts coming back `skipped`. It reads the same config file and options as `run`, and the same vault, then for each source checks that its credentials are there and makes one read per permission the source needs. It collects nothing, signs nothing and sends nothing to sos-app: no check-in, no push, no run report.
+
+```sh
+node src/cli.ts doctor --config sagentix-collector.config.json            # every source in the file
+node src/cli.ts doctor --config sagentix-collector.config.json --source okta
+```
+
+Each line is `PASS`, `FAIL` or `SKIP`, and a `FAIL` carries its fix:
+
+```
+okta
+  PASS  credentials        OKTA_DOMAIN with OKTA_API_TOKEN (a Read-Only Administrator's token) or OKTA_ACCESS_TOKEN: present
+  PASS  GET /api/v1/users  HTTP 200 (needs okta.users.read)
+  FAIL  GET /api/v1/logs   HTTP 403: The access token provided does not contain the required scopes.
+        fix: grant okta.logs.read
+doctor: 6 passed, 1 failed, 1 skipped; fix the lines marked FAIL, then run the doctor again
+```
+
+A 403 names the scope, role or permission to grant, as the source's own header cites it from the provider's documentation; a 401 names the variables that hold the credential; a 404 asks to check the organization, team, project or URL given. Every read is one the source itself makes, on the same path, with a page size of 1 where the API takes one, so a pass means the run's call will be allowed too. The reads are GET only, except Google Cloud's `:getIamPolicy`, which the GCP source also sends as a POST and which reads. A source that reads a file checks the file is there and readable; Prowler and Cinc Auditor check their executable is on `PATH` without running it; the `macos` and `windows` sources check the platform; `web` checks each URL has its proof token. The collector key is parsed locally and its public key shown, never used. The doctor exits 0 when nothing failed and 1 otherwise, so a workflow can run it as a step (`collector selftest` is the same command).
+
+### Speed, timeouts and what a run prints (0.4.1)
+
+Sources run side by side since 0.4.1, four at once by default; `--concurrency 1` restores the old order of one after another, and up to 16 are allowed. The results are taken in the order the sources were named, so the payload and its signature are byte for byte those of a serial run (`test/e2e.test.ts` holds a golden copy recorded before the change). On the recorded replies of every source, with 20 ms added to each call to stand in for a provider's round trip, a full dry run took 6.6 s one source at a time and 1.7 s four at a time (`node test/bench.ts --latency-ms 20`, 2026-11-07). Prowler runs up to two providers at once, and its OCSF output is now read as a stream, one finding at a time, so a file larger than what one string can hold (about 512 MiB in Node) no longer fails.
+
+Within one run, Entra ID, Intune and an HR source joined with Entra ID share one token request per app and scope (they used to make three), the Google Workspace source and the HR join share theirs, and BambooHR and Rippling look each leaver up in the identity provider once.
+
+Every call a source makes is abandoned after `--http-timeout` seconds (default 60; 0 never). A 429 is retried up to four times; a 502, 503 or 504, a network failure or a timeout up to twice more; each wait follows the answer's `Retry-After` when it has one, else it doubles from 1 second (0.5 second for the clouds, Jira and LMS), capped at 60 seconds (20 for those). The Cloudflare client retries too since 0.4.1, and cuts the token out of any answer that echoes it, as the other clients already did.
+
+`--verbose` prints one line per outbound call (`[okta] GET acme.okta.com/api/v1/logs -> 403 in 85 ms`): the method, the host and the path with ids replaced by `:id`, never a query string, a header or a body, and each retry with its wait. `--debug` adds the stack of a source that failed. Every run, verbose or not, ends its stderr with a table, one row per source: results, pass, fail and skipped counts, outbound calls, retries, answers that were errors, and time. Below it, each check skipped because a call was refused (HTTP 401 or 403) is listed with its source, its check, the status and the doctor command that names the missing permission. Neither the call log nor the table goes into the run report, which already holds every call by its shape and every source's counts and time; both pass through the same scrubbing as the report.
+
 ### Run report, check-in and resident mode (0.3.0)
 
 Since 0.3.0 the collector keeps a channel to sos-app beside the push, defined in `apps/sos-app/docs/COLLECTOR_CHANNEL.md`. Every message is signed with the collector key, exactly as a push is, and every connection still starts on the client's side.
@@ -1052,3 +1085,6 @@ A TLS 1.1 probe that this machine's OpenSSL cannot make, or a network failure, g
 - `src/sources/prowler.ts` and `runner/`: Prowler as an engine (`--source prowler`) and the container that runs it; see `runner/README.md`. More than 5,000 results (or 4 MB) go out as several signed pushes, each with its own nonce.
 - `src/sources/stig.ts`, `kubescape.ts`, `checkov.ts`: Cinc Auditor with MITRE's DISA STIG profiles (`--source stig`, which runs `cinc-auditor` over SSH or WinRM, or reads an InSpec report), and the Kubescape and Checkov report readers; listed through `engines.ts`, documented in `runner/README.md`, tested in `test/host_iac_engines.test.ts` with hand-built reports. Their catalogues (`stig_<profile>.json`, `kubescape.json`, `checkov.json`) come from `apps/sos-app/scripts/build_{stig,kubescape,checkov}_catalog.py`.
 - `test/`: node:test suites with a fake Cloudflare API, a fake `opa` and a fake sos-app; one suite runs the real `opa` when it is installed.
+- `src/doctor.ts`: `collector doctor`, one self-test per source (a test fails when a source has none); `src/pool.ts`: the bounded pool behind `--concurrency`.
+- `test/e2e.test.ts` and `test/e2e_fixtures.ts`: every source in one run against recorded replies, no network: the payload's shape, the signature over the exact bytes, and the results against the golden file `test/fixtures/e2e/golden-results.jsonl` (`UPDATE_GOLDEN=1 node --test test/e2e.test.ts` rewrites it after a deliberate change). The replies are the ones each source's own suite uses, kept in `test/fixtures/rest/`, `cloud/` and `local/`. `test/run_qa.test.ts` covers concurrency, retries, timeouts, the call log, the doctor, and a check that no known secret reaches stdout, stderr, the payload or the run report even when a provider echoes it. `node test/bench.ts --latency-ms 20` times a full dry run.
+- `npm run coverage` runs every suite with Node's coverage and fails below 98% of lines, 82% of branches or 94% of functions (the level of 2026-11-07); `.github/workflows/collector-qa.yml` runs it, the doctor, a dry run as a process and the timing on every push that touches the collector.

package/collector/package.json

@@ -1,5 +1,5 @@
 {
   "name": "@sagentix-inc/evidence-collector",
-  "version": "0.4.0",
+  "version": "0.4.1",
   "type": "module"
 }

package/collector/src/cli.js

@@ -11,6 +11,8 @@
 //   collector config-schema                  prints the JSON schema of the config file
 //   collector google-login --client <client JSON> [--token <file>]   one-time Google Workspace sign-in
 //   collector policies --config collector.config.json   pushes the policy folders for sos-app to match (src/policies.ts)
+//   collector doctor --config collector.config.json [--source x]   credentials and permissions per source; sends nothing (src/doctor.ts)
+//   run options for every source: --concurrency <n> (default 4), --http-timeout <s> (default 60), --verbose, --debug
 //   collector run --endpoint <url> --key-id <id> [--key <pem> | env SAGENTIX_COLLECTOR_KEY]
 //                 --source cloudflare[,...] [source options] [--catalog <dir>] [--dry-run] [--out <file>]
 //                 [--fedramp-target <id> [--class A|B|C|D]]
@@ -30,7 +32,9 @@ import { googleLogin } from "./google_login.js";
 import { V1_COLLECT_PATH, v1Payload } from "./v1/index.js";
 import { CLASSES } from "./v1/rules.js";
 import { EXIT_UPDATE, checkin, detectRunner, modeOf } from "./channel.js";
-import { RunRecorder, buildReport, deliverReport } from "./report.js";
+import { RunRecorder, buildReport, deliverReport, envSecrets, scrubText } from "./report.js";
+import { pool } from "./pool.js";
+import { PROBES, doctorHint } from "./doctor.js";
 import { agent } from "./agent.js";
 import { VAULT_FILE, Vault, keyHolderFor, runEnv } from "./vault.js";
 import { applyRefs } from "./refs.js";
@@ -65,7 +69,26 @@ const COMMON_OPTIONS = {
     class: { type: "string" },
     "cpo-uri": { type: "string" },
     config: { type: "string" },
+    concurrency: { type: "string" },
+    "http-timeout": { type: "string" },
+    verbose: { type: "boolean" },
+    debug: { type: "boolean" },
 };
+/** Sources run side by side, at most this many at once (--concurrency; 1 is the old one-after-another). */
+export const DEFAULT_CONCURRENCY = 4;
+export const MAX_CONCURRENCY = 16;
+/** Seconds before an outbound call made by a source is abandoned (--http-timeout; 0 never). */
+export const DEFAULT_HTTP_TIMEOUT_S = 60;
+/** --concurrency and --http-timeout, or why they are wrong. */
+export function runTuning(values) {
+    const c = values.concurrency === undefined ? DEFAULT_CONCURRENCY : Number(values.concurrency);
+    if (!Number.isInteger(c) || c < 1 || c > MAX_CONCURRENCY)
+        return `--concurrency must be a whole number from 1 to ${MAX_CONCURRENCY}.`;
+    const t = values["http-timeout"] === undefined ? DEFAULT_HTTP_TIMEOUT_S : Number(values["http-timeout"]);
+    if (!Number.isFinite(t) || t < 0 || t > 3600)
+        return "--http-timeout must be a number of seconds from 0 (never) to 3600.";
+    return { concurrency: c, timeoutMs: Math.round(t * 1000) };
+}
 function usage(sources) {
     const opts = sources.flatMap((s) => Object.entries(s.options).map(([n, o]) => `    --${n}${o.type === "string" ? " <value>" : ""}${o.multiple ? " (repeatable)" : ""}  [${s.name}] ${o.help}`));
     return `usage:
@@ -74,10 +97,14 @@ function usage(sources) {
   collector agent --config collector.config.json   (stays connected to sos-app; runs when it asks)
   collector config-schema
   collector google-login --client <Desktop app client JSON> [--token sagentix-google-token.json] [--port <n>] [--login-hint <email>] [--no-browser]
+  collector doctor --config collector.config.json [--source <name>]   checks each source's credentials and permissions; collects and sends nothing
   collector run --endpoint <url> --key-id <id> [--key <pem file>] --source ${sources.map((s) => s.name).join("|")}[,...]
                 [source options] [--catalog <checks dir>] [--dry-run] [--out <file>]
+                [--concurrency <1-${MAX_CONCURRENCY}, default ${DEFAULT_CONCURRENCY}>] [--http-timeout <seconds, default ${DEFAULT_HTTP_TIMEOUT_S}>] [--verbose] [--debug]
                 [--fedramp-target <id> [--class A|B|C|D] [--cpo-uri <url>]]
   The private key may come from ${KEY_ENV} (PEM text) instead of --key.
+  --verbose prints each outbound call (method, host, path with ids as :id; never a query, header or body) and each retry;
+  --debug also prints a failed source's stack. A table of every source (checks, calls, retries, time) ends each run.
   Source options:
 ${opts.join("\n")}
   Every run checks in first (an update sos-app names is taken only with a valid Sagentix release signature, and only
@@ -122,10 +149,77 @@ function serverMessage(text) {
         return text;
     }
 }
-export async function run(argv, deps) {
-    // Every line the run writes to stderr also goes into its report (scrubbed there).
-    const rec = new RunRecorder(deps.now);
-    deps = { ...deps, err: rec.tee(deps.err) };
+/**
+ * The connectors' secrets and settings, set in sos-app and sealed for this collector (src/vault.ts), when the run has a
+ * config and a key id. A vault that cannot be opened is a warning: the sources that need its values fail on their own.
+ */
+async function vaultEnv(o) {
+    const { configPath, keyId, dir, runner, deps } = o;
+    if (!configPath || !keyId)
+        return { env: deps.env, cleanup: () => { }, secrets: [] };
+    try {
+        const creds = credentialsIn(resolve(configPath));
+        const holder = deps.vaultKey?.(dir, keyId) ?? keyHolderFor(dir, keyId, { env: deps.env, platform: process.platform, runner });
+        const r = runEnv(dir, holder, deps.env, (name) => settingsOf(creds, name));
+        // Connectors read from the client's own vault (src/refs.ts) take all their names from there, after the local vault.
+        const refs = existsSync(join(dir, VAULT_FILE)) ? new Vault(dir, holder).refs() : {};
+        const f = Object.keys(refs).length ? await applyRefs(dir, refs, creds, r.env, deps.channel ?? deps.fetch) : null;
+        const used = r.used.length + (f?.used.length ?? 0), warnings = [...r.warnings, ...(f?.warnings ?? [])];
+        if (used)
+            deps.err(`vault: ${used} value(s) from ${VAULT_FILE}${f ? " and your own vault" : ""}`);
+        for (const w of warnings)
+            deps.err(`warning: vault: ${w}`);
+        if (warnings.length)
+            o.note(warnings);
+        return { env: f?.env ?? r.env, cleanup: () => { r.cleanup(); f?.cleanup(); }, secrets: [...r.secrets, ...(f?.secrets ?? [])] };
+    }
+    catch (e) {
+        const w = `the vault could not be opened: ${String(e?.message ?? e)}`;
+        deps.err(w);
+        o.note([w]);
+        return { env: deps.env, cleanup: () => { }, secrets: [] };
+    }
+}
+/** One line of the --verbose call log: the call's shape, scrubbed of the run's secrets. */
+function callLine(c, secrets) {
+    return scrubText(`[${c.source}] ${c.method} ${c.host}${c.path} -> ${c.status || "no answer"} in ${c.ms} ms${c.error ? ` (${c.error})` : ""}`, secrets);
+}
+/** The end-of-run table, one row per source: what it judged, what it called, how long it took. */
+export function summaryTable(reports, stats) {
+    const rows = reports.map((r) => {
+        const st = stats.get(r.name) ?? { calls: 0, retries: 0, httpErrors: 0 };
+        return [r.name, String(r.pass + r.fail + r.skipped), String(r.pass), String(r.fail), String(r.skipped), String(st.calls), String(st.retries), String(st.httpErrors),
+            `${(r.ms / 1000).toFixed(1)} s`, r.error ? "ERROR" : r.pass + r.fail ? "read" : "nothing read"];
+    });
+    const head = ["source", "results", "pass", "fail", "skipped", "calls", "retries", "http errors", "time", "source status"];
+    const total = rows.length > 1 ? [["total", ...[1, 2, 3, 4, 5, 6, 7].map((i) => String(rows.reduce((a, r) => a + Number(r[i]), 0))),
+            `${(reports.reduce((a, r) => Math.max(a, r.ms), 0) / 1000).toFixed(1)} s max`, ""]] : [];
+    const all = [head, ...rows, ...total];
+    const w = head.map((_, i) => Math.max(...all.map((r) => r[i].length)));
+    const line = (r) => r.map((c, i) => (i === 0 || i === r.length - 1 ? c.padEnd(w[i]) : c.padStart(w[i]))).join("  ").trimEnd();
+    return [line(head), w.map((n) => "-".repeat(n)).join("  "), ...rows.map(line), ...(total.length ? [w.map((n) => "-".repeat(n)).join("  "), line(total[0])] : [])];
+}
+/** A short fix for each refused call in the results: source, check, HTTP status and what to grant (src/doctor.ts). */
+export function problemLines(results, max = 20) {
+    const seen = new Set(), out = [];
+    for (const r of results) {
+        const st = r.apiError?.status;
+        if (r.result !== "skipped" || st === undefined || (st !== 401 && st !== 403))
+            continue;
+        const source = r.check.split(".")[0];
+        const k = `${r.check}|${st}`;
+        if (seen.has(k))
+            continue;
+        seen.add(k);
+        if (out.length < max)
+            out.push(`  ${r.check} [${r.resource}]: HTTP ${st}${st === 401 ? " (credential refused)" : " (permission missing)"}; fix: ${doctorHint(source)}`);
+    }
+    if (seen.size > max)
+        out.push(`  … and ${seen.size - max} more`);
+    return out;
+}
+/** The run's options: the command line over the config file (--source adds to the file's), or the usage exit code. */
+function parseRunArgs(argv, deps) {
     const sourceOptions = {};
     for (const s of deps.sources) {
         for (const [n, o] of Object.entries(s.options)) {
@@ -154,8 +248,147 @@ export async function run(argv, deps) {
         const src = [...(c.values.source ?? []), ...(given.source ?? [])];
         values = { ...c.values, ...given, source: src };
     }
+    return { values, configPath };
+}
+/**
+ * `collector doctor`: per configured source, its credentials and one cheap read per permission it needs (src/doctor.ts),
+ * printed as pass or fail with the fix. Nothing is collected, signed or sent: no check-in, no push, no run report.
+ * With --config, --source narrows the file's sources to those named. Exit 0 when every check passes, 1 otherwise.
+ */
+export async function doctorCommand(argv, deps) {
+    const parsed = parseRunArgs(argv, deps);
+    if (typeof parsed === "number")
+        return parsed;
+    const { values, configPath } = parsed;
+    const tuning = runTuning(values);
+    if (typeof tuning === "string") {
+        deps.err(tuning);
+        return EXIT.usage;
+    }
+    // From the config, --source picks among the file's sources rather than adding to them.
+    const cli = argv.flatMap((a, i) => (a === "--source" ? [argv[i + 1]] : a.startsWith("--source=") ? [a.slice(9)] : [])).flatMap((s) => (s ?? "").split(",")).map((s) => s.trim()).filter(Boolean);
+    const all = (values.source ?? []).flatMap((s) => s.split(",")).map((s) => s.trim()).filter(Boolean);
+    const names = [...new Set(configPath && cli.length ? cli : all)];
+    if (!names.length) {
+        deps.err(`doctor: name the sources (--config with a "sources" block, or --source ${deps.sources.map((s) => s.name).join("|")}).`);
+        return EXIT.usage;
+    }
+    const unknown = names.filter((n) => !deps.sources.some((s) => s.name === n));
+    if (unknown.length) {
+        deps.err(`unknown source ${unknown.join(", ")}; known: ${deps.sources.map((x) => x.name).join(", ")}.`);
+        return EXIT.usage;
+    }
+    const keyId = typeof values["key-id"] === "string" ? values["key-id"] : undefined;
+    const dir = configPath ? dirname(resolve(configPath)) : (deps.cwd ?? process.cwd());
+    const runner = detectRunner(deps.env);
+    const lines = [];
+    const out = (s) => lines.push(s);
+    // The collector key, read and parsed locally; it is not used to sign anything here.
+    const keyChecks = [];
+    let pem;
+    if (typeof values.key === "string") {
+        try {
+            pem = readFileSync(values.key, "utf8");
+        }
+        catch (e) {
+            keyChecks.push({ what: "collector key", verdict: "fail", detail: `${values.key}: ${e.code ?? String(e)}`, fix: "point keyFile (or --key) at the PEM `collector keygen` wrote" });
+        }
+    }
+    else
+        pem = deps.env[KEY_ENV];
+    if (pem) {
+        try {
+            keyChecks.push({ what: "collector key", verdict: "pass", detail: `Ed25519 public key ${publicKeyFromPrivate(pem).slice(0, 12)}…${keyId ? `, key id ${keyId}` : " (no keyId: only --dry-run runs)"}` });
+        }
+        catch {
+            keyChecks.push({ what: "collector key", verdict: "fail", detail: "not an Ed25519 private key", fix: "use the PEM `collector keygen` wrote" });
+        }
+    }
+    else if (!keyChecks.length)
+        keyChecks.push({ what: "collector key", verdict: "skip", detail: `no --key and no ${KEY_ENV}: only --dry-run runs` });
+    if (typeof values.endpoint === "string") {
+        try {
+            collectUrl(values.endpoint);
+            keyChecks.push({ what: "endpoint", verdict: "pass", detail: new URL(values.endpoint).origin });
+        }
+        catch (e) {
+            keyChecks.push({ what: "endpoint", verdict: "fail", detail: e.message, fix: "set endpoint to your sos-app origin, https://…" });
+        }
+    }
+    const { env, cleanup, secrets } = await vaultEnv({ configPath, keyId, dir, runner, deps, note: () => { } });
+    const scrub = [...new Set([...envSecrets(env), ...secrets.filter((v) => v.length >= 6)])].sort((a, b) => b.length - a.length);
+    const results = [{ source: "collector", checks: keyChecks }];
+    const now = deps.now();
+    // One wrapper around fetch, with the run's timeout, so a probe that hangs ends like a run's call would.
+    const rec = new RunRecorder(deps.now, undefined, { timeoutMs: tuning.timeoutMs });
+    const release = rec.hold();
+    try {
+        const outcomes = await pool(names, tuning.concurrency, (name) => rec.source(name, async () => {
+            const probe = PROBES[name];
+            if (!probe)
+                return [{ what: "self-test", verdict: "skip", detail: "this source has no self-test; try `collector run --dry-run --source " + name + "`" }];
+            return probe({ opts: values, env, clients: deps.clients, now, warn: () => { } });
+        }));
+        for (const [i, o] of outcomes.entries()) {
+            const v = o.status === "fulfilled" ? o.value : { error: o.reason, value: undefined };
+            results.push({ source: names[i], checks: v.value ?? [{ what: "self-test", verdict: "fail", detail: `the probe failed: ${String(v.error?.message ?? v.error)}`, fix: "report this to Sagentix with the line above" }] });
+        }
+    }
+    finally {
+        release();
+        cleanup();
+    }
+    let failed = 0, passed = 0, skipped = 0;
+    out(`collector doctor: ${names.length} source(s); read-only checks, nothing collected or sent`);
+    for (const r of results) {
+        out(r.source);
+        const w = Math.min(44, Math.max(...r.checks.map((c) => c.what.length), 8));
+        for (const c of r.checks) {
+            if (c.verdict === "pass")
+                passed++;
+            else if (c.verdict === "fail")
+                failed++;
+            else
+                skipped++;
+            out(`  ${c.verdict.toUpperCase().padEnd(4)}  ${c.what.padEnd(w)}  ${c.detail}`);
+            if (c.fix && c.verdict !== "pass")
+                out(`        fix: ${c.fix}`);
+        }
+    }
+    out(`doctor: ${passed} passed, ${failed} failed, ${skipped} skipped${failed ? "; fix the lines marked FAIL, then run the doctor again" : "; ready for `collector run`"}`);
+    for (const l of lines)
+        deps.out(scrubText(l, scrub));
+    return failed ? EXIT.error : EXIT.ok;
+}
+export async function run(argv, deps) {
+    // Every line the run writes to stderr also goes into its report (scrubbed there); the --verbose call log and the
+    // closing table go to stderr only (the report already holds every call, and every source's counts and time).
+    const rec = new RunRecorder(deps.now);
+    const rawErr = deps.err;
+    deps = { ...deps, err: rec.tee(deps.err) };
+    const parsed = parseRunArgs(argv, deps);
+    if (typeof parsed === "number")
+        return parsed;
+    let { values } = parsed;
+    const { configPath } = parsed;
     const dry = values["dry-run"] === true;
     const v1Target = typeof values["fedramp-target"] === "string" ? values["fedramp-target"] : undefined;
+    const tuning = runTuning(values);
+    if (typeof tuning === "string") {
+        deps.err(tuning);
+        return EXIT.usage;
+    }
+    const { concurrency } = tuning;
+    const verbose = values.verbose === true || values.debug === true, debug = values.debug === true;
+    // Read when a line is printed, by then holding the vault's values too.
+    let lineSecrets = envSecrets(deps.env);
+    rec.options = {
+        timeoutMs: tuning.timeoutMs,
+        ...(verbose ? {
+            onCall: (c) => rawErr(callLine(c, lineSecrets)),
+            onRetry: (source, e) => rawErr(scrubText(`[${source}] retry ${e.attempt} after ${e.status ? `HTTP ${e.status}` : "no answer"}, waiting ${e.waitMs} ms`, lineSecrets)),
+        } : {}),
+    };
     const names = (values.source ?? []).flatMap((s) => s.split(",")).map((s) => s.trim()).filter(Boolean);
     if (!names.length) {
         deps.err(`--source is required (${deps.sources.map((s) => s.name).join(", ")}).`);
@@ -233,35 +466,10 @@ export async function run(argv, deps) {
     }
     else
         deps.err("check-in skipped: a dry run without a key cannot sign one.");
-    // The connectors' secrets and settings, set in sos-app and sealed for this collector (src/vault.ts), when the run has a config and a key id.
-    // A vault that cannot be opened is a warning: the sources that need its values fail on their own, the run goes on.
-    let env = deps.env, cleanup = () => { }, vaultSecrets = [];
-    if (configPath && keyId) {
-        try {
-            const creds = credentialsIn(resolve(configPath));
-            const holder = deps.vaultKey?.(dir, keyId) ?? keyHolderFor(dir, keyId, { env: deps.env, platform: process.platform, runner });
-            const r = runEnv(dir, holder, deps.env, (name) => settingsOf(creds, name));
-            // Connectors read from the client's own vault (src/refs.ts) take all their names from there, after the local vault.
-            const refs = existsSync(join(dir, VAULT_FILE)) ? new Vault(dir, holder).refs() : {};
-            const f = Object.keys(refs).length ? await applyRefs(dir, refs, creds, r.env, deps.channel ?? deps.fetch) : null;
-            env = f?.env ?? r.env;
-            cleanup = () => { r.cleanup(); f?.cleanup(); };
-            vaultSecrets = [...r.secrets, ...(f?.secrets ?? [])];
-            const used = r.used.length + (f?.used.length ?? 0), warnings = [...r.warnings, ...(f?.warnings ?? [])];
-            if (used)
-                deps.err(`vault: ${used} value(s) from ${VAULT_FILE}${f ? " and your own vault" : ""}`);
-            for (const w of warnings)
-                deps.err(`warning: vault: ${w}`);
-            if (warnings.length)
-                rec.note("vault", warnings);
-        }
-        catch (e) {
-            const w = `the vault could not be opened: ${String(e?.message ?? e)}`;
-            deps.err(w);
-            rec.note("vault", [w]);
-        }
-    }
+    const { env, cleanup, secrets: vaultSecrets } = await vaultEnv({ configPath, keyId, dir, runner, deps, note: (w) => rec.note("vault", w) });
+    lineSecrets = [...new Set([...envSecrets(env), ...vaultSecrets.filter((v) => v.length >= 6)])].sort((a, b) => b.length - a.length);
     let code;
+    let judged = [];
     try {
         code = await collect();
     }
@@ -272,6 +480,19 @@ export async function run(argv, deps) {
     finally {
         cleanup();
     }
+    // The closing table (stderr only), and the refused calls with what to grant.
+    const ran = rec.sources.filter((r) => chosen.some((s) => s.name === r.name));
+    if (ran.length) {
+        rawErr(`\nsummary (${ran.length} source(s), up to ${concurrency} at once):`);
+        for (const l of summaryTable(ran, rec.stats))
+            rawErr(l);
+        const problems = problemLines(judged);
+        if (problems.length) {
+            rawErr("refused calls (their checks were sent as skipped):");
+            for (const l of problems)
+                rawErr(scrubText(l, lineSecrets));
+        }
+    }
     // The report is scrubbed of every value in the run's environment, the vault's included.
     if (connectorTest)
         return code;
@@ -283,14 +504,35 @@ export async function run(argv, deps) {
         const now = deps.now();
         const raw = [];
         const failed = [];
-        for (const s of chosen) {
-            const { value, error, report } = await rec.source(s.name, () => s.run({ opts: values, env, clients: deps.clients, now, warn: deps.err }));
+        // The sources run side by side, at most --concurrency at once, under one fetch wrapper (src/report.ts) that counts each
+        // call against its source. Their outcomes are then taken in the order the sources were named, so the results, and
+        // the bytes signed, are those of a run one source after another (test/e2e.test.ts holds a golden copy).
+        const release = rec.hold();
+        let outcomes;
+        try {
+            outcomes = (await pool(chosen, concurrency, (s) => rec.source(s.name, () => s.run({ opts: values, env, clients: deps.clients, now, warn: deps.err }))))
+                .map((o) => (o.status === "fulfilled" ? o.value : { error: o.reason, report: { name: "?", ms: 0, pass: 0, fail: 0, skipped: 0, error: null, warnings: [] } }));
+        }
+        finally {
+            release();
+        }
+        // The report lists the sources in the order named, not the order they finished.
+        const order = new Map(chosen.map((s, i) => [s.name, i]));
+        const ranked = rec.sources.filter((r) => order.has(r.name)).sort((a, b) => order.get(a.name) - order.get(b.name));
+        let k = 0;
+        for (let i = 0; i < rec.sources.length; i++)
+            if (order.has(rec.sources[i].name))
+                rec.sources[i] = ranked[k++];
+        for (const [i, s] of chosen.entries()) {
+            const { value, error, report } = outcomes[i];
             let got;
             if (error !== undefined || !Array.isArray(value)) {
                 const message = error !== undefined ? String(error?.message ?? error) : "the source returned no list of results";
                 if (!report.error)
                     report.error = { message, stack: "" };
-                deps.err(`error in source ${s.name}: ${message}; its ${s.emits.length} checks are sent as skipped and the run goes on.`);
+                deps.err(`error in source ${s.name}: ${message}; its ${s.emits.length} checks are sent as skipped and the run goes on. To see why: ${doctorHint(s.name)}.`);
+                if (debug && report.error.stack)
+                    rawErr(scrubText(report.error.stack, lineSecrets));
                 failed.push(s.name);
                 got = s.emits.map((check) => ({ check, resource: `the ${s.name} source`, result: "skipped", detail: `collector error in ${s.name}: ${message}` }));
             }
@@ -303,6 +545,7 @@ export async function run(argv, deps) {
                 report[r.result === "pass" ? "pass" : r.result === "fail" ? "fail" : "skipped"]++;
             raw.push(...got);
         }
+        judged = raw;
         // A failed source makes the run exit 1 once everything else is done, as a failed engine run always has.
         const done = (c) => (c === EXIT.ok || c === EXIT.nothing) && failed.length ? EXIT.error : c;
         const vetted = vet(raw, catalog);
@@ -476,6 +719,8 @@ export async function main(argv, deps = defaultDeps()) {
             return await googleLogin(rest, deps);
         if (cmd === "policies")
             return await policiesCommand(rest, deps);
+        if (cmd === "doctor" || cmd === "selftest")
+            return await doctorCommand(rest, deps);
     }
     catch (e) {
         deps.err(String(e.message ?? e));