specshield 3.5.3 → 3.5.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,6 +1,19 @@
1
1
  # SpecShield CLI changelog
2
2
 
3
- ## Unreleased — security audit
3
+ ## Unreleased — opt-in telemetry
4
+
5
+ ### Added
6
+
7
+ - Opt-in usage telemetry. The CLI asks once at an interactive terminal (default no), never in CI or with `--json`; `SPECSHIELD_TELEMETRY=1` opts in for CI, and `DO_NOT_TRACK=1` or `SPECSHIELD_TELEMETRY=0` always turn it off. Each event has the command, the names of the flags given (never their values), exit code, duration, CLI version, OS and CI provider; never paths, specs, hosts, repository names or results. It's sent by a detached process after the command ends, so exit codes and timing are unchanged. `specshield telemetry status|enable|disable` shows and changes the choice; `SPECSHIELD_TELEMETRY_DEBUG=1` prints events instead of sending them.
8
+
9
+ ## 3.5.4 — 2026-11-12 — shadow APIs
10
+
11
+ ### Added
12
+
13
+ - `specshield audit --logs <files>` compares real traffic with the spec (OWASP API9). It reports undocumented endpoints and methods that answer requests, retired versions still served, operations past their `x-sunset`, and exposed files such as `/.env`. Severity follows the response status; redirects count as no evidence. Catch-all pages are recognised by response size (the home page's, or one shared by other probes). Routes are folded by structure, and trailing `{path}` parameters span segments. A HAR passed with `--har` gets the same checks. Validated on six real public log datasets, 9.9M requests in all.
14
+ - Log import records the response size (combined format) and recognises combined logs whose first lines have an empty request (OpenLiteSpeed).
15
+
16
+ ## 3.5.3 — 2026-11-10 — security audit
4
17
 
5
18
  ### Added
6
19
 
package/README.md CHANGED
@@ -57,7 +57,7 @@ What the CLI does:
57
57
  - [GitHub Actions workflows](#cicd--github-actions)
58
58
 
59
59
  **Reference**
60
- - [Config File](#config-file) · [All Options](#all-options) · [Exit Codes](#exit-codes)
60
+ - [Config File](#config-file) · [All Options](#all-options) · [Exit Codes](#exit-codes) · [Telemetry (opt-in)](#telemetry-opt-in)
61
61
 
62
62
  **Project**
63
63
  - [License](#license) · [Support](#support)
@@ -604,11 +604,32 @@ Audits a whole spec and, optionally, captured traffic, by [OWASP API Security To
604
604
  ```bash
605
605
  specshield audit openapi.yaml
606
606
  specshield audit openapi.yaml --har staging.har --host api.acme.com
607
+ specshield audit openapi.yaml --logs /var/log/nginx/access.log*.gz
607
608
  specshield audit openapi.yaml --fail-on medium --sarif audit.sarif
608
609
  ```
609
610
 
610
611
  **From the spec:** missing or undefined authentication, public writes and admin endpoints, API keys and secrets in URLs, basic auth, deprecated OAuth flows, OAuth over HTTP, credentials in responses, personal data on public endpoints, privileged request fields, page sizes with no maximum, client-supplied URLs, plain-HTTP and non-production servers.
611
612
 
613
+ **From access logs (`--logs`) or a HAR file, compared with the spec (OWASP API9):**
614
+
615
+ | Finding | What it means |
616
+ |---|---|
617
+ | `shadow-endpoint` | A path the spec doesn't have is answering requests |
618
+ | `undocumented-method` | A documented path answers a method the spec doesn't list, e.g. `DELETE` |
619
+ | `zombie-version` | `/v1/...` still answers where the spec documents `/v2/...` |
620
+ | `sunset-passed` | An operation past its `x-sunset` date still answers |
621
+ | `exposed-sensitive-path` | `/.env`, `/.git/config`, `/actuator/env` and similar answered 200 (checked even without a spec) |
622
+
623
+ How sure a finding is comes from the response, not the request:
624
+
625
+ - **Status.** 2xx (or 304) means the endpoint served the request: high for writes and internal paths, medium otherwise. 401/403/400 means it exists but refused (low). 404/405 means it isn't there, and redirects prove nothing (an HTTP→HTTPS redirect answers every path), so neither is reported.
626
+ - **Catch-all pages.** Many sites answer every unknown path with the same page. When the logs record response sizes, a probe answered with the site's home page size, or the same size as other probes got, is downgraded; a size no other probe got stays high. Without sizes, a site that answered 200 to three or more probes is treated as a catch-all.
627
+ - **Routes.** IDs are templated, and object names that don't look like IDs are folded once four or more sit in the same place with the same shape after them (`/things/noxus/x`, `/things/fooab/x` → `/things/{thingId}/x`). Health and discovery endpoints (`/readyz`, `/openapi/v3`…) are reported low. A trailing `{path}` parameter may span several segments, as in Kubernetes' `/proxy/{path}`.
628
+
629
+ Logs in the same formats as `specshield usage import` are supported (combined, ALB, JSON lines, Kubernetes audit events, `.gz`). Per-site logs without a host are grouped by file name, so one site's catch-all doesn't affect another's findings. A file that can't be read is skipped with a warning. `--base-path` and `--host` narrow the logs to this API.
630
+
631
+ Validated on real, published traffic: a Kubernetes CI run (250,724 API requests against the official spec, every reported route independently confirmed absent from it), NASA's 1995 web logs (1.9M lines), Elastic's Apache sample, 90 days of secrepo.com logs, and a 2025 forensic dataset of 7.4M requests from a compromised hosting server, where every probe answered 200 was correctly identified as a catch-all page.
632
+
612
633
  **From a HAR file (`--har`):** secrets in URLs and responses, regulated and personal data, plain HTTP, missing HSTS and `nosniff`, risky CORS, unprotected session cookies, version banners, stack traces. Findings say where a secret was seen and never print its value. Only API responses are checked, so use `--host` to leave out third parties in a browser capture.
613
634
 
614
635
  | Option | Default | |
@@ -1847,6 +1868,34 @@ Per-subcommand flags:
1847
1868
 
1848
1869
  ---
1849
1870
 
1871
+ ## Telemetry (opt-in)
1872
+
1873
+ The CLI sends nothing unless you opt in. The first time you run a command in an interactive terminal it asks once, defaulting to no. It never asks in CI, without a terminal, or with `--json`.
1874
+
1875
+ ```bash
1876
+ specshield telemetry status # whether it's on, and an example of what is sent
1877
+ specshield telemetry enable
1878
+ specshield telemetry disable
1879
+ ```
1880
+
1881
+ One event per command run, sent by a background process after the command ends, so it never slows a command down or changes its exit code:
1882
+
1883
+ ```json
1884
+ {"id":"3f6c…","version":"3.5.5","command":"gate","flags":["--security","--shadow"],
1885
+ "exitCode":0,"durationMs":1840,"ci":"github","os":"linux","arch":"x64","node":20}
1886
+ ```
1887
+
1888
+ **Never sent:** file paths, spec contents, flag values, hosts, repository names or results. `id` is a random id created when you opt in; in CI it's a one-way hash of the CI's numeric project id, so one repository's runs count together without its name.
1889
+
1890
+ | Variable | Effect |
1891
+ |---|---|
1892
+ | `SPECSHIELD_TELEMETRY=1` | Opt in (use this in CI, where there is no prompt) |
1893
+ | `SPECSHIELD_TELEMETRY=0` | Off, whatever was chosen before |
1894
+ | `DO_NOT_TRACK=1` | Off, always |
1895
+ | `SPECSHIELD_TELEMETRY_DEBUG=1` | Print each event to stderr instead of sending it |
1896
+
1897
+ ---
1898
+
1850
1899
  ## License
1851
1900
 
1852
1901
  MIT © SpecShield Software Private Limited
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "specshield",
3
- "version": "3.5.3",
3
+ "version": "3.5.5",
4
4
  "description": "Contract compatibility testing for APIs — catch breaking OpenAPI changes before they reach your consumers, with can-i-deploy deploy gating and GitHub PR checks. (a.k.a. bidirectional contract testing.)",
5
5
  "main": "src/cli.js",
6
6
  "homepage": "https://specshield.io",
@@ -85,6 +85,9 @@
85
85
  ],
86
86
  "collectCoverageFrom": [
87
87
  "src/**/*.js"
88
+ ],
89
+ "setupFiles": [
90
+ "<rootDir>/tests/setup/noTelemetry.js"
88
91
  ]
89
92
  },
90
93
  "engines": {
package/src/cli.js CHANGED
@@ -19,6 +19,9 @@ const waiversCommand = require('./commands/waivers');
19
19
  const proveCommand = require('./commands/prove');
20
20
  const replayCommand = require('./commands/replay');
21
21
  const auditCommand = require('./commands/audit');
22
+ const telemetryCommand = require('./commands/telemetry');
23
+ const telemetry = require('./core/telemetry');
24
+ const { loadLocalConfig, saveLocalConfig } = require('./config/localConfig');
22
25
 
23
26
  const program = new Command();
24
27
 
@@ -48,6 +51,9 @@ program.addCommand(waiversCommand);
48
51
  program.addCommand(proveCommand);
49
52
  program.addCommand(replayCommand);
50
53
  program.addCommand(auditCommand);
54
+ program.addCommand(telemetryCommand);
55
+
56
+ telemetry.install(program, { version, loadConfig: loadLocalConfig, saveConfig: saveLocalConfig });
51
57
 
52
58
  program.parseAsync(process.argv).catch((err) => {
53
59
  const logger = require('./utils/logger');
@@ -8,6 +8,8 @@ const { loadSpec } = require('../core/loadSpec');
8
8
  const { parseSpec } = require('../core/parseSpec');
9
9
  const { auditSpec, consolidate } = require('../core/security/audit');
10
10
  const { auditHar } = require('../core/security/auditHar');
11
+ const { createInventory } = require('../core/security/inventory');
12
+ const { importLogFile } = require('../core/usage/importLogs');
11
13
  const { OWASP, RULES, SEVERITIES } = require('../core/security/rules');
12
14
  const W = require('../core/gate/waivers');
13
15
  const { plural } = require('../core/usage/format');
@@ -21,7 +23,10 @@ const audit = new Command('audit')
21
23
  .description('Security audit of an OpenAPI spec and captured traffic, by OWASP API Security Top 10 category. Sends no requests')
22
24
  .argument('[spec]', 'The OpenAPI spec to audit')
23
25
  .option('--har <file...>', 'HAR capture(s) to check for leaked secrets, personal data and missing protections')
24
- .option('--host <host...>', 'Only check these hosts in the capture')
26
+ .option('--logs <file...>', 'Access logs (.gz ok): find undocumented endpoints, methods and old versions still answering, and exposed files like /.env')
27
+ .option('--format <format>', 'Access log format: auto | combined | alb | json', 'auto')
28
+ .option('--base-path <path>', 'Path prefix the API is served under, e.g. /api (default: from the spec servers)')
29
+ .option('--host <host...>', 'Only check these hosts in the capture and logs')
25
30
  .option('--fail-on <severity>', 'Exit 1 at or above this severity: high, medium, low, or none', 'high')
26
31
  .option('--waivers <file>', `Waiver file (default: ${W.DEFAULT_FILE})`, W.DEFAULT_FILE)
27
32
  .option('--all', 'List every low-severity finding')
@@ -30,17 +35,29 @@ const audit = new Command('audit')
30
35
  .option('--json', 'Output machine-readable JSON')
31
36
  .action(async (specPath, opts) => {
32
37
  try {
33
- if (!specPath && !opts.har) throw new Error('Nothing to audit: pass a spec, --har <file>, or both');
38
+ if (!specPath && !opts.har && !opts.logs) throw new Error('Nothing to audit: pass a spec, --har <file>, --logs <file>, or a combination');
34
39
  if (!FAIL_ON.includes(opts.failOn)) throw new Error(`--fail-on must be one of ${FAIL_ON.join(', ')}`);
35
40
 
36
41
  let specText = null;
37
42
  let doc = null;
38
43
  if (specPath) { specText = await loadSpec(specPath); doc = parseSpec(specText, specPath); }
39
- const har = opts.har ? auditHar(opts.har, { hosts: opts.host ? new Set(opts.host.map((h) => h.toLowerCase())) : null }) : null;
44
+ const hosts = opts.host ? new Set(opts.host.map((h) => h.toLowerCase())) : null;
45
+ const traffic = opts.har || opts.logs ? createInventory(doc, { basePath: opts.basePath, hosts }) : null;
46
+ const har = opts.har ? auditHar(opts.har, { hosts, inventory: traffic }) : null;
47
+ const unread = [];
48
+ for (const file of opts.logs || []) {
49
+ const site = siteOf(file);
50
+ try {
51
+ await importLogFile(file, { format: opts.format, aggregator: { add: (rec) => { rec.site = site; traffic.add(rec); } } });
52
+ } catch (err) {
53
+ unread.push({ file, reason: err.message.split('\n')[0] });
54
+ }
55
+ }
56
+ if (opts.logs && unread.length === opts.logs.length) throw new Error(`No log could be read: ${unread[0].reason}`);
40
57
 
41
58
  const now = Date.now();
42
59
  const { waivers } = await W.loadWaivers(path.resolve(opts.waivers), { maxDays: W.DEFAULT_MAX_DAYS, now, file: opts.waivers });
43
- const all = [...(doc ? auditSpec(doc) : []), ...(har ? har.findings : [])];
60
+ const all = [...(doc ? auditSpec(doc) : []), ...(har ? har.findings : []), ...(traffic ? traffic.findings() : [])];
44
61
  const { blocking, waived } = W.applyWaivers(all, waivers, now);
45
62
  const findings = sort(consolidate(blocking));
46
63
  const failing = findings.filter((f) => atLeast(f.severity, opts.failOn));
@@ -48,6 +65,7 @@ const audit = new Command('audit')
48
65
  spec: specPath || null,
49
66
  title: doc && doc.info ? doc.info.title || null : null,
50
67
  har: har ? { files: opts.har, ...har.stats } : null,
68
+ traffic: traffic ? { logs: opts.logs || [], unread, ...traffic.stats } : null,
51
69
  failOn: opts.failOn,
52
70
  counts: count(findings),
53
71
  owasp: byOwasp(findings),
@@ -68,6 +86,15 @@ const audit = new Command('audit')
68
86
  }
69
87
  });
70
88
 
89
+ /** "logs/shop.example.com.access_log.2025_04_26.gz" → "shop.example.com": which site a per-site log belongs to. */
90
+ function siteOf(file) {
91
+ const name = path.basename(file)
92
+ .replace(/\.gz$/i, '')
93
+ .replace(/([._-]\d{4}[._-]\d{2}[._-]\d{2}([._]\d+)?|(\.\d+)+)$/, '')
94
+ .replace(/[._-]?(access[._-]?log|access|log)$/i, '');
95
+ return name && !/^(web|ssl|other_vhosts)?[._-]?$/i.test(name) ? name : path.basename(path.dirname(path.resolve(file)));
96
+ }
97
+
71
98
  const rank = (s) => SEVERITIES.indexOf(s);
72
99
  const atLeast = (s, min) => min !== 'none' && rank(s) <= rank(min);
73
100
  const sort = (list) => [...list].sort((a, b) => rank(a.severity) - rank(b.severity) || a.rule.localeCompare(b.rule));
@@ -102,6 +129,17 @@ const MAX_LISTED = 5;
102
129
  function render(r, opts) {
103
130
  const lines = ['', chalk.bold(` SpecShield™ Audit — ${r.title || r.spec || 'captured traffic'}`), chalk.gray(' ─────────────────────────────────────────')];
104
131
  if (r.har) lines.push(chalk.gray(` Capture: ${plural(r.har.api, 'API response')} checked of ${plural(r.har.entries, 'request')}, ${plural(r.har.hosts, 'host')}`));
132
+ if (r.traffic && r.traffic.unread.length) {
133
+ lines.push(chalk.yellow(` ${plural(r.traffic.unread.length, 'log file')} skipped: ${r.traffic.unread.slice(0, 2).map((u) => u.reason).join('; ')}${r.traffic.unread.length > 2 ? '; …' : ''}`));
134
+ }
135
+ if (r.traffic && r.spec) {
136
+ const t = r.traffic;
137
+ lines.push(chalk.gray(` Traffic: ${plural(t.requests, 'request')}; ${t.documentedSeen} of ${t.documentedTotal} documented operations called, ${plural(t.endpoints, 'undocumented route')} seen`));
138
+ if (t.requests && t.considered === 0) lines.push(chalk.yellow(' No request matched this API. Check --host and --base-path.'));
139
+ if (t.overflow) lines.push(chalk.yellow(` ${plural(t.overflow, 'request')} not checked: too many distinct undocumented routes.`));
140
+ } else if (r.traffic) {
141
+ lines.push(chalk.gray(' Traffic: checked for exposed files only. Pass the spec to find undocumented endpoints.'));
142
+ }
105
143
  if (r.har && r.har.hosts > 1) lines.push(chalk.gray(' Several hosts in this capture: use --host to check only your own.'));
106
144
  lines.push('');
107
145
  lines.push(` ${chalk.red(`${r.counts.high} high`)} · ${chalk.yellow(`${r.counts.medium} medium`)} · ${chalk.gray(`${r.counts.low} low`)}${r.waived.length ? chalk.gray(` · ${r.waived.length} waived`) : ''}`);
@@ -167,7 +205,7 @@ function sarif(r, specText, harFiles) {
167
205
  level: LEVEL[f.severity],
168
206
  message: { text: `${f.title}${where(f) ? `: ${where(f)}` : ''}` },
169
207
  locations: [{ physicalLocation: {
170
- artifactLocation: { uri: f.source === 'spec' ? r.spec : harFiles[0] },
208
+ artifactLocation: { uri: f.source === 'spec' || !harFiles ? r.spec || 'traffic' : harFiles[0] },
171
209
  region: { startLine: f.source === 'spec' ? lineOf(specText, (f.affectedEndpoints[0] || '').split(' ')[1] || f.field) : 1 },
172
210
  } }],
173
211
  })),
@@ -176,4 +214,4 @@ function sarif(r, specText, harFiles) {
176
214
  }
177
215
 
178
216
  module.exports = audit;
179
- module.exports._internal = { sarif, render };
217
+ module.exports._internal = { sarif, render, siteOf };
@@ -0,0 +1,43 @@
1
+ 'use strict';
2
+
3
+ const { Command } = require('commander');
4
+ const chalk = require('chalk');
5
+ const { loadLocalConfig, saveLocalConfig } = require('../config/localConfig');
6
+ const telemetry = require('../core/telemetry');
7
+
8
+ const EXAMPLE = {
9
+ id: '3f6c…', version: '3.5.5', command: 'gate', flags: ['--security', '--shadow'],
10
+ exitCode: 0, durationMs: 1840, ci: 'github', os: 'linux', arch: 'x64', node: 20,
11
+ };
12
+
13
+ async function setChoice(enabled) {
14
+ await saveLocalConfig(telemetry.withChoice(await loadLocalConfig(), enabled));
15
+ }
16
+
17
+ const cmd = new Command('telemetry');
18
+
19
+ cmd
20
+ .description('Anonymous usage telemetry (opt-in): enable, disable or show what is sent')
21
+ .argument('[action]', 'enable | disable | status', 'status')
22
+ .action(async (action) => {
23
+ if (action === 'enable') {
24
+ await setChoice(true);
25
+ console.log(chalk.green('Telemetry enabled. Thank you — it shows which commands people actually use.'));
26
+ } else if (action === 'disable') {
27
+ await setChoice(false);
28
+ console.log(chalk.green('Telemetry disabled. Nothing will be sent.'));
29
+ } else if (action !== 'status') {
30
+ throw new Error(`unknown action "${action}": use enable, disable or status`);
31
+ }
32
+
33
+ const s = telemetry.state(await loadLocalConfig());
34
+ const label = s.status === 'on' ? chalk.green('on') : s.status === 'off' ? chalk.yellow('off') : chalk.gray('off (not chosen yet)');
35
+ console.log(`\nTelemetry: ${label}${s.source && s.source !== 'config' ? chalk.gray(` (set by ${s.source})`) : ''}`);
36
+ console.log('\nOne event per command run. Example:');
37
+ console.log(chalk.gray(JSON.stringify(EXAMPLE, null, 2)));
38
+ console.log('\nNever sent: file paths, spec contents, flag values, hosts, repository names or results.');
39
+ console.log('In CI, set SPECSHIELD_TELEMETRY=1 to opt in. DO_NOT_TRACK=1 or SPECSHIELD_TELEMETRY=0 always turn it off.');
40
+ console.log(`SPECSHIELD_TELEMETRY_DEBUG=1 prints each event instead of sending it. ${telemetry.DOCS}`);
41
+ });
42
+
43
+ module.exports = cmd;
@@ -28,7 +28,7 @@ const SECRET_VALUES = [
28
28
  ];
29
29
  const STACK = /Traceback \(most recent call last\)|Exception in thread "|\n\s+at [\w.$<>]+\([\w.]+\.(?:java|kt|scala):\d+\)|\n\s+at .+ \(.+\.[jt]s:\d+:\d+\)|\.php on line \d+/;
30
30
 
31
- function auditHar(files, { hosts = null } = {}) {
31
+ function auditHar(files, { hosts = null, inventory = null } = {}) {
32
32
  const found = new Map();
33
33
  const stats = { entries: 0, api: 0, otherHosts: 0, hosts: new Set() };
34
34
  const add = (rule, severity, opts) => {
@@ -38,6 +38,7 @@ function auditHar(files, { hosts = null } = {}) {
38
38
  for (const file of files) {
39
39
  for (const e of entriesOf(file)) {
40
40
  stats.entries++;
41
+ if (inventory) inventory.add(recordOf(e));
41
42
  checkEntry(e, { add, hosts, stats });
42
43
  }
43
44
  }
@@ -51,6 +52,24 @@ function entriesOf(file) {
51
52
  return doc.log.entries.filter((e) => e && e.request && e.request.url && e.response);
52
53
  }
53
54
 
55
+ /** The same shape the access-log parsers produce, plus the response type. */
56
+ function recordOf(e) {
57
+ let url;
58
+ try { url = new URL(e.request.url); } catch { return null; }
59
+ const time = Date.parse(e.startedDateTime);
60
+ return {
61
+ method: String(e.request.method || 'GET').toUpperCase(),
62
+ path: url.pathname,
63
+ query: url.search.replace(/^\?/, ''),
64
+ host: url.hostname.toLowerCase(),
65
+ status: e.response.status > 0 ? e.response.status : null,
66
+ time: Number.isNaN(time) ? null : time,
67
+ userAgent: ((e.request.headers || []).find((h) => h && /^user-agent$/i.test(h.name)) || {}).value || null,
68
+ mime: (e.response.content && e.response.content.mimeType) || null,
69
+ bytes: e.response.content && e.response.content.size >= 0 ? e.response.content.size : null,
70
+ };
71
+ }
72
+
54
73
  function headerMap(list) {
55
74
  const map = new Map();
56
75
  for (const h of list || []) {
@@ -0,0 +1,367 @@
1
+ 'use strict';
2
+
3
+ /**
4
+ * Improper inventory (OWASP API9) from real traffic: what the API serves that
5
+ * the spec doesn't say.
6
+ *
7
+ * shadow-endpoint a path the spec doesn't have, answering requests
8
+ * undocumented-method a documented path, answering a method the spec doesn't list
9
+ * zombie-version /v1/... still answering, where the spec documents /v2/...
10
+ * sunset-passed an operation past its x-sunset date, still answering
11
+ * exposed-sensitive-path /.env, /.git/config, /actuator/env... answering 200
12
+ *
13
+ * The response status decides how sure a finding is: a 2xx (or 304) means the
14
+ * endpoint served the request; 401/403/400 mean it exists; 404/405 and
15
+ * redirects say nothing.
16
+ * A host that answers 200 to scanner probes is most likely a catch-all (a
17
+ * single-page app's index.html), so its findings are downgraded, not trusted.
18
+ */
19
+
20
+ const { createRouteMatcher } = require('../usage/routeMatcher');
21
+ const { templatePath } = require('../har/pathTemplate');
22
+ const { finding } = require('./rules');
23
+
24
+ const MAX_ENTRIES = 2000;
25
+ const MAX_CLIENTS = 100;
26
+ const SKIP_METHODS = new Set(['HEAD', 'OPTIONS']);
27
+ const WRITE = new Set(['POST', 'PUT', 'PATCH', 'DELETE']);
28
+ const STATIC = /\.(m?js|css|map|png|jpe?g|gif|svg|ico|webp|avif|woff2?|ttf|eot|html?|txt|webmanifest)$/i;
29
+ const PRIVILEGED_PATH = /(^|\/)(admin|internal|debug|actuator|management|private|_[a-z]+)(\/|$)/i;
30
+ const VERSION = /^v\d+(\.\d+)?$/i;
31
+ // Every API has these, and few specs list them: worth knowing about, not alarming.
32
+ const CONVENTIONAL = /^\/(health|healthz|livez|readyz|ping|status|version|openapi|swagger|api-docs|v\d+\/api-docs|\.well-known)(\/|$|\.json$|\.yaml$)/i;
33
+ // A position with this many distinct values is a parameter, whatever the values look like.
34
+ const DISTINCT = 4;
35
+
36
+ // Paths no production server should answer with content.
37
+ const PROBES = [
38
+ ['environment file', /^\/(.*\/)?\.env(\.[\w-]+)?$/i],
39
+ ['git repository', /^\/(.*\/)?\.git\/(config|HEAD|index)$/i],
40
+ ['cloud credentials', /^\/(.*\/)?\.aws\/credentials$/i],
41
+ ['Spring Boot actuator', /^\/(.*\/)?actuator\/(env|heapdump|threaddump|configprops|beans|mappings|loggers|jolokia|shutdown)$/i],
42
+ ['heap dump', /^\/(.*\/)?heapdump$/i],
43
+ ['Go debug endpoint', /^\/debug\/(pprof|vars)(\/.*)?$/i],
44
+ ['server status page', /^\/server-(status|info)$/i],
45
+ ['PHP info page', /^\/(.*\/)?phpinfo\.php$/i],
46
+ ['database backup', /^\/(.*\/)?[\w-]*(backup|dump|db)[\w-]*\.(sql|sql\.gz|bak)$/i],
47
+ ];
48
+ // This many probe paths answered with the same page means the host answers everything with it.
49
+ const CATCH_ALL = 3;
50
+ const MAX_SIZES = 50;
51
+
52
+ function createInventory(spec, { basePath, hosts = null, now = Date.now() } = {}) {
53
+ const matcher = spec ? createRouteMatcher(spec, { basePath }) : null;
54
+ const versions = spec ? knownVersions(spec, matcher) : [];
55
+ const entries = new Map();
56
+ const probes = new Map();
57
+ const probeHits = new Map(); // site → probe path → probe answered 2xx
58
+ const home = new Map(); // site → size buckets of its home page
59
+ const documented = new Map();
60
+ const stats = { requests: 0, considered: 0, outside: 0, static: 0, otherHosts: 0, overflow: 0 };
61
+
62
+ function add(rec) {
63
+ if (!rec || !rec.method || typeof rec.path !== 'string') return;
64
+ stats.requests++;
65
+ const method = String(rec.method).toUpperCase();
66
+ const host = rec.host ? String(rec.host).toLowerCase() : null;
67
+ if (hosts && host && !hosts.has(host)) { stats.otherHosts++; return; }
68
+ // Per-site logs record no host; the file they came from stands in, so one
69
+ // site's catch-all page doesn't cast doubt on another's findings.
70
+ const site = host || (rec.site ? String(rec.site) : null);
71
+ if (SKIP_METHODS.has(method)) return;
72
+ let path = rec.path.replace(/\/{2,}/g, '/');
73
+ if (path.length > 1) path = path.replace(/\/$/, '');
74
+ // A capture tells us what came back; an HTML page is the app, not the API.
75
+ const html = rec.mime ? /html/i.test(rec.mime) : false;
76
+
77
+ // Single-page apps and CMS rewrites answer unknown paths with the home page.
78
+ if (path === '/' && method === 'GET' && ok(rec.status) && typeof rec.bytes === 'number') {
79
+ if (!home.has(site)) home.set(site, new Set());
80
+ const sizes = home.get(site);
81
+ const b = bucket(rec.bytes);
82
+ if (sizes.size < MAX_SIZES) [b - 1, b, b + 1].forEach((x) => sizes.add(x));
83
+ }
84
+
85
+ const probe = PROBES.find(([, re]) => re.test(path));
86
+ if (probe) {
87
+ // "/.env/" is a directory request: whatever answered it, it wasn't the file.
88
+ const asDirectory = (rec.trailingSlash || (rec.path.length > 1 && rec.path.endsWith('/'))) && !/^\/debug\//.test(path);
89
+ if (ok(rec.status) && !html && !asDirectory) {
90
+ const key = `${site || ''} ${path}`;
91
+ if (!probes.has(key)) probes.set(key, { label: probe[0], path, host: site, calls: 0, last: null, sizes: [] });
92
+ const p = probes.get(key);
93
+ p.calls++;
94
+ p.last = latest(p.last, rec.time);
95
+ if (typeof rec.bytes === 'number' && p.sizes.length < MAX_SIZES) p.sizes.push(rec.bytes);
96
+ if (!probeHits.has(site)) probeHits.set(site, new Map());
97
+ probeHits.get(site).set(path, p);
98
+ }
99
+ return;
100
+ }
101
+ if (!matcher) return;
102
+ if (path === '/' || STATIC.test(path)) { stats.static++; return; }
103
+ if (html) return;
104
+ stats.considered++;
105
+
106
+ const m = matcher.match(method, path) || greedy(method, path);
107
+ if (m && m.operation) { record(documented, `${method} ${m.template}`, { method, template: m.template }, rec, site); return; }
108
+ if (m) { remember('undocumented-method', method, m.template, rec, site); return; }
109
+ const older = versions.length ? otherVersion(method, path) : null;
110
+ if (older) { remember('zombie-version', method, templatePath(path).templated, rec, site, older); return; }
111
+ // Named like the spec names its paths, so a waiver for it reads the same.
112
+ const base = matcher.basePaths.find((b) => path === b || path.startsWith(`${b}/`));
113
+ if (matcher.basePaths.length && !base) { stats.outside++; return; }
114
+ remember('shadow-endpoint', method, templatePath(base ? path.slice(base.length) || '/' : path).templated, rec, site);
115
+ }
116
+
117
+ /**
118
+ * Kubernetes and others document "/proxy/{path}" where {path} spans several
119
+ * segments. OpenAPI doesn't say a parameter can't hold a slash, so try the
120
+ * path with its tail folded into the last segment.
121
+ */
122
+ function greedy(method, path) {
123
+ const segs = path.split('/');
124
+ for (let n = segs.length - 1; n > 1; n--) {
125
+ const m = matcher.match(method, [...segs.slice(0, n), segs.slice(n).join('%2F')].join('/'));
126
+ if (m && /\/\{[^}/]+\}$/.test(m.template)) return m;
127
+ }
128
+ return null;
129
+ }
130
+
131
+ /** /v1/orders when the spec serves /v2/orders: the same operation under a version the spec dropped. */
132
+ function otherVersion(method, path) {
133
+ const segs = path.split('/');
134
+ for (let i = 0; i < segs.length; i++) {
135
+ if (!VERSION.test(segs[i]) || versions.includes(segs[i].toLowerCase())) continue;
136
+ for (const v of versions) {
137
+ const swapped = [...segs.slice(0, i), v, ...segs.slice(i + 1)].join('/');
138
+ const m = matcher.match(method, swapped);
139
+ if (m && m.operation) return `${method} ${m.template} (${v})`;
140
+ }
141
+ }
142
+ return null;
143
+ }
144
+
145
+ function remember(kind, method, template, rec, host, current = null) {
146
+ const key = `${kind}|${method} ${template}`;
147
+ if (!entries.has(key) && entries.size >= MAX_ENTRIES) { stats.overflow++; return; }
148
+ record(entries, key, { kind, method, template, current, fixed: kind === 'shadow-endpoint' && CONVENTIONAL.test(template) }, rec, host);
149
+ }
150
+
151
+ /**
152
+ * Whether a probe's 2xx was the site's catch-all page rather than the file.
153
+ * With sizes: other probe paths got a response of the same size, so it isn't
154
+ * this file's content. Comparing per probe copes with a site that has several
155
+ * such pages, or whose page changed. Only probes count: an ordinary API
156
+ * response of a similar size says nothing. Without sizes: the site answered
157
+ * 2xx to several probes.
158
+ */
159
+ function catchAllAnswer(p) {
160
+ const size = median(p.sizes);
161
+ const siblings = [...probeHits.get(p.host).values()].filter((q) => q !== p);
162
+ if (size === null) {
163
+ const n = siblings.length + 1;
164
+ return n >= CATCH_ALL ? `this host answers 200 to ${n} probe paths, so it is probably a catch-all page` : null;
165
+ }
166
+ const b = bucket(size);
167
+ if ((home.get(p.host) || new Set()).has(b)) return `${size} bytes, the size of this host's home page; probably a rewrite to it`;
168
+ const same = siblings.filter((q) => { const m = median(q.sizes); return m !== null && Math.abs(bucket(m) - b) <= 1; });
169
+ if (same.length + 1 < CATCH_ALL) return null;
170
+ return `${size} bytes, the same as ${same.length} other probe paths got (e.g. ${same[0].path}); probably a catch-all page`;
171
+ }
172
+
173
+ function findings() {
174
+ const verdicts = new Map([...probes.values()].map((p) => [p, catchAllAnswer(p)]));
175
+ const catchAll = new Set([...verdicts].filter(([, v]) => v).map(([p]) => p.host));
176
+ const out = [];
177
+ for (const e of collapse([...entries.values()])) {
178
+ const f = judge(e, catchAll);
179
+ if (f) out.push(f);
180
+ }
181
+ for (const op of sunsetPassed(spec, documented, now)) out.push(op);
182
+ for (const [p, why] of verdicts) {
183
+ const size = median(p.sizes);
184
+ const detail = why ? `${p.label}; ${why}` : `${p.label}, answered 2xx ${times(p.calls)}${size !== null ? ` with ${size} bytes no other path got` : ''}`;
185
+ out.push(finding('exposed-sensitive-path', why ? 'low' : 'high', { endpoint: `GET ${p.path}`, field: p.host, detail, source: 'traffic' }));
186
+ }
187
+ return out;
188
+ }
189
+
190
+ return {
191
+ add,
192
+ findings,
193
+ get stats() {
194
+ const real = [...entries.values()].filter((e) => count(e, (st) => st !== 404 && st !== 405) > 0).length;
195
+ return { ...stats, documentedSeen: documented.size, documentedTotal: matcher ? matcher.operations.length : 0, endpoints: real };
196
+ },
197
+ };
198
+ }
199
+
200
+ /**
201
+ * Folds routes that differ only where a value varies: /things/noxus/x and
202
+ * /things/fooab/x become /things/{thingId}/x once DISTINCT siblings share both
203
+ * a parent and the shape of what follows them. Object names that don't look
204
+ * like IDs would otherwise each be a finding, and fixed words (/readyz,
205
+ * /metrics, /openapi) differ in what follows, so they stay apart. The first
206
+ * segment is never folded.
207
+ */
208
+ function collapse(list) {
209
+ let routes = list.map((e) => ({ ...e, segs: e.template.split('/'), variants: e.variants || 1, examples: e.examples || [] }));
210
+ const depth = Math.max(0, ...routes.map((r) => r.segs.length));
211
+ const isParam = (v) => v.startsWith('{');
212
+ for (let i = 2; i < depth; i++) {
213
+ const familyOf = (r) => [r.kind, r.method, r.segs.slice(0, i).join('/')].join('|');
214
+ const shapes = new Map(); // family → value → shapes of what follows
215
+ for (const r of routes) {
216
+ if (r.segs.length <= i || isParam(r.segs[i])) continue;
217
+ const fam = familyOf(r);
218
+ if (!shapes.has(fam)) shapes.set(fam, new Map());
219
+ const byValue = shapes.get(fam);
220
+ if (!byValue.has(r.segs[i])) byValue.set(r.segs[i], new Set());
221
+ byValue.get(r.segs[i]).add(r.segs.slice(i + 1).map((x) => (isParam(x) ? '{}' : '*')).join('/'));
222
+ }
223
+ const fold = new Set();
224
+ for (const [fam, byValue] of shapes) {
225
+ const bySignature = new Map();
226
+ for (const [value, set] of byValue) {
227
+ const sig = [...set].sort().join('|');
228
+ if (!bySignature.has(sig)) bySignature.set(sig, []);
229
+ bySignature.get(sig).push(value);
230
+ }
231
+ for (const values of bySignature.values()) {
232
+ if (values.length >= DISTINCT) values.forEach((v) => fold.add(`${fam}|${v}`));
233
+ }
234
+ }
235
+ if (!fold.size) continue;
236
+ const merged = new Map();
237
+ for (const r of routes) {
238
+ const folded = r.segs.length > i && fold.has(`${familyOf(r)}|${r.segs[i]}`);
239
+ const segs = folded ? [...r.segs.slice(0, i), `{${paramName(r.segs[i - 1])}}`, ...r.segs.slice(i + 1)] : r.segs;
240
+ const key = [r.kind, r.method, segs.join('/')].join('|');
241
+ const into = merged.get(key);
242
+ const examples = folded && !r.examples.length ? [r.template] : r.examples;
243
+ if (!into) {
244
+ merged.set(key, { ...r, segs, examples: [...examples], statuses: new Map(r.statuses), clients: new Set(r.clients), hosts: new Set(r.hosts) });
245
+ continue;
246
+ }
247
+ into.calls += r.calls;
248
+ into.variants += r.variants;
249
+ for (const x of examples) if (into.examples.length < 3) into.examples.push(x);
250
+ for (const [st, c] of r.statuses) into.statuses.set(st, (into.statuses.get(st) || 0) + c);
251
+ for (const c of r.clients) if (into.clients.size < MAX_CLIENTS) into.clients.add(c);
252
+ for (const h of r.hosts) into.hosts.add(h);
253
+ into.last = latest(into.last, r.last);
254
+ into.lastOk = latest(into.lastOk, r.lastOk) ?? into.lastOk ?? r.lastOk;
255
+ }
256
+ routes = [...merged.values()];
257
+ }
258
+ return routes.map((r) => ({ ...r, template: unique(r.segs).join('/') }));
259
+ }
260
+
261
+ /** {id}/{id} → {id}/{id2}: a template can't repeat a parameter name. */
262
+ function unique(segs) {
263
+ const seen = new Map();
264
+ return segs.map((s) => {
265
+ const m = /^\{(.+)\}$/.exec(s);
266
+ if (!m) return s;
267
+ const base = m[1].replace(/\d+$/, '');
268
+ const n = (seen.get(base) || 0) + 1;
269
+ seen.set(base, n);
270
+ return n === 1 ? `{${base}}` : `{${base}${n}}`;
271
+ });
272
+ }
273
+
274
+ function paramName(prev) {
275
+ const m = /^([a-z][a-z0-9]*?)(e?s)?$/i.exec(prev || '');
276
+ return m && m[1].length > 1 && !prev.startsWith('{') ? `${m[1]}Id` : 'id';
277
+ }
278
+
279
+ const median = (xs) => (xs.length ? [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)] : null);
280
+ // Buckets about 3% wide: a catch-all page that echoes the path varies by a few bytes.
281
+ const bucket = (bytes) => Math.round(Math.log(bytes + 64) / Math.log(1.03));
282
+
283
+ function record(map, key, base, rec, host) {
284
+ if (!map.has(key)) map.set(key, { ...base, calls: 0, statuses: new Map(), clients: new Set(), hosts: new Set(), last: null, lastOk: null });
285
+ const e = map.get(key);
286
+ e.calls++;
287
+ const s = rec.status || null;
288
+ e.statuses.set(s, (e.statuses.get(s) || 0) + 1);
289
+ if (rec.userAgent && e.clients.size < MAX_CLIENTS) e.clients.add(rec.userAgent);
290
+ if (host) e.hosts.add(host);
291
+ e.last = latest(e.last, rec.time);
292
+ if (ok(s)) e.lastOk = latest(e.lastOk, rec.time) ?? e.lastOk ?? true;
293
+ }
294
+
295
+ // 304 means the client's copy of an existing resource is current. Other 3xx
296
+ // prove nothing: an HTTP→HTTPS redirect answers every path, real or not.
297
+ const ok = (s) => (s >= 200 && s < 300) || s === 304;
298
+ const latest = (a, t) => (typeof t === 'number' && (typeof a !== 'number' || t > a) ? t : a);
299
+ const times = (n) => (n === 1 ? 'once' : `${n} times`);
300
+
301
+ function count(e, test) {
302
+ let n = 0;
303
+ for (const [s, c] of e.statuses) if (test(s)) n += c;
304
+ return n;
305
+ }
306
+
307
+ function judge(e, catchAll) {
308
+ const served = count(e, ok);
309
+ const exists = count(e, (s) => s === 401 || s === 403 || s === 400 || s === 422 || s === 429 || (s >= 500 && s < 600));
310
+ const unknown = count(e, (s) => s === null);
311
+ if (!served && !exists && !unknown) return null; // only 404/405: it isn't there
312
+ const write = WRITE.has(e.method);
313
+ const doubtful = [...e.hosts].some((h) => catchAll.has(h));
314
+ const conventional = Boolean(e.fixed);
315
+ let severity = 'low';
316
+ if (served && !doubtful && !conventional) {
317
+ if (e.kind === 'shadow-endpoint') severity = write || PRIVILEGED_PATH.test(e.template) ? 'high' : 'medium';
318
+ else severity = write ? 'high' : 'medium';
319
+ }
320
+ const notes = [describe(e)];
321
+ if (e.variants > 1) notes.push(`${e.variants} routes folded into this one, e.g. ${e.examples.join(', ')}`);
322
+ if (e.current) notes.push(`now ${e.current}`);
323
+ if (conventional) notes.push('a conventional health or discovery endpoint');
324
+ if (!served && unknown === e.calls) notes.push('the logs have no status, so it may not exist');
325
+ else if (!served) notes.push('never answered 2xx, but it exists');
326
+ if (doubtful) notes.push('this host looks like a catch-all');
327
+ return finding(e.kind, severity, { endpoint: `${e.method} ${e.template}`, detail: notes.join('; '), source: 'traffic' });
328
+ }
329
+
330
+ function describe(e) {
331
+ const statuses = [...e.statuses].sort((a, b) => b[1] - a[1]).slice(0, 4).map(([s, n]) => `${s ?? '?'}×${n}`).join(' ');
332
+ const last = typeof e.last === 'number' ? `, last ${new Date(e.last).toISOString().slice(0, 10)}` : '';
333
+ return `${e.calls} ${e.calls === 1 ? 'call' : 'calls'} from ${e.clients.size || '?'} ${e.clients.size === 1 ? 'client' : 'clients'} (${statuses})${last}`;
334
+ }
335
+
336
+ /** Operations whose x-sunset date has passed and that still answered 2xx after it. */
337
+ function sunsetPassed(spec, documented, now) {
338
+ const out = [];
339
+ if (!spec) return out;
340
+ for (const [path, item] of Object.entries(spec.paths || {})) {
341
+ if (!item || typeof item !== 'object') continue;
342
+ for (const [method, op] of Object.entries(item)) {
343
+ if (!op || typeof op !== 'object' || !op['x-sunset']) continue;
344
+ const v = op['x-sunset'];
345
+ const sunset = (v instanceof Date ? v : new Date(String(v))).getTime();
346
+ if (Number.isNaN(sunset) || sunset > now) continue;
347
+ const e = documented.get(`${method.toUpperCase()} ${path}`);
348
+ if (!e || !e.lastOk || (typeof e.lastOk === 'number' && e.lastOk < sunset)) continue;
349
+ out.push(finding('sunset-passed', 'medium', {
350
+ endpoint: `${method.toUpperCase()} ${path}`,
351
+ detail: `sunset ${new Date(sunset).toISOString().slice(0, 10)}; ${describe(e)}`,
352
+ source: 'traffic',
353
+ }));
354
+ }
355
+ }
356
+ return out;
357
+ }
358
+
359
+ function knownVersions(spec, matcher) {
360
+ const found = new Set();
361
+ const scan = (p) => p.split('/').forEach((s) => { if (VERSION.test(s)) found.add(s.toLowerCase()); });
362
+ for (const p of Object.keys(spec.paths || {})) scan(p);
363
+ matcher.basePaths.forEach(scan);
364
+ return [...found];
365
+ }
366
+
367
+ module.exports = { createInventory, PROBES };
@@ -48,6 +48,12 @@ const RULES = {
48
48
  'cors-any-origin': ['API8', 'CORS allows any origin', 'Confirm this API holds nothing private; otherwise allow specific origins.'],
49
49
  'insecure-cookie': ['API8', 'Session cookie missing protections', 'Set Secure, HttpOnly and SameSite.'],
50
50
  'version-disclosure': ['API8', 'Server software version disclosed', 'Remove version details from response headers.'],
51
+ // Traffic vs spec
52
+ 'shadow-endpoint': ['API9', 'Undocumented endpoint receiving traffic', 'Document it and give it the same review as the rest, or remove it.'],
53
+ 'undocumented-method': ['API5', 'Undocumented method on a documented path', 'Answer 405 for it, or document and secure it.'],
54
+ 'zombie-version': ['API9', 'Retired API version still served', 'Turn the old version off, or keep patching it like the current one.'],
55
+ 'sunset-passed': ['API9', 'Still served after its sunset date', 'Turn it off as announced, or move x-sunset in the spec.'],
56
+ 'exposed-sensitive-path': ['API8', 'Sensitive file or debug endpoint served', 'Block it at the server or proxy, and rotate anything it exposed.'],
51
57
  'stack-trace': ['API8', 'Stack trace in an error response', 'Return a generic error and log the detail on the server.'],
52
58
  };
53
59
 
@@ -0,0 +1,179 @@
1
+ 'use strict';
2
+
3
+ // Opt-in usage telemetry. Off unless the user says yes once, or sets
4
+ // SPECSHIELD_TELEMETRY=1 (CI). DO_NOT_TRACK=1 and SPECSHIELD_TELEMETRY=0 always win.
5
+ //
6
+ // One event per command run: the command name, the names of the flags given (never
7
+ // their values), the exit code, the duration, the CLI version, OS, Node major and
8
+ // which CI it ran in. No paths, spec contents, hosts, repo names or results.
9
+ //
10
+ // The event is sent by a detached child process started from the 'exit' handler,
11
+ // so it never delays the command, never changes its exit code and survives
12
+ // process.exit().
13
+
14
+ const crypto = require('crypto');
15
+ const path = require('path');
16
+ const { spawn } = require('child_process');
17
+
18
+ const DEFAULT_URL = 'https://specshield.io/api/telemetry/cli';
19
+ const DOCS = 'https://specshield.io/docs#telemetry';
20
+
21
+ const CI_VARS = [
22
+ ['GITHUB_ACTIONS', 'github'], ['GITLAB_CI', 'gitlab'], ['BITBUCKET_BUILD_NUMBER', 'bitbucket'],
23
+ ['CIRCLECI', 'circleci'], ['JENKINS_URL', 'jenkins'], ['BUILDKITE', 'buildkite'],
24
+ ['TF_BUILD', 'azure'], ['TRAVIS', 'travis'], ['CODEBUILD_BUILD_ID', 'codebuild'], ['CI', 'other'],
25
+ ];
26
+
27
+ function ciName(env = process.env) {
28
+ const hit = CI_VARS.find(([v]) => env[v]);
29
+ return hit ? hit[1] : null;
30
+ }
31
+
32
+ const truthy = (v) => /^(1|true|yes|on)$/i.test(String(v || '').trim());
33
+ const falsy = (v) => /^(0|false|no|off)$/i.test(String(v || '').trim());
34
+
35
+ /** 'on' | 'off' | 'unset', and why. Environment beats the saved choice. */
36
+ function state(config = {}, env = process.env) {
37
+ if (truthy(env.DO_NOT_TRACK)) return { status: 'off', source: 'DO_NOT_TRACK' };
38
+ if (falsy(env.SPECSHIELD_TELEMETRY)) return { status: 'off', source: 'SPECSHIELD_TELEMETRY' };
39
+ if (truthy(env.SPECSHIELD_TELEMETRY)) return { status: 'on', source: 'SPECSHIELD_TELEMETRY' };
40
+ if (config.telemetry === true) return { status: 'on', source: 'config' };
41
+ if (config.telemetry === false) return { status: 'off', source: 'config' };
42
+ return { status: 'unset', source: null };
43
+ }
44
+
45
+ /**
46
+ * The id that lets runs from one install (or one CI repository) be counted together.
47
+ * Locally: a random id saved on opt-in. In CI: a one-way hash of the CI's numeric
48
+ * project id, so a repository's runs group together without its name being sent.
49
+ */
50
+ function installId(config = {}, env = process.env) {
51
+ if (config.telemetryId) return config.telemetryId;
52
+ const project = env.GITHUB_REPOSITORY_ID || env.CI_PROJECT_ID || env.BITBUCKET_REPO_UUID;
53
+ if (project) return 'ci-' + crypto.createHash('sha256').update('specshield-telemetry:' + project).digest('hex').slice(0, 32);
54
+ return 'run-' + crypto.randomUUID();
55
+ }
56
+
57
+ /** "gate", "bdct can-i-deploy", "usage import" — the path below the root program. */
58
+ function commandName(cmd) {
59
+ const names = [];
60
+ for (let c = cmd; c && c.parent; c = c.parent) names.unshift(c.name());
61
+ return names.join(' ');
62
+ }
63
+
64
+ /** Long names of the flags the user typed — never their values. */
65
+ function flagNames(cmd) {
66
+ return cmd.options
67
+ .filter((o) => cmd.getOptionValueSource(o.attributeName()) === 'cli')
68
+ .map((o) => o.long || o.short)
69
+ .filter(Boolean)
70
+ .sort();
71
+ }
72
+
73
+ function buildEvent({ cmd, exitCode, durationMs, version, config, env = process.env }) {
74
+ return {
75
+ id: installId(config, env),
76
+ version,
77
+ command: commandName(cmd),
78
+ flags: flagNames(cmd),
79
+ exitCode: Number.isInteger(exitCode) ? exitCode : 0,
80
+ durationMs: Math.max(0, Math.round(durationMs)),
81
+ ci: ciName(env),
82
+ os: process.platform,
83
+ arch: process.arch,
84
+ node: Number(process.versions.node.split('.')[0]),
85
+ };
86
+ }
87
+
88
+ /** Ask once, only when a person is clearly at the keyboard. */
89
+ function shouldPrompt({ config = {}, env = process.env, argv = process.argv, stdin = process.stdin, stderr = process.stderr, command = '' }) {
90
+ if (state(config, env).status !== 'unset') return false;
91
+ if (ciName(env)) return false;
92
+ if (!stdin.isTTY || !stderr.isTTY) return false;
93
+ if (command === 'telemetry' || command.startsWith('telemetry ')) return false;
94
+ if (argv.includes('--json')) return false;
95
+ return true;
96
+ }
97
+
98
+ const PROMPT = [
99
+ '',
100
+ 'Help improve SpecShield? Send anonymous usage data: the command and flag names you',
101
+ 'use, exit code, duration, CLI version and OS. Never file paths, specs, values or',
102
+ `results. Change it any time with "specshield telemetry disable". ${DOCS}`,
103
+ 'Enable anonymous telemetry? [y/N] ',
104
+ ].join('\n');
105
+
106
+ async function ask(stdin = process.stdin, stderr = process.stderr) {
107
+ const readline = require('readline');
108
+ const rl = readline.createInterface({ input: stdin, output: stderr });
109
+ try {
110
+ // Ctrl-D or a closed stdin ends the question with no answer: that's a no. Without the
111
+ // 'close' listener the promise never settles, Node exits 0 and the command never runs.
112
+ const answer = await new Promise((resolve) => {
113
+ rl.on('close', () => resolve(''));
114
+ rl.question(PROMPT, resolve);
115
+ });
116
+ return /^y(es)?$/i.test(String(answer).trim());
117
+ } finally {
118
+ rl.close();
119
+ }
120
+ }
121
+
122
+ function withChoice(config, enabled) {
123
+ const next = { ...config, telemetry: enabled };
124
+ if (enabled && !next.telemetryId) next.telemetryId = crypto.randomUUID();
125
+ if (!enabled) delete next.telemetryId;
126
+ return next;
127
+ }
128
+
129
+ /** Fire and forget from a detached child; prints instead with SPECSHIELD_TELEMETRY_DEBUG=1. */
130
+ function send(event, env = process.env) {
131
+ const body = JSON.stringify(event);
132
+ if (truthy(env.SPECSHIELD_TELEMETRY_DEBUG)) {
133
+ process.stderr.write(`[telemetry] ${body}\n`);
134
+ return;
135
+ }
136
+ try {
137
+ const child = spawn(process.execPath, [path.join(__dirname, 'telemetrySend.js'), env.SPECSHIELD_TELEMETRY_URL || DEFAULT_URL, body], {
138
+ detached: true, stdio: 'ignore', windowsHide: true,
139
+ });
140
+ child.unref();
141
+ } catch {
142
+ // Telemetry must never affect the command.
143
+ }
144
+ }
145
+
146
+ /**
147
+ * Wire telemetry into the program: ask once before the first interactive command,
148
+ * and send one event when the process exits.
149
+ */
150
+ function install(program, { version, loadConfig, saveConfig }) {
151
+ let started = null;
152
+ let config = {};
153
+
154
+ program.hook('preAction', async (_root, actionCommand) => {
155
+ try {
156
+ config = await loadConfig();
157
+ if (shouldPrompt({ config, command: commandName(actionCommand) })) {
158
+ config = withChoice(config, await ask());
159
+ await saveConfig(config);
160
+ process.stderr.write(config.telemetry ? 'Thanks — telemetry enabled.\n\n' : 'Telemetry stays off.\n\n');
161
+ }
162
+ } catch {
163
+ // A broken config file or a closed stdin must not stop the command.
164
+ }
165
+ if (state(config).status !== 'on') return;
166
+ started = { cmd: actionCommand, at: Date.now() };
167
+ });
168
+
169
+ process.on('exit', (code) => {
170
+ if (!started) return;
171
+ const exitCode = typeof code === 'number' ? code : (process.exitCode || 0);
172
+ send(buildEvent({ cmd: started.cmd, exitCode, durationMs: Date.now() - started.at, version, config }));
173
+ });
174
+ }
175
+
176
+ module.exports = {
177
+ install, state, installId, commandName, flagNames, buildEvent, shouldPrompt, withChoice, send, ciName, ask,
178
+ DEFAULT_URL, DOCS,
179
+ };
@@ -0,0 +1,21 @@
1
+ 'use strict';
2
+
3
+ // Runs detached from the CLI: POST one telemetry event, give up after 3 s, exit quietly.
4
+ // argv: <url> <json body>
5
+
6
+ const [url, body] = process.argv.slice(2);
7
+
8
+ try {
9
+ const target = new URL(url);
10
+ const lib = target.protocol === 'http:' ? require('http') : require('https');
11
+ const req = lib.request(target, {
12
+ method: 'POST',
13
+ headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(body) },
14
+ timeout: 3000,
15
+ }, (res) => { res.resume(); res.on('end', () => process.exit(0)); });
16
+ req.on('timeout', () => req.destroy());
17
+ req.on('error', () => process.exit(0));
18
+ req.end(body);
19
+ } catch {
20
+ process.exit(0);
21
+ }
@@ -15,6 +15,7 @@
15
15
  * JSON with a host field, absolute request URLs), else null
16
16
  * query raw query string without the leading '?' ('' when absent)
17
17
  * status numeric status code, or null
18
+ * bytes response body size (combined format only), or null
18
19
  * time epoch ms, or null when the line carries no parseable timestamp
19
20
  * userAgent raw User-Agent, or null
20
21
  * extras quoted fields after the User-Agent (combined format only) —
@@ -34,7 +35,7 @@ const HTTP_METHODS = new Set(['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'HEAD', '
34
35
  // ─── combined / common ──────────────────────────────────────────────────────
35
36
 
36
37
  // host ident authuser [time] "request" status bytes ["referer" "user-agent"] [extras...]
37
- const COMBINED_RE = /^(\S+) \S+ \S+ \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}|-) \S+(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(.*)$/;
38
+ const COMBINED_RE = /^(\S+) \S+ \S+ \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}|-) (\S+)(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(.*)$/;
38
39
 
39
40
  function parseCombined(line) {
40
41
  const m = COMBINED_RE.exec(line);
@@ -44,9 +45,10 @@ function parseCombined(line) {
44
45
  return {
45
46
  ...req,
46
47
  status: m[4] === '-' ? null : Number(m[4]),
48
+ bytes: /^\d+$/.test(m[5]) ? Number(m[5]) : null,
47
49
  time: parseClfTime(m[2]),
48
- userAgent: m[6] !== undefined ? nullIfDash(unescape(m[6])) : null,
49
- extras: m[7] ? tokenize(m[7]).map(nullIfDash) : [],
50
+ userAgent: m[7] !== undefined ? nullIfDash(unescape(m[7])) : null,
51
+ extras: m[8] ? tokenize(m[8]).map(nullIfDash) : [],
50
52
  json: null,
51
53
  };
52
54
  }
@@ -221,12 +223,17 @@ function parseAnyTime(v) {
221
223
 
222
224
  const PARSERS = { combined: parseCombined, alb: parseAlb, json: parseJsonLine };
223
225
 
224
- /** Picks the format that parses the most of the sample lines. */
226
+ // A line can have a format's shape without being a request: OpenLiteSpeed logs
227
+ // TLS probes and dropped connections as `"" 200`. They still say which format it is.
228
+ const SHAPES = { combined: COMBINED_RE };
229
+
230
+ /** Picks the format that parses (or at least fits) the most of the sample lines. */
225
231
  function detectFormat(sampleLines) {
226
232
  let best = null;
227
233
  let bestHits = 0;
228
234
  for (const [name, parse] of Object.entries(PARSERS)) {
229
- const hits = sampleLines.filter((l) => parse(l)).length;
235
+ const shape = SHAPES[name];
236
+ const hits = sampleLines.filter((l) => parse(l) || (shape && shape.test(l))).length;
230
237
  if (hits > bestHits) { best = name; bestHits = hits; }
231
238
  }
232
239
  return best;
@@ -241,9 +248,9 @@ function splitRequestLine(s) {
241
248
  if (parts.length < 2) return null;
242
249
  const method = parts[0].toUpperCase();
243
250
  if (!HTTP_METHODS.has(method)) return null;
244
- const { path, query, host } = splitTarget(parts[1]);
251
+ const { path, query, host, trailingSlash } = splitTarget(parts[1]);
245
252
  if (!path) return null;
246
- return { method, path, query, host };
253
+ return { method, path, query, host, trailingSlash };
247
254
  }
248
255
 
249
256
  function splitTarget(target) {
@@ -262,8 +269,9 @@ function splitTarget(target) {
262
269
  let path = q >= 0 ? t.slice(0, q) : t;
263
270
  const query = q >= 0 ? t.slice(q + 1) : '';
264
271
  path = path.replace(/\/{2,}/g, '/');
265
- if (path.length > 1 && path.endsWith('/')) path = path.slice(0, -1);
266
- return { path, query, host };
272
+ const trailingSlash = path.length > 1 && path.endsWith('/');
273
+ if (trailingSlash) path = path.slice(0, -1);
274
+ return { path, query, host, trailingSlash };
267
275
  }
268
276
 
269
277
  /** "API.Example.com:443" → "api.example.com"; "[::1]:80" → "[::1]". */