specshield 3.5.3 → 3.5.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,6 +1,13 @@
1
1
  # SpecShield CLI changelog
2
2
 
3
- ## Unreleased — security audit
3
+ ## Unreleased — shadow APIs
4
+
5
+ ### Added
6
+
7
+ - `specshield audit --logs <files>` compares real traffic with the spec (OWASP API9). It reports undocumented endpoints and methods that answer requests, retired versions still served, operations past their `x-sunset`, and exposed files such as `/.env`. Severity follows the response status; redirects count as no evidence. Catch-all pages are recognised by response size (the home page's, or one shared by other probes). Routes are folded by structure, and trailing `{path}` parameters span segments. A HAR passed with `--har` gets the same checks. Validated on six real public log datasets, 9.9M requests in all.
8
+ - Log import records the response size (combined format) and recognises combined logs whose first lines have an empty request (OpenLiteSpeed).
9
+
10
+ ## 3.5.3 — 2026-11-10 — security audit
4
11
 
5
12
  ### Added
6
13
 
package/README.md CHANGED
@@ -604,11 +604,32 @@ Audits a whole spec and, optionally, captured traffic, by [OWASP API Security To
604
604
  ```bash
605
605
  specshield audit openapi.yaml
606
606
  specshield audit openapi.yaml --har staging.har --host api.acme.com
607
+ specshield audit openapi.yaml --logs /var/log/nginx/access.log*.gz
607
608
  specshield audit openapi.yaml --fail-on medium --sarif audit.sarif
608
609
  ```
609
610
 
610
611
  **From the spec:** missing or undefined authentication, public writes and admin endpoints, API keys and secrets in URLs, basic auth, deprecated OAuth flows, OAuth over HTTP, credentials in responses, personal data on public endpoints, privileged request fields, page sizes with no maximum, client-supplied URLs, plain-HTTP and non-production servers.
611
612
 
613
+ **From access logs (`--logs`) or a HAR file, compared with the spec (OWASP API9):**
614
+
615
+ | Finding | What it means |
616
+ |---|---|
617
+ | `shadow-endpoint` | A path the spec doesn't have is answering requests |
618
+ | `undocumented-method` | A documented path answers a method the spec doesn't list, e.g. `DELETE` |
619
+ | `zombie-version` | `/v1/...` still answers where the spec documents `/v2/...` |
620
+ | `sunset-passed` | An operation past its `x-sunset` date still answers |
621
+ | `exposed-sensitive-path` | `/.env`, `/.git/config`, `/actuator/env` and similar answered 200 (checked even without a spec) |
622
+
623
+ How sure a finding is comes from the response, not the request:
624
+
625
+ - **Status.** 2xx (or 304) means the endpoint served the request: high for writes and internal paths, medium otherwise. 401/403/400 means it exists but refused (low). 404/405 means it isn't there, and redirects prove nothing (an HTTP→HTTPS redirect answers every path), so neither is reported.
626
+ - **Catch-all pages.** Many sites answer every unknown path with the same page. When the logs record response sizes, a probe answered with the site's home page size, or the same size as other probes got, is downgraded; a size no other probe got stays high. Without sizes, a site that answered 200 to three or more probes is treated as a catch-all.
627
+ - **Routes.** IDs are templated, and object names that don't look like IDs are folded once four or more sit in the same place with the same shape after them (`/things/noxus/x`, `/things/fooab/x` → `/things/{thingId}/x`). Health and discovery endpoints (`/readyz`, `/openapi/v3`…) are reported low. A trailing `{path}` parameter may span several segments, as in Kubernetes' `/proxy/{path}`.
628
+
629
+ Logs in the same formats as `specshield usage import` are supported (combined, ALB, JSON lines, Kubernetes audit events, `.gz`). Per-site logs without a host are grouped by file name, so one site's catch-all doesn't affect another's findings. A file that can't be read is skipped with a warning. `--base-path` and `--host` narrow the logs to this API.
630
+
631
+ Validated on real, published traffic: a Kubernetes CI run (250,724 API requests against the official spec, every reported route independently confirmed absent from it), NASA's 1995 web logs (1.9M lines), Elastic's Apache sample, 90 days of secrepo.com logs, and a 2025 forensic dataset of 7.4M requests from a compromised hosting server, where every probe answered 200 was correctly identified as a catch-all page.
632
+
612
633
  **From a HAR file (`--har`):** secrets in URLs and responses, regulated and personal data, plain HTTP, missing HSTS and `nosniff`, risky CORS, unprotected session cookies, version banners, stack traces. Findings say where a secret was seen and never print its value. Only API responses are checked, so use `--host` to leave out third parties in a browser capture.
613
634
 
614
635
  | Option | Default | |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "specshield",
3
- "version": "3.5.3",
3
+ "version": "3.5.4",
4
4
  "description": "Contract compatibility testing for APIs — catch breaking OpenAPI changes before they reach your consumers, with can-i-deploy deploy gating and GitHub PR checks. (a.k.a. bidirectional contract testing.)",
5
5
  "main": "src/cli.js",
6
6
  "homepage": "https://specshield.io",
@@ -8,6 +8,8 @@ const { loadSpec } = require('../core/loadSpec');
8
8
  const { parseSpec } = require('../core/parseSpec');
9
9
  const { auditSpec, consolidate } = require('../core/security/audit');
10
10
  const { auditHar } = require('../core/security/auditHar');
11
+ const { createInventory } = require('../core/security/inventory');
12
+ const { importLogFile } = require('../core/usage/importLogs');
11
13
  const { OWASP, RULES, SEVERITIES } = require('../core/security/rules');
12
14
  const W = require('../core/gate/waivers');
13
15
  const { plural } = require('../core/usage/format');
@@ -21,7 +23,10 @@ const audit = new Command('audit')
21
23
  .description('Security audit of an OpenAPI spec and captured traffic, by OWASP API Security Top 10 category. Sends no requests')
22
24
  .argument('[spec]', 'The OpenAPI spec to audit')
23
25
  .option('--har <file...>', 'HAR capture(s) to check for leaked secrets, personal data and missing protections')
24
- .option('--host <host...>', 'Only check these hosts in the capture')
26
+ .option('--logs <file...>', 'Access logs (.gz ok): find undocumented endpoints, methods and old versions still answering, and exposed files like /.env')
27
+ .option('--format <format>', 'Access log format: auto | combined | alb | json', 'auto')
28
+ .option('--base-path <path>', 'Path prefix the API is served under, e.g. /api (default: from the spec servers)')
29
+ .option('--host <host...>', 'Only check these hosts in the capture and logs')
25
30
  .option('--fail-on <severity>', 'Exit 1 at or above this severity: high, medium, low, or none', 'high')
26
31
  .option('--waivers <file>', `Waiver file (default: ${W.DEFAULT_FILE})`, W.DEFAULT_FILE)
27
32
  .option('--all', 'List every low-severity finding')
@@ -30,17 +35,29 @@ const audit = new Command('audit')
30
35
  .option('--json', 'Output machine-readable JSON')
31
36
  .action(async (specPath, opts) => {
32
37
  try {
33
- if (!specPath && !opts.har) throw new Error('Nothing to audit: pass a spec, --har <file>, or both');
38
+ if (!specPath && !opts.har && !opts.logs) throw new Error('Nothing to audit: pass a spec, --har <file>, --logs <file>, or a combination');
34
39
  if (!FAIL_ON.includes(opts.failOn)) throw new Error(`--fail-on must be one of ${FAIL_ON.join(', ')}`);
35
40
 
36
41
  let specText = null;
37
42
  let doc = null;
38
43
  if (specPath) { specText = await loadSpec(specPath); doc = parseSpec(specText, specPath); }
39
- const har = opts.har ? auditHar(opts.har, { hosts: opts.host ? new Set(opts.host.map((h) => h.toLowerCase())) : null }) : null;
44
+ const hosts = opts.host ? new Set(opts.host.map((h) => h.toLowerCase())) : null;
45
+ const traffic = opts.har || opts.logs ? createInventory(doc, { basePath: opts.basePath, hosts }) : null;
46
+ const har = opts.har ? auditHar(opts.har, { hosts, inventory: traffic }) : null;
47
+ const unread = [];
48
+ for (const file of opts.logs || []) {
49
+ const site = siteOf(file);
50
+ try {
51
+ await importLogFile(file, { format: opts.format, aggregator: { add: (rec) => { rec.site = site; traffic.add(rec); } } });
52
+ } catch (err) {
53
+ unread.push({ file, reason: err.message.split('\n')[0] });
54
+ }
55
+ }
56
+ if (opts.logs && unread.length === opts.logs.length) throw new Error(`No log could be read: ${unread[0].reason}`);
40
57
 
41
58
  const now = Date.now();
42
59
  const { waivers } = await W.loadWaivers(path.resolve(opts.waivers), { maxDays: W.DEFAULT_MAX_DAYS, now, file: opts.waivers });
43
- const all = [...(doc ? auditSpec(doc) : []), ...(har ? har.findings : [])];
60
+ const all = [...(doc ? auditSpec(doc) : []), ...(har ? har.findings : []), ...(traffic ? traffic.findings() : [])];
44
61
  const { blocking, waived } = W.applyWaivers(all, waivers, now);
45
62
  const findings = sort(consolidate(blocking));
46
63
  const failing = findings.filter((f) => atLeast(f.severity, opts.failOn));
@@ -48,6 +65,7 @@ const audit = new Command('audit')
48
65
  spec: specPath || null,
49
66
  title: doc && doc.info ? doc.info.title || null : null,
50
67
  har: har ? { files: opts.har, ...har.stats } : null,
68
+ traffic: traffic ? { logs: opts.logs || [], unread, ...traffic.stats } : null,
51
69
  failOn: opts.failOn,
52
70
  counts: count(findings),
53
71
  owasp: byOwasp(findings),
@@ -68,6 +86,15 @@ const audit = new Command('audit')
68
86
  }
69
87
  });
70
88
 
89
+ /** "logs/shop.example.com.access_log.2025_04_26.gz" → "shop.example.com": which site a per-site log belongs to. */
90
+ function siteOf(file) {
91
+ const name = path.basename(file)
92
+ .replace(/\.gz$/i, '')
93
+ .replace(/([._-]\d{4}[._-]\d{2}[._-]\d{2}([._]\d+)?|(\.\d+)+)$/, '')
94
+ .replace(/[._-]?(access[._-]?log|access|log)$/i, '');
95
+ return name && !/^(web|ssl|other_vhosts)?[._-]?$/i.test(name) ? name : path.basename(path.dirname(path.resolve(file)));
96
+ }
97
+
71
98
  const rank = (s) => SEVERITIES.indexOf(s);
72
99
  const atLeast = (s, min) => min !== 'none' && rank(s) <= rank(min);
73
100
  const sort = (list) => [...list].sort((a, b) => rank(a.severity) - rank(b.severity) || a.rule.localeCompare(b.rule));
@@ -102,6 +129,17 @@ const MAX_LISTED = 5;
102
129
  function render(r, opts) {
103
130
  const lines = ['', chalk.bold(` SpecShield™ Audit — ${r.title || r.spec || 'captured traffic'}`), chalk.gray(' ─────────────────────────────────────────')];
104
131
  if (r.har) lines.push(chalk.gray(` Capture: ${plural(r.har.api, 'API response')} checked of ${plural(r.har.entries, 'request')}, ${plural(r.har.hosts, 'host')}`));
132
+ if (r.traffic && r.traffic.unread.length) {
133
+ lines.push(chalk.yellow(` ${plural(r.traffic.unread.length, 'log file')} skipped: ${r.traffic.unread.slice(0, 2).map((u) => u.reason).join('; ')}${r.traffic.unread.length > 2 ? '; …' : ''}`));
134
+ }
135
+ if (r.traffic && r.spec) {
136
+ const t = r.traffic;
137
+ lines.push(chalk.gray(` Traffic: ${plural(t.requests, 'request')}; ${t.documentedSeen} of ${t.documentedTotal} documented operations called, ${plural(t.endpoints, 'undocumented route')} seen`));
138
+ if (t.requests && t.considered === 0) lines.push(chalk.yellow(' No request matched this API. Check --host and --base-path.'));
139
+ if (t.overflow) lines.push(chalk.yellow(` ${plural(t.overflow, 'request')} not checked: too many distinct undocumented routes.`));
140
+ } else if (r.traffic) {
141
+ lines.push(chalk.gray(' Traffic: checked for exposed files only. Pass the spec to find undocumented endpoints.'));
142
+ }
105
143
  if (r.har && r.har.hosts > 1) lines.push(chalk.gray(' Several hosts in this capture: use --host to check only your own.'));
106
144
  lines.push('');
107
145
  lines.push(` ${chalk.red(`${r.counts.high} high`)} · ${chalk.yellow(`${r.counts.medium} medium`)} · ${chalk.gray(`${r.counts.low} low`)}${r.waived.length ? chalk.gray(` · ${r.waived.length} waived`) : ''}`);
@@ -167,7 +205,7 @@ function sarif(r, specText, harFiles) {
167
205
  level: LEVEL[f.severity],
168
206
  message: { text: `${f.title}${where(f) ? `: ${where(f)}` : ''}` },
169
207
  locations: [{ physicalLocation: {
170
- artifactLocation: { uri: f.source === 'spec' ? r.spec : harFiles[0] },
208
+ artifactLocation: { uri: f.source === 'spec' || !harFiles ? r.spec || 'traffic' : harFiles[0] },
171
209
  region: { startLine: f.source === 'spec' ? lineOf(specText, (f.affectedEndpoints[0] || '').split(' ')[1] || f.field) : 1 },
172
210
  } }],
173
211
  })),
@@ -176,4 +214,4 @@ function sarif(r, specText, harFiles) {
176
214
  }
177
215
 
178
216
  module.exports = audit;
179
- module.exports._internal = { sarif, render };
217
+ module.exports._internal = { sarif, render, siteOf };
@@ -28,7 +28,7 @@ const SECRET_VALUES = [
28
28
  ];
29
29
  const STACK = /Traceback \(most recent call last\)|Exception in thread "|\n\s+at [\w.$<>]+\([\w.]+\.(?:java|kt|scala):\d+\)|\n\s+at .+ \(.+\.[jt]s:\d+:\d+\)|\.php on line \d+/;
30
30
 
31
- function auditHar(files, { hosts = null } = {}) {
31
+ function auditHar(files, { hosts = null, inventory = null } = {}) {
32
32
  const found = new Map();
33
33
  const stats = { entries: 0, api: 0, otherHosts: 0, hosts: new Set() };
34
34
  const add = (rule, severity, opts) => {
@@ -38,6 +38,7 @@ function auditHar(files, { hosts = null } = {}) {
38
38
  for (const file of files) {
39
39
  for (const e of entriesOf(file)) {
40
40
  stats.entries++;
41
+ if (inventory) inventory.add(recordOf(e));
41
42
  checkEntry(e, { add, hosts, stats });
42
43
  }
43
44
  }
@@ -51,6 +52,24 @@ function entriesOf(file) {
51
52
  return doc.log.entries.filter((e) => e && e.request && e.request.url && e.response);
52
53
  }
53
54
 
55
+ /** The same shape the access-log parsers produce, plus the response type. */
56
+ function recordOf(e) {
57
+ let url;
58
+ try { url = new URL(e.request.url); } catch { return null; }
59
+ const time = Date.parse(e.startedDateTime);
60
+ return {
61
+ method: String(e.request.method || 'GET').toUpperCase(),
62
+ path: url.pathname,
63
+ query: url.search.replace(/^\?/, ''),
64
+ host: url.hostname.toLowerCase(),
65
+ status: e.response.status > 0 ? e.response.status : null,
66
+ time: Number.isNaN(time) ? null : time,
67
+ userAgent: ((e.request.headers || []).find((h) => h && /^user-agent$/i.test(h.name)) || {}).value || null,
68
+ mime: (e.response.content && e.response.content.mimeType) || null,
69
+ bytes: e.response.content && e.response.content.size >= 0 ? e.response.content.size : null,
70
+ };
71
+ }
72
+
54
73
  function headerMap(list) {
55
74
  const map = new Map();
56
75
  for (const h of list || []) {
@@ -0,0 +1,367 @@
1
+ 'use strict';
2
+
3
+ /**
4
+ * Improper inventory (OWASP API9) from real traffic: what the API serves that
5
+ * the spec doesn't say.
6
+ *
7
+ * shadow-endpoint a path the spec doesn't have, answering requests
8
+ * undocumented-method a documented path, answering a method the spec doesn't list
9
+ * zombie-version /v1/... still answering, where the spec documents /v2/...
10
+ * sunset-passed an operation past its x-sunset date, still answering
11
+ * exposed-sensitive-path /.env, /.git/config, /actuator/env... answering 200
12
+ *
13
+ * The response status decides how sure a finding is: a 2xx (or 304) means the
14
+ * endpoint served the request; 401/403/400 mean it exists; 404/405 and
15
+ * redirects say nothing.
16
+ * A host that answers 200 to scanner probes is most likely a catch-all (a
17
+ * single-page app's index.html), so its findings are downgraded, not trusted.
18
+ */
19
+
20
+ const { createRouteMatcher } = require('../usage/routeMatcher');
21
+ const { templatePath } = require('../har/pathTemplate');
22
+ const { finding } = require('./rules');
23
+
24
+ const MAX_ENTRIES = 2000;
25
+ const MAX_CLIENTS = 100;
26
+ const SKIP_METHODS = new Set(['HEAD', 'OPTIONS']);
27
+ const WRITE = new Set(['POST', 'PUT', 'PATCH', 'DELETE']);
28
+ const STATIC = /\.(m?js|css|map|png|jpe?g|gif|svg|ico|webp|avif|woff2?|ttf|eot|html?|txt|webmanifest)$/i;
29
+ const PRIVILEGED_PATH = /(^|\/)(admin|internal|debug|actuator|management|private|_[a-z]+)(\/|$)/i;
30
+ const VERSION = /^v\d+(\.\d+)?$/i;
31
+ // Every API has these, and few specs list them: worth knowing about, not alarming.
32
+ const CONVENTIONAL = /^\/(health|healthz|livez|readyz|ping|status|version|openapi|swagger|api-docs|v\d+\/api-docs|\.well-known)(\/|$|\.json$|\.yaml$)/i;
33
+ // A position with this many distinct values is a parameter, whatever the values look like.
34
+ const DISTINCT = 4;
35
+
36
+ // Paths no production server should answer with content.
37
+ const PROBES = [
38
+ ['environment file', /^\/(.*\/)?\.env(\.[\w-]+)?$/i],
39
+ ['git repository', /^\/(.*\/)?\.git\/(config|HEAD|index)$/i],
40
+ ['cloud credentials', /^\/(.*\/)?\.aws\/credentials$/i],
41
+ ['Spring Boot actuator', /^\/(.*\/)?actuator\/(env|heapdump|threaddump|configprops|beans|mappings|loggers|jolokia|shutdown)$/i],
42
+ ['heap dump', /^\/(.*\/)?heapdump$/i],
43
+ ['Go debug endpoint', /^\/debug\/(pprof|vars)(\/.*)?$/i],
44
+ ['server status page', /^\/server-(status|info)$/i],
45
+ ['PHP info page', /^\/(.*\/)?phpinfo\.php$/i],
46
+ ['database backup', /^\/(.*\/)?[\w-]*(backup|dump|db)[\w-]*\.(sql|sql\.gz|bak)$/i],
47
+ ];
48
+ // This many probe paths answered with the same page means the host answers everything with it.
49
+ const CATCH_ALL = 3;
50
+ const MAX_SIZES = 50;
51
+
52
+ function createInventory(spec, { basePath, hosts = null, now = Date.now() } = {}) {
53
+ const matcher = spec ? createRouteMatcher(spec, { basePath }) : null;
54
+ const versions = spec ? knownVersions(spec, matcher) : [];
55
+ const entries = new Map();
56
+ const probes = new Map();
57
+ const probeHits = new Map(); // site → probe path → probe answered 2xx
58
+ const home = new Map(); // site → size buckets of its home page
59
+ const documented = new Map();
60
+ const stats = { requests: 0, considered: 0, outside: 0, static: 0, otherHosts: 0, overflow: 0 };
61
+
62
+ function add(rec) {
63
+ if (!rec || !rec.method || typeof rec.path !== 'string') return;
64
+ stats.requests++;
65
+ const method = String(rec.method).toUpperCase();
66
+ const host = rec.host ? String(rec.host).toLowerCase() : null;
67
+ if (hosts && host && !hosts.has(host)) { stats.otherHosts++; return; }
68
+ // Per-site logs record no host; the file they came from stands in, so one
69
+ // site's catch-all page doesn't cast doubt on another's findings.
70
+ const site = host || (rec.site ? String(rec.site) : null);
71
+ if (SKIP_METHODS.has(method)) return;
72
+ let path = rec.path.replace(/\/{2,}/g, '/');
73
+ if (path.length > 1) path = path.replace(/\/$/, '');
74
+ // A capture tells us what came back; an HTML page is the app, not the API.
75
+ const html = rec.mime ? /html/i.test(rec.mime) : false;
76
+
77
+ // Single-page apps and CMS rewrites answer unknown paths with the home page.
78
+ if (path === '/' && method === 'GET' && ok(rec.status) && typeof rec.bytes === 'number') {
79
+ if (!home.has(site)) home.set(site, new Set());
80
+ const sizes = home.get(site);
81
+ const b = bucket(rec.bytes);
82
+ if (sizes.size < MAX_SIZES) [b - 1, b, b + 1].forEach((x) => sizes.add(x));
83
+ }
84
+
85
+ const probe = PROBES.find(([, re]) => re.test(path));
86
+ if (probe) {
87
+ // "/.env/" is a directory request: whatever answered it, it wasn't the file.
88
+ const asDirectory = (rec.trailingSlash || (rec.path.length > 1 && rec.path.endsWith('/'))) && !/^\/debug\//.test(path);
89
+ if (ok(rec.status) && !html && !asDirectory) {
90
+ const key = `${site || ''} ${path}`;
91
+ if (!probes.has(key)) probes.set(key, { label: probe[0], path, host: site, calls: 0, last: null, sizes: [] });
92
+ const p = probes.get(key);
93
+ p.calls++;
94
+ p.last = latest(p.last, rec.time);
95
+ if (typeof rec.bytes === 'number' && p.sizes.length < MAX_SIZES) p.sizes.push(rec.bytes);
96
+ if (!probeHits.has(site)) probeHits.set(site, new Map());
97
+ probeHits.get(site).set(path, p);
98
+ }
99
+ return;
100
+ }
101
+ if (!matcher) return;
102
+ if (path === '/' || STATIC.test(path)) { stats.static++; return; }
103
+ if (html) return;
104
+ stats.considered++;
105
+
106
+ const m = matcher.match(method, path) || greedy(method, path);
107
+ if (m && m.operation) { record(documented, `${method} ${m.template}`, { method, template: m.template }, rec, site); return; }
108
+ if (m) { remember('undocumented-method', method, m.template, rec, site); return; }
109
+ const older = versions.length ? otherVersion(method, path) : null;
110
+ if (older) { remember('zombie-version', method, templatePath(path).templated, rec, site, older); return; }
111
+ // Named like the spec names its paths, so a waiver for it reads the same.
112
+ const base = matcher.basePaths.find((b) => path === b || path.startsWith(`${b}/`));
113
+ if (matcher.basePaths.length && !base) { stats.outside++; return; }
114
+ remember('shadow-endpoint', method, templatePath(base ? path.slice(base.length) || '/' : path).templated, rec, site);
115
+ }
116
+
117
+ /**
118
+ * Kubernetes and others document "/proxy/{path}" where {path} spans several
119
+ * segments. OpenAPI doesn't say a parameter can't hold a slash, so try the
120
+ * path with its tail folded into the last segment.
121
+ */
122
+ function greedy(method, path) {
123
+ const segs = path.split('/');
124
+ for (let n = segs.length - 1; n > 1; n--) {
125
+ const m = matcher.match(method, [...segs.slice(0, n), segs.slice(n).join('%2F')].join('/'));
126
+ if (m && /\/\{[^}/]+\}$/.test(m.template)) return m;
127
+ }
128
+ return null;
129
+ }
130
+
131
+ /** /v1/orders when the spec serves /v2/orders: the same operation under a version the spec dropped. */
132
+ function otherVersion(method, path) {
133
+ const segs = path.split('/');
134
+ for (let i = 0; i < segs.length; i++) {
135
+ if (!VERSION.test(segs[i]) || versions.includes(segs[i].toLowerCase())) continue;
136
+ for (const v of versions) {
137
+ const swapped = [...segs.slice(0, i), v, ...segs.slice(i + 1)].join('/');
138
+ const m = matcher.match(method, swapped);
139
+ if (m && m.operation) return `${method} ${m.template} (${v})`;
140
+ }
141
+ }
142
+ return null;
143
+ }
144
+
145
+ function remember(kind, method, template, rec, host, current = null) {
146
+ const key = `${kind}|${method} ${template}`;
147
+ if (!entries.has(key) && entries.size >= MAX_ENTRIES) { stats.overflow++; return; }
148
+ record(entries, key, { kind, method, template, current, fixed: kind === 'shadow-endpoint' && CONVENTIONAL.test(template) }, rec, host);
149
+ }
150
+
151
+ /**
152
+ * Whether a probe's 2xx was the site's catch-all page rather than the file.
153
+ * With sizes: other probe paths got a response of the same size, so it isn't
154
+ * this file's content. Comparing per probe copes with a site that has several
155
+ * such pages, or whose page changed. Only probes count: an ordinary API
156
+ * response of a similar size says nothing. Without sizes: the site answered
157
+ * 2xx to several probes.
158
+ */
159
+ function catchAllAnswer(p) {
160
+ const size = median(p.sizes);
161
+ const siblings = [...probeHits.get(p.host).values()].filter((q) => q !== p);
162
+ if (size === null) {
163
+ const n = siblings.length + 1;
164
+ return n >= CATCH_ALL ? `this host answers 200 to ${n} probe paths, so it is probably a catch-all page` : null;
165
+ }
166
+ const b = bucket(size);
167
+ if ((home.get(p.host) || new Set()).has(b)) return `${size} bytes, the size of this host's home page; probably a rewrite to it`;
168
+ const same = siblings.filter((q) => { const m = median(q.sizes); return m !== null && Math.abs(bucket(m) - b) <= 1; });
169
+ if (same.length + 1 < CATCH_ALL) return null;
170
+ return `${size} bytes, the same as ${same.length} other probe paths got (e.g. ${same[0].path}); probably a catch-all page`;
171
+ }
172
+
173
+ function findings() {
174
+ const verdicts = new Map([...probes.values()].map((p) => [p, catchAllAnswer(p)]));
175
+ const catchAll = new Set([...verdicts].filter(([, v]) => v).map(([p]) => p.host));
176
+ const out = [];
177
+ for (const e of collapse([...entries.values()])) {
178
+ const f = judge(e, catchAll);
179
+ if (f) out.push(f);
180
+ }
181
+ for (const op of sunsetPassed(spec, documented, now)) out.push(op);
182
+ for (const [p, why] of verdicts) {
183
+ const size = median(p.sizes);
184
+ const detail = why ? `${p.label}; ${why}` : `${p.label}, answered 2xx ${times(p.calls)}${size !== null ? ` with ${size} bytes no other path got` : ''}`;
185
+ out.push(finding('exposed-sensitive-path', why ? 'low' : 'high', { endpoint: `GET ${p.path}`, field: p.host, detail, source: 'traffic' }));
186
+ }
187
+ return out;
188
+ }
189
+
190
+ return {
191
+ add,
192
+ findings,
193
+ get stats() {
194
+ const real = [...entries.values()].filter((e) => count(e, (st) => st !== 404 && st !== 405) > 0).length;
195
+ return { ...stats, documentedSeen: documented.size, documentedTotal: matcher ? matcher.operations.length : 0, endpoints: real };
196
+ },
197
+ };
198
+ }
199
+
200
+ /**
201
+ * Folds routes that differ only where a value varies: /things/noxus/x and
202
+ * /things/fooab/x become /things/{thingId}/x once DISTINCT siblings share both
203
+ * a parent and the shape of what follows them. Object names that don't look
204
+ * like IDs would otherwise each be a finding, and fixed words (/readyz,
205
+ * /metrics, /openapi) differ in what follows, so they stay apart. The first
206
+ * segment is never folded.
207
+ */
208
+ function collapse(list) {
209
+ let routes = list.map((e) => ({ ...e, segs: e.template.split('/'), variants: e.variants || 1, examples: e.examples || [] }));
210
+ const depth = Math.max(0, ...routes.map((r) => r.segs.length));
211
+ const isParam = (v) => v.startsWith('{');
212
+ for (let i = 2; i < depth; i++) {
213
+ const familyOf = (r) => [r.kind, r.method, r.segs.slice(0, i).join('/')].join('|');
214
+ const shapes = new Map(); // family → value → shapes of what follows
215
+ for (const r of routes) {
216
+ if (r.segs.length <= i || isParam(r.segs[i])) continue;
217
+ const fam = familyOf(r);
218
+ if (!shapes.has(fam)) shapes.set(fam, new Map());
219
+ const byValue = shapes.get(fam);
220
+ if (!byValue.has(r.segs[i])) byValue.set(r.segs[i], new Set());
221
+ byValue.get(r.segs[i]).add(r.segs.slice(i + 1).map((x) => (isParam(x) ? '{}' : '*')).join('/'));
222
+ }
223
+ const fold = new Set();
224
+ for (const [fam, byValue] of shapes) {
225
+ const bySignature = new Map();
226
+ for (const [value, set] of byValue) {
227
+ const sig = [...set].sort().join('|');
228
+ if (!bySignature.has(sig)) bySignature.set(sig, []);
229
+ bySignature.get(sig).push(value);
230
+ }
231
+ for (const values of bySignature.values()) {
232
+ if (values.length >= DISTINCT) values.forEach((v) => fold.add(`${fam}|${v}`));
233
+ }
234
+ }
235
+ if (!fold.size) continue;
236
+ const merged = new Map();
237
+ for (const r of routes) {
238
+ const folded = r.segs.length > i && fold.has(`${familyOf(r)}|${r.segs[i]}`);
239
+ const segs = folded ? [...r.segs.slice(0, i), `{${paramName(r.segs[i - 1])}}`, ...r.segs.slice(i + 1)] : r.segs;
240
+ const key = [r.kind, r.method, segs.join('/')].join('|');
241
+ const into = merged.get(key);
242
+ const examples = folded && !r.examples.length ? [r.template] : r.examples;
243
+ if (!into) {
244
+ merged.set(key, { ...r, segs, examples: [...examples], statuses: new Map(r.statuses), clients: new Set(r.clients), hosts: new Set(r.hosts) });
245
+ continue;
246
+ }
247
+ into.calls += r.calls;
248
+ into.variants += r.variants;
249
+ for (const x of examples) if (into.examples.length < 3) into.examples.push(x);
250
+ for (const [st, c] of r.statuses) into.statuses.set(st, (into.statuses.get(st) || 0) + c);
251
+ for (const c of r.clients) if (into.clients.size < MAX_CLIENTS) into.clients.add(c);
252
+ for (const h of r.hosts) into.hosts.add(h);
253
+ into.last = latest(into.last, r.last);
254
+ into.lastOk = latest(into.lastOk, r.lastOk) ?? into.lastOk ?? r.lastOk;
255
+ }
256
+ routes = [...merged.values()];
257
+ }
258
+ return routes.map((r) => ({ ...r, template: unique(r.segs).join('/') }));
259
+ }
260
+
261
+ /** {id}/{id} → {id}/{id2}: a template can't repeat a parameter name. */
262
+ function unique(segs) {
263
+ const seen = new Map();
264
+ return segs.map((s) => {
265
+ const m = /^\{(.+)\}$/.exec(s);
266
+ if (!m) return s;
267
+ const base = m[1].replace(/\d+$/, '');
268
+ const n = (seen.get(base) || 0) + 1;
269
+ seen.set(base, n);
270
+ return n === 1 ? `{${base}}` : `{${base}${n}}`;
271
+ });
272
+ }
273
+
274
+ function paramName(prev) {
275
+ const m = /^([a-z][a-z0-9]*?)(e?s)?$/i.exec(prev || '');
276
+ return m && m[1].length > 1 && !prev.startsWith('{') ? `${m[1]}Id` : 'id';
277
+ }
278
+
279
+ const median = (xs) => (xs.length ? [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)] : null);
280
+ // Buckets about 3% wide: a catch-all page that echoes the path varies by a few bytes.
281
+ const bucket = (bytes) => Math.round(Math.log(bytes + 64) / Math.log(1.03));
282
+
283
+ function record(map, key, base, rec, host) {
284
+ if (!map.has(key)) map.set(key, { ...base, calls: 0, statuses: new Map(), clients: new Set(), hosts: new Set(), last: null, lastOk: null });
285
+ const e = map.get(key);
286
+ e.calls++;
287
+ const s = rec.status || null;
288
+ e.statuses.set(s, (e.statuses.get(s) || 0) + 1);
289
+ if (rec.userAgent && e.clients.size < MAX_CLIENTS) e.clients.add(rec.userAgent);
290
+ if (host) e.hosts.add(host);
291
+ e.last = latest(e.last, rec.time);
292
+ if (ok(s)) e.lastOk = latest(e.lastOk, rec.time) ?? e.lastOk ?? true;
293
+ }
294
+
295
+ // 304 means the client's copy of an existing resource is current. Other 3xx
296
+ // prove nothing: an HTTP→HTTPS redirect answers every path, real or not.
297
+ const ok = (s) => (s >= 200 && s < 300) || s === 304;
298
+ const latest = (a, t) => (typeof t === 'number' && (typeof a !== 'number' || t > a) ? t : a);
299
+ const times = (n) => (n === 1 ? 'once' : `${n} times`);
300
+
301
+ function count(e, test) {
302
+ let n = 0;
303
+ for (const [s, c] of e.statuses) if (test(s)) n += c;
304
+ return n;
305
+ }
306
+
307
+ function judge(e, catchAll) {
308
+ const served = count(e, ok);
309
+ const exists = count(e, (s) => s === 401 || s === 403 || s === 400 || s === 422 || s === 429 || (s >= 500 && s < 600));
310
+ const unknown = count(e, (s) => s === null);
311
+ if (!served && !exists && !unknown) return null; // only 404/405: it isn't there
312
+ const write = WRITE.has(e.method);
313
+ const doubtful = [...e.hosts].some((h) => catchAll.has(h));
314
+ const conventional = Boolean(e.fixed);
315
+ let severity = 'low';
316
+ if (served && !doubtful && !conventional) {
317
+ if (e.kind === 'shadow-endpoint') severity = write || PRIVILEGED_PATH.test(e.template) ? 'high' : 'medium';
318
+ else severity = write ? 'high' : 'medium';
319
+ }
320
+ const notes = [describe(e)];
321
+ if (e.variants > 1) notes.push(`${e.variants} routes folded into this one, e.g. ${e.examples.join(', ')}`);
322
+ if (e.current) notes.push(`now ${e.current}`);
323
+ if (conventional) notes.push('a conventional health or discovery endpoint');
324
+ if (!served && unknown === e.calls) notes.push('the logs have no status, so it may not exist');
325
+ else if (!served) notes.push('never answered 2xx, but it exists');
326
+ if (doubtful) notes.push('this host looks like a catch-all');
327
+ return finding(e.kind, severity, { endpoint: `${e.method} ${e.template}`, detail: notes.join('; '), source: 'traffic' });
328
+ }
329
+
330
+ function describe(e) {
331
+ const statuses = [...e.statuses].sort((a, b) => b[1] - a[1]).slice(0, 4).map(([s, n]) => `${s ?? '?'}×${n}`).join(' ');
332
+ const last = typeof e.last === 'number' ? `, last ${new Date(e.last).toISOString().slice(0, 10)}` : '';
333
+ return `${e.calls} ${e.calls === 1 ? 'call' : 'calls'} from ${e.clients.size || '?'} ${e.clients.size === 1 ? 'client' : 'clients'} (${statuses})${last}`;
334
+ }
335
+
336
+ /** Operations whose x-sunset date has passed and that still answered 2xx after it. */
337
+ function sunsetPassed(spec, documented, now) {
338
+ const out = [];
339
+ if (!spec) return out;
340
+ for (const [path, item] of Object.entries(spec.paths || {})) {
341
+ if (!item || typeof item !== 'object') continue;
342
+ for (const [method, op] of Object.entries(item)) {
343
+ if (!op || typeof op !== 'object' || !op['x-sunset']) continue;
344
+ const v = op['x-sunset'];
345
+ const sunset = (v instanceof Date ? v : new Date(String(v))).getTime();
346
+ if (Number.isNaN(sunset) || sunset > now) continue;
347
+ const e = documented.get(`${method.toUpperCase()} ${path}`);
348
+ if (!e || !e.lastOk || (typeof e.lastOk === 'number' && e.lastOk < sunset)) continue;
349
+ out.push(finding('sunset-passed', 'medium', {
350
+ endpoint: `${method.toUpperCase()} ${path}`,
351
+ detail: `sunset ${new Date(sunset).toISOString().slice(0, 10)}; ${describe(e)}`,
352
+ source: 'traffic',
353
+ }));
354
+ }
355
+ }
356
+ return out;
357
+ }
358
+
359
+ function knownVersions(spec, matcher) {
360
+ const found = new Set();
361
+ const scan = (p) => p.split('/').forEach((s) => { if (VERSION.test(s)) found.add(s.toLowerCase()); });
362
+ for (const p of Object.keys(spec.paths || {})) scan(p);
363
+ matcher.basePaths.forEach(scan);
364
+ return [...found];
365
+ }
366
+
367
+ module.exports = { createInventory, PROBES };
@@ -48,6 +48,12 @@ const RULES = {
48
48
  'cors-any-origin': ['API8', 'CORS allows any origin', 'Confirm this API holds nothing private; otherwise allow specific origins.'],
49
49
  'insecure-cookie': ['API8', 'Session cookie missing protections', 'Set Secure, HttpOnly and SameSite.'],
50
50
  'version-disclosure': ['API8', 'Server software version disclosed', 'Remove version details from response headers.'],
51
+ // Traffic vs spec
52
+ 'shadow-endpoint': ['API9', 'Undocumented endpoint receiving traffic', 'Document it and give it the same review as the rest, or remove it.'],
53
+ 'undocumented-method': ['API5', 'Undocumented method on a documented path', 'Answer 405 for it, or document and secure it.'],
54
+ 'zombie-version': ['API9', 'Retired API version still served', 'Turn the old version off, or keep patching it like the current one.'],
55
+ 'sunset-passed': ['API9', 'Still served after its sunset date', 'Turn it off as announced, or move x-sunset in the spec.'],
56
+ 'exposed-sensitive-path': ['API8', 'Sensitive file or debug endpoint served', 'Block it at the server or proxy, and rotate anything it exposed.'],
51
57
  'stack-trace': ['API8', 'Stack trace in an error response', 'Return a generic error and log the detail on the server.'],
52
58
  };
53
59
 
@@ -15,6 +15,7 @@
15
15
  * JSON with a host field, absolute request URLs), else null
16
16
  * query raw query string without the leading '?' ('' when absent)
17
17
  * status numeric status code, or null
18
+ * bytes response body size (combined format only), or null
18
19
  * time epoch ms, or null when the line carries no parseable timestamp
19
20
  * userAgent raw User-Agent, or null
20
21
  * extras quoted fields after the User-Agent (combined format only) —
@@ -34,7 +35,7 @@ const HTTP_METHODS = new Set(['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'HEAD', '
34
35
  // ─── combined / common ──────────────────────────────────────────────────────
35
36
 
36
37
  // host ident authuser [time] "request" status bytes ["referer" "user-agent"] [extras...]
37
- const COMBINED_RE = /^(\S+) \S+ \S+ \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}|-) \S+(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(.*)$/;
38
+ const COMBINED_RE = /^(\S+) \S+ \S+ \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}|-) (\S+)(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(.*)$/;
38
39
 
39
40
  function parseCombined(line) {
40
41
  const m = COMBINED_RE.exec(line);
@@ -44,9 +45,10 @@ function parseCombined(line) {
44
45
  return {
45
46
  ...req,
46
47
  status: m[4] === '-' ? null : Number(m[4]),
48
+ bytes: /^\d+$/.test(m[5]) ? Number(m[5]) : null,
47
49
  time: parseClfTime(m[2]),
48
- userAgent: m[6] !== undefined ? nullIfDash(unescape(m[6])) : null,
49
- extras: m[7] ? tokenize(m[7]).map(nullIfDash) : [],
50
+ userAgent: m[7] !== undefined ? nullIfDash(unescape(m[7])) : null,
51
+ extras: m[8] ? tokenize(m[8]).map(nullIfDash) : [],
50
52
  json: null,
51
53
  };
52
54
  }
@@ -221,12 +223,17 @@ function parseAnyTime(v) {
221
223
 
222
224
  const PARSERS = { combined: parseCombined, alb: parseAlb, json: parseJsonLine };
223
225
 
224
- /** Picks the format that parses the most of the sample lines. */
226
+ // A line can have a format's shape without being a request: OpenLiteSpeed logs
227
+ // TLS probes and dropped connections as `"" 200`. They still say which format it is.
228
+ const SHAPES = { combined: COMBINED_RE };
229
+
230
+ /** Picks the format that parses (or at least fits) the most of the sample lines. */
225
231
  function detectFormat(sampleLines) {
226
232
  let best = null;
227
233
  let bestHits = 0;
228
234
  for (const [name, parse] of Object.entries(PARSERS)) {
229
- const hits = sampleLines.filter((l) => parse(l)).length;
235
+ const shape = SHAPES[name];
236
+ const hits = sampleLines.filter((l) => parse(l) || (shape && shape.test(l))).length;
230
237
  if (hits > bestHits) { best = name; bestHits = hits; }
231
238
  }
232
239
  return best;
@@ -241,9 +248,9 @@ function splitRequestLine(s) {
241
248
  if (parts.length < 2) return null;
242
249
  const method = parts[0].toUpperCase();
243
250
  if (!HTTP_METHODS.has(method)) return null;
244
- const { path, query, host } = splitTarget(parts[1]);
251
+ const { path, query, host, trailingSlash } = splitTarget(parts[1]);
245
252
  if (!path) return null;
246
- return { method, path, query, host };
253
+ return { method, path, query, host, trailingSlash };
247
254
  }
248
255
 
249
256
  function splitTarget(target) {
@@ -262,8 +269,9 @@ function splitTarget(target) {
262
269
  let path = q >= 0 ? t.slice(0, q) : t;
263
270
  const query = q >= 0 ? t.slice(q + 1) : '';
264
271
  path = path.replace(/\/{2,}/g, '/');
265
- if (path.length > 1 && path.endsWith('/')) path = path.slice(0, -1);
266
- return { path, query, host };
272
+ const trailingSlash = path.length > 1 && path.endsWith('/');
273
+ if (trailingSlash) path = path.slice(0, -1);
274
+ return { path, query, host, trailingSlash };
267
275
  }
268
276
 
269
277
  /** "API.Example.com:443" → "api.example.com"; "[::1]:80" → "[::1]". */