specshield 3.5.3 → 3.5.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -1
- package/README.md +21 -0
- package/package.json +1 -1
- package/src/commands/audit.js +44 -6
- package/src/core/security/auditHar.js +20 -1
- package/src/core/security/inventory.js +367 -0
- package/src/core/security/rules.js +6 -0
- package/src/core/usage/logParsers.js +17 -9
package/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,13 @@
|
|
|
1
1
|
# SpecShield CLI changelog
|
|
2
2
|
|
|
3
|
-
## Unreleased —
|
|
3
|
+
## Unreleased — shadow APIs
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- `specshield audit --logs <files>` compares real traffic with the spec (OWASP API9). It reports undocumented endpoints and methods that answer requests, retired versions still served, operations past their `x-sunset`, and exposed files such as `/.env`. Severity follows the response status; redirects count as no evidence. Catch-all pages are recognised by response size (the home page's, or one shared by other probes). Routes are folded by structure, and trailing `{path}` parameters span segments. A HAR passed with `--har` gets the same checks. Validated on six real public log datasets, 9.9M requests in all.
|
|
8
|
+
- Log import records the response size (combined format) and recognises combined logs whose first lines have an empty request (OpenLiteSpeed).
|
|
9
|
+
|
|
10
|
+
## 3.5.3 — 2026-11-10 — security audit
|
|
4
11
|
|
|
5
12
|
### Added
|
|
6
13
|
|
package/README.md
CHANGED
|
@@ -604,11 +604,32 @@ Audits a whole spec and, optionally, captured traffic, by [OWASP API Security To
|
|
|
604
604
|
```bash
|
|
605
605
|
specshield audit openapi.yaml
|
|
606
606
|
specshield audit openapi.yaml --har staging.har --host api.acme.com
|
|
607
|
+
specshield audit openapi.yaml --logs /var/log/nginx/access.log*.gz
|
|
607
608
|
specshield audit openapi.yaml --fail-on medium --sarif audit.sarif
|
|
608
609
|
```
|
|
609
610
|
|
|
610
611
|
**From the spec:** missing or undefined authentication, public writes and admin endpoints, API keys and secrets in URLs, basic auth, deprecated OAuth flows, OAuth over HTTP, credentials in responses, personal data on public endpoints, privileged request fields, page sizes with no maximum, client-supplied URLs, plain-HTTP and non-production servers.
|
|
611
612
|
|
|
613
|
+
**From access logs (`--logs`) or a HAR file, compared with the spec (OWASP API9):**
|
|
614
|
+
|
|
615
|
+
| Finding | What it means |
|
|
616
|
+
|---|---|
|
|
617
|
+
| `shadow-endpoint` | A path the spec doesn't have is answering requests |
|
|
618
|
+
| `undocumented-method` | A documented path answers a method the spec doesn't list, e.g. `DELETE` |
|
|
619
|
+
| `zombie-version` | `/v1/...` still answers where the spec documents `/v2/...` |
|
|
620
|
+
| `sunset-passed` | An operation past its `x-sunset` date still answers |
|
|
621
|
+
| `exposed-sensitive-path` | `/.env`, `/.git/config`, `/actuator/env` and similar answered 200 (checked even without a spec) |
|
|
622
|
+
|
|
623
|
+
How sure a finding is comes from the response, not the request:
|
|
624
|
+
|
|
625
|
+
- **Status.** 2xx (or 304) means the endpoint served the request: high for writes and internal paths, medium otherwise. 401/403/400 means it exists but refused (low). 404/405 means it isn't there, and redirects prove nothing (an HTTP→HTTPS redirect answers every path), so neither is reported.
|
|
626
|
+
- **Catch-all pages.** Many sites answer every unknown path with the same page. When the logs record response sizes, a probe answered with the site's home page size, or the same size as other probes got, is downgraded; a size no other probe got stays high. Without sizes, a site that answered 200 to three or more probes is treated as a catch-all.
|
|
627
|
+
- **Routes.** IDs are templated, and object names that don't look like IDs are folded once four or more sit in the same place with the same shape after them (`/things/noxus/x`, `/things/fooab/x` → `/things/{thingId}/x`). Health and discovery endpoints (`/readyz`, `/openapi/v3`…) are reported low. A trailing `{path}` parameter may span several segments, as in Kubernetes' `/proxy/{path}`.
|
|
628
|
+
|
|
629
|
+
Logs in the same formats as `specshield usage import` are supported (combined, ALB, JSON lines, Kubernetes audit events, `.gz`). Per-site logs without a host are grouped by file name, so one site's catch-all doesn't affect another's findings. A file that can't be read is skipped with a warning. `--base-path` and `--host` narrow the logs to this API.
|
|
630
|
+
|
|
631
|
+
Validated on real, published traffic: a Kubernetes CI run (250,724 API requests against the official spec, every reported route independently confirmed absent from it), NASA's 1995 web logs (1.9M lines), Elastic's Apache sample, 90 days of secrepo.com logs, and a 2025 forensic dataset of 7.4M requests from a compromised hosting server, where every probe answered 200 was correctly identified as a catch-all page.
|
|
632
|
+
|
|
612
633
|
**From a HAR file (`--har`):** secrets in URLs and responses, regulated and personal data, plain HTTP, missing HSTS and `nosniff`, risky CORS, unprotected session cookies, version banners, stack traces. Findings say where a secret was seen and never print its value. Only API responses are checked, so use `--host` to leave out third parties in a browser capture.
|
|
613
634
|
|
|
614
635
|
| Option | Default | |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "specshield",
|
|
3
|
-
"version": "3.5.
|
|
3
|
+
"version": "3.5.4",
|
|
4
4
|
"description": "Contract compatibility testing for APIs — catch breaking OpenAPI changes before they reach your consumers, with can-i-deploy deploy gating and GitHub PR checks. (a.k.a. bidirectional contract testing.)",
|
|
5
5
|
"main": "src/cli.js",
|
|
6
6
|
"homepage": "https://specshield.io",
|
package/src/commands/audit.js
CHANGED
|
@@ -8,6 +8,8 @@ const { loadSpec } = require('../core/loadSpec');
|
|
|
8
8
|
const { parseSpec } = require('../core/parseSpec');
|
|
9
9
|
const { auditSpec, consolidate } = require('../core/security/audit');
|
|
10
10
|
const { auditHar } = require('../core/security/auditHar');
|
|
11
|
+
const { createInventory } = require('../core/security/inventory');
|
|
12
|
+
const { importLogFile } = require('../core/usage/importLogs');
|
|
11
13
|
const { OWASP, RULES, SEVERITIES } = require('../core/security/rules');
|
|
12
14
|
const W = require('../core/gate/waivers');
|
|
13
15
|
const { plural } = require('../core/usage/format');
|
|
@@ -21,7 +23,10 @@ const audit = new Command('audit')
|
|
|
21
23
|
.description('Security audit of an OpenAPI spec and captured traffic, by OWASP API Security Top 10 category. Sends no requests')
|
|
22
24
|
.argument('[spec]', 'The OpenAPI spec to audit')
|
|
23
25
|
.option('--har <file...>', 'HAR capture(s) to check for leaked secrets, personal data and missing protections')
|
|
24
|
-
.option('--
|
|
26
|
+
.option('--logs <file...>', 'Access logs (.gz ok): find undocumented endpoints, methods and old versions still answering, and exposed files like /.env')
|
|
27
|
+
.option('--format <format>', 'Access log format: auto | combined | alb | json', 'auto')
|
|
28
|
+
.option('--base-path <path>', 'Path prefix the API is served under, e.g. /api (default: from the spec servers)')
|
|
29
|
+
.option('--host <host...>', 'Only check these hosts in the capture and logs')
|
|
25
30
|
.option('--fail-on <severity>', 'Exit 1 at or above this severity: high, medium, low, or none', 'high')
|
|
26
31
|
.option('--waivers <file>', `Waiver file (default: ${W.DEFAULT_FILE})`, W.DEFAULT_FILE)
|
|
27
32
|
.option('--all', 'List every low-severity finding')
|
|
@@ -30,17 +35,29 @@ const audit = new Command('audit')
|
|
|
30
35
|
.option('--json', 'Output machine-readable JSON')
|
|
31
36
|
.action(async (specPath, opts) => {
|
|
32
37
|
try {
|
|
33
|
-
if (!specPath && !opts.har) throw new Error('Nothing to audit: pass a spec, --har <file>, or
|
|
38
|
+
if (!specPath && !opts.har && !opts.logs) throw new Error('Nothing to audit: pass a spec, --har <file>, --logs <file>, or a combination');
|
|
34
39
|
if (!FAIL_ON.includes(opts.failOn)) throw new Error(`--fail-on must be one of ${FAIL_ON.join(', ')}`);
|
|
35
40
|
|
|
36
41
|
let specText = null;
|
|
37
42
|
let doc = null;
|
|
38
43
|
if (specPath) { specText = await loadSpec(specPath); doc = parseSpec(specText, specPath); }
|
|
39
|
-
const
|
|
44
|
+
const hosts = opts.host ? new Set(opts.host.map((h) => h.toLowerCase())) : null;
|
|
45
|
+
const traffic = opts.har || opts.logs ? createInventory(doc, { basePath: opts.basePath, hosts }) : null;
|
|
46
|
+
const har = opts.har ? auditHar(opts.har, { hosts, inventory: traffic }) : null;
|
|
47
|
+
const unread = [];
|
|
48
|
+
for (const file of opts.logs || []) {
|
|
49
|
+
const site = siteOf(file);
|
|
50
|
+
try {
|
|
51
|
+
await importLogFile(file, { format: opts.format, aggregator: { add: (rec) => { rec.site = site; traffic.add(rec); } } });
|
|
52
|
+
} catch (err) {
|
|
53
|
+
unread.push({ file, reason: err.message.split('\n')[0] });
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
if (opts.logs && unread.length === opts.logs.length) throw new Error(`No log could be read: ${unread[0].reason}`);
|
|
40
57
|
|
|
41
58
|
const now = Date.now();
|
|
42
59
|
const { waivers } = await W.loadWaivers(path.resolve(opts.waivers), { maxDays: W.DEFAULT_MAX_DAYS, now, file: opts.waivers });
|
|
43
|
-
const all = [...(doc ? auditSpec(doc) : []), ...(har ? har.findings : [])];
|
|
60
|
+
const all = [...(doc ? auditSpec(doc) : []), ...(har ? har.findings : []), ...(traffic ? traffic.findings() : [])];
|
|
44
61
|
const { blocking, waived } = W.applyWaivers(all, waivers, now);
|
|
45
62
|
const findings = sort(consolidate(blocking));
|
|
46
63
|
const failing = findings.filter((f) => atLeast(f.severity, opts.failOn));
|
|
@@ -48,6 +65,7 @@ const audit = new Command('audit')
|
|
|
48
65
|
spec: specPath || null,
|
|
49
66
|
title: doc && doc.info ? doc.info.title || null : null,
|
|
50
67
|
har: har ? { files: opts.har, ...har.stats } : null,
|
|
68
|
+
traffic: traffic ? { logs: opts.logs || [], unread, ...traffic.stats } : null,
|
|
51
69
|
failOn: opts.failOn,
|
|
52
70
|
counts: count(findings),
|
|
53
71
|
owasp: byOwasp(findings),
|
|
@@ -68,6 +86,15 @@ const audit = new Command('audit')
|
|
|
68
86
|
}
|
|
69
87
|
});
|
|
70
88
|
|
|
89
|
+
/** "logs/shop.example.com.access_log.2025_04_26.gz" → "shop.example.com": which site a per-site log belongs to. */
|
|
90
|
+
function siteOf(file) {
|
|
91
|
+
const name = path.basename(file)
|
|
92
|
+
.replace(/\.gz$/i, '')
|
|
93
|
+
.replace(/([._-]\d{4}[._-]\d{2}[._-]\d{2}([._]\d+)?|(\.\d+)+)$/, '')
|
|
94
|
+
.replace(/[._-]?(access[._-]?log|access|log)$/i, '');
|
|
95
|
+
return name && !/^(web|ssl|other_vhosts)?[._-]?$/i.test(name) ? name : path.basename(path.dirname(path.resolve(file)));
|
|
96
|
+
}
|
|
97
|
+
|
|
71
98
|
const rank = (s) => SEVERITIES.indexOf(s);
|
|
72
99
|
const atLeast = (s, min) => min !== 'none' && rank(s) <= rank(min);
|
|
73
100
|
const sort = (list) => [...list].sort((a, b) => rank(a.severity) - rank(b.severity) || a.rule.localeCompare(b.rule));
|
|
@@ -102,6 +129,17 @@ const MAX_LISTED = 5;
|
|
|
102
129
|
function render(r, opts) {
|
|
103
130
|
const lines = ['', chalk.bold(` SpecShield™ Audit — ${r.title || r.spec || 'captured traffic'}`), chalk.gray(' ─────────────────────────────────────────')];
|
|
104
131
|
if (r.har) lines.push(chalk.gray(` Capture: ${plural(r.har.api, 'API response')} checked of ${plural(r.har.entries, 'request')}, ${plural(r.har.hosts, 'host')}`));
|
|
132
|
+
if (r.traffic && r.traffic.unread.length) {
|
|
133
|
+
lines.push(chalk.yellow(` ${plural(r.traffic.unread.length, 'log file')} skipped: ${r.traffic.unread.slice(0, 2).map((u) => u.reason).join('; ')}${r.traffic.unread.length > 2 ? '; …' : ''}`));
|
|
134
|
+
}
|
|
135
|
+
if (r.traffic && r.spec) {
|
|
136
|
+
const t = r.traffic;
|
|
137
|
+
lines.push(chalk.gray(` Traffic: ${plural(t.requests, 'request')}; ${t.documentedSeen} of ${t.documentedTotal} documented operations called, ${plural(t.endpoints, 'undocumented route')} seen`));
|
|
138
|
+
if (t.requests && t.considered === 0) lines.push(chalk.yellow(' No request matched this API. Check --host and --base-path.'));
|
|
139
|
+
if (t.overflow) lines.push(chalk.yellow(` ${plural(t.overflow, 'request')} not checked: too many distinct undocumented routes.`));
|
|
140
|
+
} else if (r.traffic) {
|
|
141
|
+
lines.push(chalk.gray(' Traffic: checked for exposed files only. Pass the spec to find undocumented endpoints.'));
|
|
142
|
+
}
|
|
105
143
|
if (r.har && r.har.hosts > 1) lines.push(chalk.gray(' Several hosts in this capture: use --host to check only your own.'));
|
|
106
144
|
lines.push('');
|
|
107
145
|
lines.push(` ${chalk.red(`${r.counts.high} high`)} · ${chalk.yellow(`${r.counts.medium} medium`)} · ${chalk.gray(`${r.counts.low} low`)}${r.waived.length ? chalk.gray(` · ${r.waived.length} waived`) : ''}`);
|
|
@@ -167,7 +205,7 @@ function sarif(r, specText, harFiles) {
|
|
|
167
205
|
level: LEVEL[f.severity],
|
|
168
206
|
message: { text: `${f.title}${where(f) ? `: ${where(f)}` : ''}` },
|
|
169
207
|
locations: [{ physicalLocation: {
|
|
170
|
-
artifactLocation: { uri: f.source === 'spec' ? r.spec : harFiles[0] },
|
|
208
|
+
artifactLocation: { uri: f.source === 'spec' || !harFiles ? r.spec || 'traffic' : harFiles[0] },
|
|
171
209
|
region: { startLine: f.source === 'spec' ? lineOf(specText, (f.affectedEndpoints[0] || '').split(' ')[1] || f.field) : 1 },
|
|
172
210
|
} }],
|
|
173
211
|
})),
|
|
@@ -176,4 +214,4 @@ function sarif(r, specText, harFiles) {
|
|
|
176
214
|
}
|
|
177
215
|
|
|
178
216
|
module.exports = audit;
|
|
179
|
-
module.exports._internal = { sarif, render };
|
|
217
|
+
module.exports._internal = { sarif, render, siteOf };
|
|
@@ -28,7 +28,7 @@ const SECRET_VALUES = [
|
|
|
28
28
|
];
|
|
29
29
|
const STACK = /Traceback \(most recent call last\)|Exception in thread "|\n\s+at [\w.$<>]+\([\w.]+\.(?:java|kt|scala):\d+\)|\n\s+at .+ \(.+\.[jt]s:\d+:\d+\)|\.php on line \d+/;
|
|
30
30
|
|
|
31
|
-
function auditHar(files, { hosts = null } = {}) {
|
|
31
|
+
function auditHar(files, { hosts = null, inventory = null } = {}) {
|
|
32
32
|
const found = new Map();
|
|
33
33
|
const stats = { entries: 0, api: 0, otherHosts: 0, hosts: new Set() };
|
|
34
34
|
const add = (rule, severity, opts) => {
|
|
@@ -38,6 +38,7 @@ function auditHar(files, { hosts = null } = {}) {
|
|
|
38
38
|
for (const file of files) {
|
|
39
39
|
for (const e of entriesOf(file)) {
|
|
40
40
|
stats.entries++;
|
|
41
|
+
if (inventory) inventory.add(recordOf(e));
|
|
41
42
|
checkEntry(e, { add, hosts, stats });
|
|
42
43
|
}
|
|
43
44
|
}
|
|
@@ -51,6 +52,24 @@ function entriesOf(file) {
|
|
|
51
52
|
return doc.log.entries.filter((e) => e && e.request && e.request.url && e.response);
|
|
52
53
|
}
|
|
53
54
|
|
|
55
|
+
/** The same shape the access-log parsers produce, plus the response type. */
|
|
56
|
+
function recordOf(e) {
|
|
57
|
+
let url;
|
|
58
|
+
try { url = new URL(e.request.url); } catch { return null; }
|
|
59
|
+
const time = Date.parse(e.startedDateTime);
|
|
60
|
+
return {
|
|
61
|
+
method: String(e.request.method || 'GET').toUpperCase(),
|
|
62
|
+
path: url.pathname,
|
|
63
|
+
query: url.search.replace(/^\?/, ''),
|
|
64
|
+
host: url.hostname.toLowerCase(),
|
|
65
|
+
status: e.response.status > 0 ? e.response.status : null,
|
|
66
|
+
time: Number.isNaN(time) ? null : time,
|
|
67
|
+
userAgent: ((e.request.headers || []).find((h) => h && /^user-agent$/i.test(h.name)) || {}).value || null,
|
|
68
|
+
mime: (e.response.content && e.response.content.mimeType) || null,
|
|
69
|
+
bytes: e.response.content && e.response.content.size >= 0 ? e.response.content.size : null,
|
|
70
|
+
};
|
|
71
|
+
}
|
|
72
|
+
|
|
54
73
|
function headerMap(list) {
|
|
55
74
|
const map = new Map();
|
|
56
75
|
for (const h of list || []) {
|
|
@@ -0,0 +1,367 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Improper inventory (OWASP API9) from real traffic: what the API serves that
|
|
5
|
+
* the spec doesn't say.
|
|
6
|
+
*
|
|
7
|
+
* shadow-endpoint a path the spec doesn't have, answering requests
|
|
8
|
+
* undocumented-method a documented path, answering a method the spec doesn't list
|
|
9
|
+
* zombie-version /v1/... still answering, where the spec documents /v2/...
|
|
10
|
+
* sunset-passed an operation past its x-sunset date, still answering
|
|
11
|
+
* exposed-sensitive-path /.env, /.git/config, /actuator/env... answering 200
|
|
12
|
+
*
|
|
13
|
+
* The response status decides how sure a finding is: a 2xx (or 304) means the
|
|
14
|
+
* endpoint served the request; 401/403/400 mean it exists; 404/405 and
|
|
15
|
+
* redirects say nothing.
|
|
16
|
+
* A host that answers 200 to scanner probes is most likely a catch-all (a
|
|
17
|
+
* single-page app's index.html), so its findings are downgraded, not trusted.
|
|
18
|
+
*/
|
|
19
|
+
|
|
20
|
+
const { createRouteMatcher } = require('../usage/routeMatcher');
|
|
21
|
+
const { templatePath } = require('../har/pathTemplate');
|
|
22
|
+
const { finding } = require('./rules');
|
|
23
|
+
|
|
24
|
+
const MAX_ENTRIES = 2000;
|
|
25
|
+
const MAX_CLIENTS = 100;
|
|
26
|
+
const SKIP_METHODS = new Set(['HEAD', 'OPTIONS']);
|
|
27
|
+
const WRITE = new Set(['POST', 'PUT', 'PATCH', 'DELETE']);
|
|
28
|
+
const STATIC = /\.(m?js|css|map|png|jpe?g|gif|svg|ico|webp|avif|woff2?|ttf|eot|html?|txt|webmanifest)$/i;
|
|
29
|
+
const PRIVILEGED_PATH = /(^|\/)(admin|internal|debug|actuator|management|private|_[a-z]+)(\/|$)/i;
|
|
30
|
+
const VERSION = /^v\d+(\.\d+)?$/i;
|
|
31
|
+
// Every API has these, and few specs list them: worth knowing about, not alarming.
|
|
32
|
+
const CONVENTIONAL = /^\/(health|healthz|livez|readyz|ping|status|version|openapi|swagger|api-docs|v\d+\/api-docs|\.well-known)(\/|$|\.json$|\.yaml$)/i;
|
|
33
|
+
// A position with this many distinct values is a parameter, whatever the values look like.
|
|
34
|
+
const DISTINCT = 4;
|
|
35
|
+
|
|
36
|
+
// Paths no production server should answer with content.
|
|
37
|
+
const PROBES = [
|
|
38
|
+
['environment file', /^\/(.*\/)?\.env(\.[\w-]+)?$/i],
|
|
39
|
+
['git repository', /^\/(.*\/)?\.git\/(config|HEAD|index)$/i],
|
|
40
|
+
['cloud credentials', /^\/(.*\/)?\.aws\/credentials$/i],
|
|
41
|
+
['Spring Boot actuator', /^\/(.*\/)?actuator\/(env|heapdump|threaddump|configprops|beans|mappings|loggers|jolokia|shutdown)$/i],
|
|
42
|
+
['heap dump', /^\/(.*\/)?heapdump$/i],
|
|
43
|
+
['Go debug endpoint', /^\/debug\/(pprof|vars)(\/.*)?$/i],
|
|
44
|
+
['server status page', /^\/server-(status|info)$/i],
|
|
45
|
+
['PHP info page', /^\/(.*\/)?phpinfo\.php$/i],
|
|
46
|
+
['database backup', /^\/(.*\/)?[\w-]*(backup|dump|db)[\w-]*\.(sql|sql\.gz|bak)$/i],
|
|
47
|
+
];
|
|
48
|
+
// This many probe paths answered with the same page means the host answers everything with it.
|
|
49
|
+
const CATCH_ALL = 3;
|
|
50
|
+
const MAX_SIZES = 50;
|
|
51
|
+
|
|
52
|
+
function createInventory(spec, { basePath, hosts = null, now = Date.now() } = {}) {
|
|
53
|
+
const matcher = spec ? createRouteMatcher(spec, { basePath }) : null;
|
|
54
|
+
const versions = spec ? knownVersions(spec, matcher) : [];
|
|
55
|
+
const entries = new Map();
|
|
56
|
+
const probes = new Map();
|
|
57
|
+
const probeHits = new Map(); // site → probe path → probe answered 2xx
|
|
58
|
+
const home = new Map(); // site → size buckets of its home page
|
|
59
|
+
const documented = new Map();
|
|
60
|
+
const stats = { requests: 0, considered: 0, outside: 0, static: 0, otherHosts: 0, overflow: 0 };
|
|
61
|
+
|
|
62
|
+
function add(rec) {
|
|
63
|
+
if (!rec || !rec.method || typeof rec.path !== 'string') return;
|
|
64
|
+
stats.requests++;
|
|
65
|
+
const method = String(rec.method).toUpperCase();
|
|
66
|
+
const host = rec.host ? String(rec.host).toLowerCase() : null;
|
|
67
|
+
if (hosts && host && !hosts.has(host)) { stats.otherHosts++; return; }
|
|
68
|
+
// Per-site logs record no host; the file they came from stands in, so one
|
|
69
|
+
// site's catch-all page doesn't cast doubt on another's findings.
|
|
70
|
+
const site = host || (rec.site ? String(rec.site) : null);
|
|
71
|
+
if (SKIP_METHODS.has(method)) return;
|
|
72
|
+
let path = rec.path.replace(/\/{2,}/g, '/');
|
|
73
|
+
if (path.length > 1) path = path.replace(/\/$/, '');
|
|
74
|
+
// A capture tells us what came back; an HTML page is the app, not the API.
|
|
75
|
+
const html = rec.mime ? /html/i.test(rec.mime) : false;
|
|
76
|
+
|
|
77
|
+
// Single-page apps and CMS rewrites answer unknown paths with the home page.
|
|
78
|
+
if (path === '/' && method === 'GET' && ok(rec.status) && typeof rec.bytes === 'number') {
|
|
79
|
+
if (!home.has(site)) home.set(site, new Set());
|
|
80
|
+
const sizes = home.get(site);
|
|
81
|
+
const b = bucket(rec.bytes);
|
|
82
|
+
if (sizes.size < MAX_SIZES) [b - 1, b, b + 1].forEach((x) => sizes.add(x));
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
const probe = PROBES.find(([, re]) => re.test(path));
|
|
86
|
+
if (probe) {
|
|
87
|
+
// "/.env/" is a directory request: whatever answered it, it wasn't the file.
|
|
88
|
+
const asDirectory = (rec.trailingSlash || (rec.path.length > 1 && rec.path.endsWith('/'))) && !/^\/debug\//.test(path);
|
|
89
|
+
if (ok(rec.status) && !html && !asDirectory) {
|
|
90
|
+
const key = `${site || ''} ${path}`;
|
|
91
|
+
if (!probes.has(key)) probes.set(key, { label: probe[0], path, host: site, calls: 0, last: null, sizes: [] });
|
|
92
|
+
const p = probes.get(key);
|
|
93
|
+
p.calls++;
|
|
94
|
+
p.last = latest(p.last, rec.time);
|
|
95
|
+
if (typeof rec.bytes === 'number' && p.sizes.length < MAX_SIZES) p.sizes.push(rec.bytes);
|
|
96
|
+
if (!probeHits.has(site)) probeHits.set(site, new Map());
|
|
97
|
+
probeHits.get(site).set(path, p);
|
|
98
|
+
}
|
|
99
|
+
return;
|
|
100
|
+
}
|
|
101
|
+
if (!matcher) return;
|
|
102
|
+
if (path === '/' || STATIC.test(path)) { stats.static++; return; }
|
|
103
|
+
if (html) return;
|
|
104
|
+
stats.considered++;
|
|
105
|
+
|
|
106
|
+
const m = matcher.match(method, path) || greedy(method, path);
|
|
107
|
+
if (m && m.operation) { record(documented, `${method} ${m.template}`, { method, template: m.template }, rec, site); return; }
|
|
108
|
+
if (m) { remember('undocumented-method', method, m.template, rec, site); return; }
|
|
109
|
+
const older = versions.length ? otherVersion(method, path) : null;
|
|
110
|
+
if (older) { remember('zombie-version', method, templatePath(path).templated, rec, site, older); return; }
|
|
111
|
+
// Named like the spec names its paths, so a waiver for it reads the same.
|
|
112
|
+
const base = matcher.basePaths.find((b) => path === b || path.startsWith(`${b}/`));
|
|
113
|
+
if (matcher.basePaths.length && !base) { stats.outside++; return; }
|
|
114
|
+
remember('shadow-endpoint', method, templatePath(base ? path.slice(base.length) || '/' : path).templated, rec, site);
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
/**
|
|
118
|
+
* Kubernetes and others document "/proxy/{path}" where {path} spans several
|
|
119
|
+
* segments. OpenAPI doesn't say a parameter can't hold a slash, so try the
|
|
120
|
+
* path with its tail folded into the last segment.
|
|
121
|
+
*/
|
|
122
|
+
function greedy(method, path) {
|
|
123
|
+
const segs = path.split('/');
|
|
124
|
+
for (let n = segs.length - 1; n > 1; n--) {
|
|
125
|
+
const m = matcher.match(method, [...segs.slice(0, n), segs.slice(n).join('%2F')].join('/'));
|
|
126
|
+
if (m && /\/\{[^}/]+\}$/.test(m.template)) return m;
|
|
127
|
+
}
|
|
128
|
+
return null;
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
/** /v1/orders when the spec serves /v2/orders: the same operation under a version the spec dropped. */
|
|
132
|
+
function otherVersion(method, path) {
|
|
133
|
+
const segs = path.split('/');
|
|
134
|
+
for (let i = 0; i < segs.length; i++) {
|
|
135
|
+
if (!VERSION.test(segs[i]) || versions.includes(segs[i].toLowerCase())) continue;
|
|
136
|
+
for (const v of versions) {
|
|
137
|
+
const swapped = [...segs.slice(0, i), v, ...segs.slice(i + 1)].join('/');
|
|
138
|
+
const m = matcher.match(method, swapped);
|
|
139
|
+
if (m && m.operation) return `${method} ${m.template} (${v})`;
|
|
140
|
+
}
|
|
141
|
+
}
|
|
142
|
+
return null;
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
function remember(kind, method, template, rec, host, current = null) {
|
|
146
|
+
const key = `${kind}|${method} ${template}`;
|
|
147
|
+
if (!entries.has(key) && entries.size >= MAX_ENTRIES) { stats.overflow++; return; }
|
|
148
|
+
record(entries, key, { kind, method, template, current, fixed: kind === 'shadow-endpoint' && CONVENTIONAL.test(template) }, rec, host);
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
/**
|
|
152
|
+
* Whether a probe's 2xx was the site's catch-all page rather than the file.
|
|
153
|
+
* With sizes: other probe paths got a response of the same size, so it isn't
|
|
154
|
+
* this file's content. Comparing per probe copes with a site that has several
|
|
155
|
+
* such pages, or whose page changed. Only probes count: an ordinary API
|
|
156
|
+
* response of a similar size says nothing. Without sizes: the site answered
|
|
157
|
+
* 2xx to several probes.
|
|
158
|
+
*/
|
|
159
|
+
function catchAllAnswer(p) {
|
|
160
|
+
const size = median(p.sizes);
|
|
161
|
+
const siblings = [...probeHits.get(p.host).values()].filter((q) => q !== p);
|
|
162
|
+
if (size === null) {
|
|
163
|
+
const n = siblings.length + 1;
|
|
164
|
+
return n >= CATCH_ALL ? `this host answers 200 to ${n} probe paths, so it is probably a catch-all page` : null;
|
|
165
|
+
}
|
|
166
|
+
const b = bucket(size);
|
|
167
|
+
if ((home.get(p.host) || new Set()).has(b)) return `${size} bytes, the size of this host's home page; probably a rewrite to it`;
|
|
168
|
+
const same = siblings.filter((q) => { const m = median(q.sizes); return m !== null && Math.abs(bucket(m) - b) <= 1; });
|
|
169
|
+
if (same.length + 1 < CATCH_ALL) return null;
|
|
170
|
+
return `${size} bytes, the same as ${same.length} other probe paths got (e.g. ${same[0].path}); probably a catch-all page`;
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
function findings() {
|
|
174
|
+
const verdicts = new Map([...probes.values()].map((p) => [p, catchAllAnswer(p)]));
|
|
175
|
+
const catchAll = new Set([...verdicts].filter(([, v]) => v).map(([p]) => p.host));
|
|
176
|
+
const out = [];
|
|
177
|
+
for (const e of collapse([...entries.values()])) {
|
|
178
|
+
const f = judge(e, catchAll);
|
|
179
|
+
if (f) out.push(f);
|
|
180
|
+
}
|
|
181
|
+
for (const op of sunsetPassed(spec, documented, now)) out.push(op);
|
|
182
|
+
for (const [p, why] of verdicts) {
|
|
183
|
+
const size = median(p.sizes);
|
|
184
|
+
const detail = why ? `${p.label}; ${why}` : `${p.label}, answered 2xx ${times(p.calls)}${size !== null ? ` with ${size} bytes no other path got` : ''}`;
|
|
185
|
+
out.push(finding('exposed-sensitive-path', why ? 'low' : 'high', { endpoint: `GET ${p.path}`, field: p.host, detail, source: 'traffic' }));
|
|
186
|
+
}
|
|
187
|
+
return out;
|
|
188
|
+
}
|
|
189
|
+
|
|
190
|
+
return {
|
|
191
|
+
add,
|
|
192
|
+
findings,
|
|
193
|
+
get stats() {
|
|
194
|
+
const real = [...entries.values()].filter((e) => count(e, (st) => st !== 404 && st !== 405) > 0).length;
|
|
195
|
+
return { ...stats, documentedSeen: documented.size, documentedTotal: matcher ? matcher.operations.length : 0, endpoints: real };
|
|
196
|
+
},
|
|
197
|
+
};
|
|
198
|
+
}
|
|
199
|
+
|
|
200
|
+
/**
|
|
201
|
+
* Folds routes that differ only where a value varies: /things/noxus/x and
|
|
202
|
+
* /things/fooab/x become /things/{thingId}/x once DISTINCT siblings share both
|
|
203
|
+
* a parent and the shape of what follows them. Object names that don't look
|
|
204
|
+
* like IDs would otherwise each be a finding, and fixed words (/readyz,
|
|
205
|
+
* /metrics, /openapi) differ in what follows, so they stay apart. The first
|
|
206
|
+
* segment is never folded.
|
|
207
|
+
*/
|
|
208
|
+
function collapse(list) {
|
|
209
|
+
let routes = list.map((e) => ({ ...e, segs: e.template.split('/'), variants: e.variants || 1, examples: e.examples || [] }));
|
|
210
|
+
const depth = Math.max(0, ...routes.map((r) => r.segs.length));
|
|
211
|
+
const isParam = (v) => v.startsWith('{');
|
|
212
|
+
for (let i = 2; i < depth; i++) {
|
|
213
|
+
const familyOf = (r) => [r.kind, r.method, r.segs.slice(0, i).join('/')].join('|');
|
|
214
|
+
const shapes = new Map(); // family → value → shapes of what follows
|
|
215
|
+
for (const r of routes) {
|
|
216
|
+
if (r.segs.length <= i || isParam(r.segs[i])) continue;
|
|
217
|
+
const fam = familyOf(r);
|
|
218
|
+
if (!shapes.has(fam)) shapes.set(fam, new Map());
|
|
219
|
+
const byValue = shapes.get(fam);
|
|
220
|
+
if (!byValue.has(r.segs[i])) byValue.set(r.segs[i], new Set());
|
|
221
|
+
byValue.get(r.segs[i]).add(r.segs.slice(i + 1).map((x) => (isParam(x) ? '{}' : '*')).join('/'));
|
|
222
|
+
}
|
|
223
|
+
const fold = new Set();
|
|
224
|
+
for (const [fam, byValue] of shapes) {
|
|
225
|
+
const bySignature = new Map();
|
|
226
|
+
for (const [value, set] of byValue) {
|
|
227
|
+
const sig = [...set].sort().join('|');
|
|
228
|
+
if (!bySignature.has(sig)) bySignature.set(sig, []);
|
|
229
|
+
bySignature.get(sig).push(value);
|
|
230
|
+
}
|
|
231
|
+
for (const values of bySignature.values()) {
|
|
232
|
+
if (values.length >= DISTINCT) values.forEach((v) => fold.add(`${fam}|${v}`));
|
|
233
|
+
}
|
|
234
|
+
}
|
|
235
|
+
if (!fold.size) continue;
|
|
236
|
+
const merged = new Map();
|
|
237
|
+
for (const r of routes) {
|
|
238
|
+
const folded = r.segs.length > i && fold.has(`${familyOf(r)}|${r.segs[i]}`);
|
|
239
|
+
const segs = folded ? [...r.segs.slice(0, i), `{${paramName(r.segs[i - 1])}}`, ...r.segs.slice(i + 1)] : r.segs;
|
|
240
|
+
const key = [r.kind, r.method, segs.join('/')].join('|');
|
|
241
|
+
const into = merged.get(key);
|
|
242
|
+
const examples = folded && !r.examples.length ? [r.template] : r.examples;
|
|
243
|
+
if (!into) {
|
|
244
|
+
merged.set(key, { ...r, segs, examples: [...examples], statuses: new Map(r.statuses), clients: new Set(r.clients), hosts: new Set(r.hosts) });
|
|
245
|
+
continue;
|
|
246
|
+
}
|
|
247
|
+
into.calls += r.calls;
|
|
248
|
+
into.variants += r.variants;
|
|
249
|
+
for (const x of examples) if (into.examples.length < 3) into.examples.push(x);
|
|
250
|
+
for (const [st, c] of r.statuses) into.statuses.set(st, (into.statuses.get(st) || 0) + c);
|
|
251
|
+
for (const c of r.clients) if (into.clients.size < MAX_CLIENTS) into.clients.add(c);
|
|
252
|
+
for (const h of r.hosts) into.hosts.add(h);
|
|
253
|
+
into.last = latest(into.last, r.last);
|
|
254
|
+
into.lastOk = latest(into.lastOk, r.lastOk) ?? into.lastOk ?? r.lastOk;
|
|
255
|
+
}
|
|
256
|
+
routes = [...merged.values()];
|
|
257
|
+
}
|
|
258
|
+
return routes.map((r) => ({ ...r, template: unique(r.segs).join('/') }));
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
/** {id}/{id} → {id}/{id2}: a template can't repeat a parameter name. */
|
|
262
|
+
function unique(segs) {
|
|
263
|
+
const seen = new Map();
|
|
264
|
+
return segs.map((s) => {
|
|
265
|
+
const m = /^\{(.+)\}$/.exec(s);
|
|
266
|
+
if (!m) return s;
|
|
267
|
+
const base = m[1].replace(/\d+$/, '');
|
|
268
|
+
const n = (seen.get(base) || 0) + 1;
|
|
269
|
+
seen.set(base, n);
|
|
270
|
+
return n === 1 ? `{${base}}` : `{${base}${n}}`;
|
|
271
|
+
});
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
function paramName(prev) {
|
|
275
|
+
const m = /^([a-z][a-z0-9]*?)(e?s)?$/i.exec(prev || '');
|
|
276
|
+
return m && m[1].length > 1 && !prev.startsWith('{') ? `${m[1]}Id` : 'id';
|
|
277
|
+
}
|
|
278
|
+
|
|
279
|
+
const median = (xs) => (xs.length ? [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)] : null);
|
|
280
|
+
// Buckets about 3% wide: a catch-all page that echoes the path varies by a few bytes.
|
|
281
|
+
const bucket = (bytes) => Math.round(Math.log(bytes + 64) / Math.log(1.03));
|
|
282
|
+
|
|
283
|
+
function record(map, key, base, rec, host) {
|
|
284
|
+
if (!map.has(key)) map.set(key, { ...base, calls: 0, statuses: new Map(), clients: new Set(), hosts: new Set(), last: null, lastOk: null });
|
|
285
|
+
const e = map.get(key);
|
|
286
|
+
e.calls++;
|
|
287
|
+
const s = rec.status || null;
|
|
288
|
+
e.statuses.set(s, (e.statuses.get(s) || 0) + 1);
|
|
289
|
+
if (rec.userAgent && e.clients.size < MAX_CLIENTS) e.clients.add(rec.userAgent);
|
|
290
|
+
if (host) e.hosts.add(host);
|
|
291
|
+
e.last = latest(e.last, rec.time);
|
|
292
|
+
if (ok(s)) e.lastOk = latest(e.lastOk, rec.time) ?? e.lastOk ?? true;
|
|
293
|
+
}
|
|
294
|
+
|
|
295
|
+
// 304 means the client's copy of an existing resource is current. Other 3xx
|
|
296
|
+
// prove nothing: an HTTP→HTTPS redirect answers every path, real or not.
|
|
297
|
+
const ok = (s) => (s >= 200 && s < 300) || s === 304;
|
|
298
|
+
const latest = (a, t) => (typeof t === 'number' && (typeof a !== 'number' || t > a) ? t : a);
|
|
299
|
+
const times = (n) => (n === 1 ? 'once' : `${n} times`);
|
|
300
|
+
|
|
301
|
+
function count(e, test) {
|
|
302
|
+
let n = 0;
|
|
303
|
+
for (const [s, c] of e.statuses) if (test(s)) n += c;
|
|
304
|
+
return n;
|
|
305
|
+
}
|
|
306
|
+
|
|
307
|
+
function judge(e, catchAll) {
|
|
308
|
+
const served = count(e, ok);
|
|
309
|
+
const exists = count(e, (s) => s === 401 || s === 403 || s === 400 || s === 422 || s === 429 || (s >= 500 && s < 600));
|
|
310
|
+
const unknown = count(e, (s) => s === null);
|
|
311
|
+
if (!served && !exists && !unknown) return null; // only 404/405: it isn't there
|
|
312
|
+
const write = WRITE.has(e.method);
|
|
313
|
+
const doubtful = [...e.hosts].some((h) => catchAll.has(h));
|
|
314
|
+
const conventional = Boolean(e.fixed);
|
|
315
|
+
let severity = 'low';
|
|
316
|
+
if (served && !doubtful && !conventional) {
|
|
317
|
+
if (e.kind === 'shadow-endpoint') severity = write || PRIVILEGED_PATH.test(e.template) ? 'high' : 'medium';
|
|
318
|
+
else severity = write ? 'high' : 'medium';
|
|
319
|
+
}
|
|
320
|
+
const notes = [describe(e)];
|
|
321
|
+
if (e.variants > 1) notes.push(`${e.variants} routes folded into this one, e.g. ${e.examples.join(', ')}`);
|
|
322
|
+
if (e.current) notes.push(`now ${e.current}`);
|
|
323
|
+
if (conventional) notes.push('a conventional health or discovery endpoint');
|
|
324
|
+
if (!served && unknown === e.calls) notes.push('the logs have no status, so it may not exist');
|
|
325
|
+
else if (!served) notes.push('never answered 2xx, but it exists');
|
|
326
|
+
if (doubtful) notes.push('this host looks like a catch-all');
|
|
327
|
+
return finding(e.kind, severity, { endpoint: `${e.method} ${e.template}`, detail: notes.join('; '), source: 'traffic' });
|
|
328
|
+
}
|
|
329
|
+
|
|
330
|
+
function describe(e) {
|
|
331
|
+
const statuses = [...e.statuses].sort((a, b) => b[1] - a[1]).slice(0, 4).map(([s, n]) => `${s ?? '?'}×${n}`).join(' ');
|
|
332
|
+
const last = typeof e.last === 'number' ? `, last ${new Date(e.last).toISOString().slice(0, 10)}` : '';
|
|
333
|
+
return `${e.calls} ${e.calls === 1 ? 'call' : 'calls'} from ${e.clients.size || '?'} ${e.clients.size === 1 ? 'client' : 'clients'} (${statuses})${last}`;
|
|
334
|
+
}
|
|
335
|
+
|
|
336
|
+
/** Operations whose x-sunset date has passed and that still answered 2xx after it. */
|
|
337
|
+
function sunsetPassed(spec, documented, now) {
|
|
338
|
+
const out = [];
|
|
339
|
+
if (!spec) return out;
|
|
340
|
+
for (const [path, item] of Object.entries(spec.paths || {})) {
|
|
341
|
+
if (!item || typeof item !== 'object') continue;
|
|
342
|
+
for (const [method, op] of Object.entries(item)) {
|
|
343
|
+
if (!op || typeof op !== 'object' || !op['x-sunset']) continue;
|
|
344
|
+
const v = op['x-sunset'];
|
|
345
|
+
const sunset = (v instanceof Date ? v : new Date(String(v))).getTime();
|
|
346
|
+
if (Number.isNaN(sunset) || sunset > now) continue;
|
|
347
|
+
const e = documented.get(`${method.toUpperCase()} ${path}`);
|
|
348
|
+
if (!e || !e.lastOk || (typeof e.lastOk === 'number' && e.lastOk < sunset)) continue;
|
|
349
|
+
out.push(finding('sunset-passed', 'medium', {
|
|
350
|
+
endpoint: `${method.toUpperCase()} ${path}`,
|
|
351
|
+
detail: `sunset ${new Date(sunset).toISOString().slice(0, 10)}; ${describe(e)}`,
|
|
352
|
+
source: 'traffic',
|
|
353
|
+
}));
|
|
354
|
+
}
|
|
355
|
+
}
|
|
356
|
+
return out;
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
function knownVersions(spec, matcher) {
|
|
360
|
+
const found = new Set();
|
|
361
|
+
const scan = (p) => p.split('/').forEach((s) => { if (VERSION.test(s)) found.add(s.toLowerCase()); });
|
|
362
|
+
for (const p of Object.keys(spec.paths || {})) scan(p);
|
|
363
|
+
matcher.basePaths.forEach(scan);
|
|
364
|
+
return [...found];
|
|
365
|
+
}
|
|
366
|
+
|
|
367
|
+
module.exports = { createInventory, PROBES };
|
|
@@ -48,6 +48,12 @@ const RULES = {
|
|
|
48
48
|
'cors-any-origin': ['API8', 'CORS allows any origin', 'Confirm this API holds nothing private; otherwise allow specific origins.'],
|
|
49
49
|
'insecure-cookie': ['API8', 'Session cookie missing protections', 'Set Secure, HttpOnly and SameSite.'],
|
|
50
50
|
'version-disclosure': ['API8', 'Server software version disclosed', 'Remove version details from response headers.'],
|
|
51
|
+
// Traffic vs spec
|
|
52
|
+
'shadow-endpoint': ['API9', 'Undocumented endpoint receiving traffic', 'Document it and give it the same review as the rest, or remove it.'],
|
|
53
|
+
'undocumented-method': ['API5', 'Undocumented method on a documented path', 'Answer 405 for it, or document and secure it.'],
|
|
54
|
+
'zombie-version': ['API9', 'Retired API version still served', 'Turn the old version off, or keep patching it like the current one.'],
|
|
55
|
+
'sunset-passed': ['API9', 'Still served after its sunset date', 'Turn it off as announced, or move x-sunset in the spec.'],
|
|
56
|
+
'exposed-sensitive-path': ['API8', 'Sensitive file or debug endpoint served', 'Block it at the server or proxy, and rotate anything it exposed.'],
|
|
51
57
|
'stack-trace': ['API8', 'Stack trace in an error response', 'Return a generic error and log the detail on the server.'],
|
|
52
58
|
};
|
|
53
59
|
|
|
@@ -15,6 +15,7 @@
|
|
|
15
15
|
* JSON with a host field, absolute request URLs), else null
|
|
16
16
|
* query raw query string without the leading '?' ('' when absent)
|
|
17
17
|
* status numeric status code, or null
|
|
18
|
+
* bytes response body size (combined format only), or null
|
|
18
19
|
* time epoch ms, or null when the line carries no parseable timestamp
|
|
19
20
|
* userAgent raw User-Agent, or null
|
|
20
21
|
* extras quoted fields after the User-Agent (combined format only) —
|
|
@@ -34,7 +35,7 @@ const HTTP_METHODS = new Set(['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'HEAD', '
|
|
|
34
35
|
// ─── combined / common ──────────────────────────────────────────────────────
|
|
35
36
|
|
|
36
37
|
// host ident authuser [time] "request" status bytes ["referer" "user-agent"] [extras...]
|
|
37
|
-
const COMBINED_RE = /^(\S+) \S+ \S+ \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}|-) \S+(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(.*)$/;
|
|
38
|
+
const COMBINED_RE = /^(\S+) \S+ \S+ \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}|-) (\S+)(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(.*)$/;
|
|
38
39
|
|
|
39
40
|
function parseCombined(line) {
|
|
40
41
|
const m = COMBINED_RE.exec(line);
|
|
@@ -44,9 +45,10 @@ function parseCombined(line) {
|
|
|
44
45
|
return {
|
|
45
46
|
...req,
|
|
46
47
|
status: m[4] === '-' ? null : Number(m[4]),
|
|
48
|
+
bytes: /^\d+$/.test(m[5]) ? Number(m[5]) : null,
|
|
47
49
|
time: parseClfTime(m[2]),
|
|
48
|
-
userAgent: m[
|
|
49
|
-
extras: m[
|
|
50
|
+
userAgent: m[7] !== undefined ? nullIfDash(unescape(m[7])) : null,
|
|
51
|
+
extras: m[8] ? tokenize(m[8]).map(nullIfDash) : [],
|
|
50
52
|
json: null,
|
|
51
53
|
};
|
|
52
54
|
}
|
|
@@ -221,12 +223,17 @@ function parseAnyTime(v) {
|
|
|
221
223
|
|
|
222
224
|
const PARSERS = { combined: parseCombined, alb: parseAlb, json: parseJsonLine };
|
|
223
225
|
|
|
224
|
-
|
|
226
|
+
// A line can have a format's shape without being a request: OpenLiteSpeed logs
|
|
227
|
+
// TLS probes and dropped connections as `"" 200`. They still say which format it is.
|
|
228
|
+
const SHAPES = { combined: COMBINED_RE };
|
|
229
|
+
|
|
230
|
+
/** Picks the format that parses (or at least fits) the most of the sample lines. */
|
|
225
231
|
function detectFormat(sampleLines) {
|
|
226
232
|
let best = null;
|
|
227
233
|
let bestHits = 0;
|
|
228
234
|
for (const [name, parse] of Object.entries(PARSERS)) {
|
|
229
|
-
const
|
|
235
|
+
const shape = SHAPES[name];
|
|
236
|
+
const hits = sampleLines.filter((l) => parse(l) || (shape && shape.test(l))).length;
|
|
230
237
|
if (hits > bestHits) { best = name; bestHits = hits; }
|
|
231
238
|
}
|
|
232
239
|
return best;
|
|
@@ -241,9 +248,9 @@ function splitRequestLine(s) {
|
|
|
241
248
|
if (parts.length < 2) return null;
|
|
242
249
|
const method = parts[0].toUpperCase();
|
|
243
250
|
if (!HTTP_METHODS.has(method)) return null;
|
|
244
|
-
const { path, query, host } = splitTarget(parts[1]);
|
|
251
|
+
const { path, query, host, trailingSlash } = splitTarget(parts[1]);
|
|
245
252
|
if (!path) return null;
|
|
246
|
-
return { method, path, query, host };
|
|
253
|
+
return { method, path, query, host, trailingSlash };
|
|
247
254
|
}
|
|
248
255
|
|
|
249
256
|
function splitTarget(target) {
|
|
@@ -262,8 +269,9 @@ function splitTarget(target) {
|
|
|
262
269
|
let path = q >= 0 ? t.slice(0, q) : t;
|
|
263
270
|
const query = q >= 0 ? t.slice(q + 1) : '';
|
|
264
271
|
path = path.replace(/\/{2,}/g, '/');
|
|
265
|
-
|
|
266
|
-
|
|
272
|
+
const trailingSlash = path.length > 1 && path.endsWith('/');
|
|
273
|
+
if (trailingSlash) path = path.slice(0, -1);
|
|
274
|
+
return { path, query, host, trailingSlash };
|
|
267
275
|
}
|
|
268
276
|
|
|
269
277
|
/** "API.Example.com:443" → "api.example.com"; "[::1]:80" → "[::1]". */
|