echelongraph-mcp 1.0.2 → 1.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +108 -12
- package/dist/index.js +530 -30
- package/package.json +5 -2
- package/server.json +6 -2
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 EchelonGraph
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -21,7 +21,7 @@ API call a tool needs to answer.
|
|
|
21
21
|
| `search_cves` | Search/filter CVEs (severity, min CVSS, text, sort) with EchelonGraph scores. |
|
|
22
22
|
| `get_cve` | Full detail for one CVE (CVSS v3/v4, EG score, EPSS, KEV+ransomware, GHSA, CWE, references). |
|
|
23
23
|
| `cve_exposure` | Internet-exposure footprint for a CVE: exposed service count (distinct ip:port, the `exposed_hosts` field) + country/product breakdown, from the KEV-exposure radar. |
|
|
24
|
-
| `exposure_radar` | Aggregate totals across the exposure radars:
|
|
24
|
+
| `exposure_radar` | Aggregate totals across the exposure radars: services running CISA-KEV CVEs; unauthenticated data stores and observability UIs, found through Shodan (LeakIX when Shodan query credits run low) and then confirmed by EchelonGraph's own identified check, which is not a pure read (on Redis it names its client; on ClickHouse its query lands in the server's query log); leaked credentials; shadow AI. Every number is labelled by what it counts, and a field the tool cannot label is left out and named. |
|
|
25
25
|
|
|
26
26
|
Every figure is as fresh as the schedule that refreshes it. The CVE feed is polled from its
|
|
27
27
|
sources on a schedule; each radar refreshes on its own.
|
|
@@ -40,8 +40,9 @@ Shodan InternetDB refreshes `last_seen` without re-reading the banner, so a patc
|
|
|
40
40
|
stay counted while its port stays open.
|
|
41
41
|
|
|
42
42
|
The unit is a service, not a machine. Shodan returns one banner per port and the radar keys
|
|
43
|
-
each observation on ip:port, so a machine answering on two ports counts twice. The API field
|
|
44
|
-
|
|
43
|
+
each observation on ip:port, so a machine answering on two ports counts twice. The API's field
|
|
44
|
+
names are kept (`exposed_hosts` here; `distinct_hosts`, `ransomware_hosts` and each ranked
|
|
45
|
+
row's count in `exposure_radar`), but every one of them counts ip:port services.
|
|
45
46
|
|
|
46
47
|
The radar only looks for its **tracked** set of CVEs, so a zero means different things:
|
|
47
48
|
|
|
@@ -52,13 +53,105 @@ The radar only looks for its **tracked** set of CVEs, so a zero means different
|
|
|
52
53
|
| no `tracked` field | The API did not say whether the CVE is in the tracked set: an older API, or the radar cannot decide (for example a CISA-KEV or high-EPSS CVE in a tracked product with 0 services on record). 0 exposed services on record, with no claim either way. |
|
|
53
54
|
| HTTP 400, or an id that is not `CVE-YYYY-NNNN…` | An error result tagged `invalid_input`; nothing was looked up. |
|
|
54
55
|
|
|
55
|
-
### How `exposure_radar` counts
|
|
56
|
+
### How `exposure_radar` counts
|
|
56
57
|
|
|
57
|
-
`
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
58
|
+
`exposure_radar` relays each radar's answer cut to the fields it can label, and its note labels
|
|
59
|
+
every number by what it counts. A field this version does not know is left out and the note
|
|
60
|
+
names it, whether it is a total or a field inside a ranked row; so is a known field in an
|
|
61
|
+
unexpected shape, and a list with one malformed row is left out whole, since a partial list
|
|
62
|
+
reads as a complete one. `kev_exposure`, `exposed_databases` and `leaked_credentials` also
|
|
63
|
+
carry `generated_at`, when the API computed their totals, and, when the API can tell,
|
|
64
|
+
`last_run_at`: when that radar last completed a check (each table below says what a completed
|
|
65
|
+
check is for that radar).
|
|
66
|
+
|
|
67
|
+
`last_run_at` is a timestamp, not a count. It is recorded for the radar as a whole, not for
|
|
68
|
+
the server instance that answered, in UTC to the second. It moves only at the end of a cycle
|
|
69
|
+
whose reads succeeded, so a cycle that read nothing leaves it where it was. It is not the time
|
|
70
|
+
of every record a radar's numbers count: those cover everything still on record, not only what
|
|
71
|
+
the last check found. Nor is it `generated_at`, which is only when the API recomputed the
|
|
72
|
+
totals. The API omits `last_run_at` when it has no completed check on record or cannot read
|
|
73
|
+
it; the result then carries none, and the note says nothing about it. When it is there, the
|
|
74
|
+
note gives it as, for example, "kev_exposure last completed check: 2026-09-27T03:12:44Z".
|
|
75
|
+
|
|
76
|
+
#### `kev_exposure`
|
|
77
|
+
|
|
78
|
+
Every count is over the services the KEV-exposure radar keeps: a Shodan banner whose version
|
|
79
|
+
maps to a CISA-KEV-listed CVE (see "How `cve_exposure` counts").
|
|
80
|
+
|
|
81
|
+
| Field | What it counts |
|
|
82
|
+
|---|---|
|
|
83
|
+
| `kev_exposure.distinct_hosts` | Distinct ip:port services, not machines, with at least one CISA-KEV-listed CVE on record. A machine answering on two ports counts twice. |
|
|
84
|
+
| `kev_exposure.ransomware_hosts` | The services among them with at least one ransomware-linked KEV CVE (one whose KEV entry notes known use in ransomware campaigns). |
|
|
85
|
+
| `kev_exposure.kev_cves_exposed` | Distinct CISA-KEV-listed CVEs with at least one service on record. |
|
|
86
|
+
| `kev_exposure.ransomware_cves` | The ransomware-linked CVEs among them. |
|
|
87
|
+
| `kev_exposure.correlations` | Service×CVE pairs, not services: a service with three KEV CVEs counts three times. |
|
|
88
|
+
| `kev_exposure.top_products`, `kev_exposure.top_countries`, `kev_exposure.top_cves` | Up to 12 products, 10 countries and 12 CVEs, ranked by distinct ip:port services. |
|
|
89
|
+
| `kev_exposure.top_cves[].cvss_v3_score`, `kev_exposure.top_cves[].epss_score` | Scores, not counts: the highest CVSS v3 base score and EPSS probability (0 to 1) recorded on that CVE's observations. |
|
|
90
|
+
| `kev_exposure.trend` | Service×CVE pairs, not services, by the week (starting Monday) in which each pair was first recorded, over the last 12 weeks. Only pairs still on record are counted, so earlier weeks read low. |
|
|
91
|
+
| `kev_exposure.newest_kev` | The 15 CVEs that EchelonGraph's CVE records most recently mark as CISA-KEV-listed (by `added_date`), each with an `exposure_state` (below). |
|
|
92
|
+
| `kev_exposure.newest_kev[].exposed_hosts` | Only on a row whose `exposure_state` is `exposed` (or `measured_zero`): distinct ip:port services on record with that CVE. |
|
|
93
|
+
| `kev_exposure.newest_kev[].cvss_v3_score`, `kev_exposure.newest_kev[].epss_score` | Scores, not counts: the CVE record's CVSS v3 base score and EPSS probability (0 to 1), absent when the record has none. |
|
|
94
|
+
| `kev_exposure.last_run_at` | A timestamp, not a count: when the radar last completed a check, a cycle in which its Shodan search answered at least one query (others may have failed) and its list of services due for a re-check was read. A cycle that skipped the search for want of Shodan query credits, or whose reads failed, does not move it. |
|
|
95
|
+
|
|
96
|
+
A `newest_kev` row's `exposure_state` says whether its count is a measurement:
|
|
97
|
+
|
|
98
|
+
| `exposure_state` | What it means |
|
|
99
|
+
|---|---|
|
|
100
|
+
| `exposed` | The radar has services on record with this CVE, and `exposed_hosts` counts them. |
|
|
101
|
+
| `not_assessed` | **NOT ASSESSED**: the answer holds no measurement for this CVE, so no count is relayed. The API answers 0 for every CVE the radar holds no service for, whether or not the radar looks for that CVE at all, and it never marks a CVE as tracked with 0 services, so that 0 is not a measurement. `cve_exposure` says per CVE whether it is in the radar's tracked set. |
|
|
102
|
+
| `measured_zero` | Only if the API ever marks a row `tracked: true` with 0 services: a zero in the radar's sample, not an internet-wide zero. If it marks a row `tracked: false`, that row is `not_assessed` whatever its count. |
|
|
103
|
+
|
|
104
|
+
#### `exposed_databases`
|
|
105
|
+
|
|
106
|
+
| Field | What it counts |
|
|
107
|
+
|---|---|
|
|
108
|
+
| `exposed_databases.distinct_hosts` | Distinct ip:port services, not machines, that EchelonGraph's check confirmed answering without authentication: data stores and observability UIs alike. |
|
|
109
|
+
| `exposed_databases.engines` | Distinct engine types among them (for example `redis` or `grafana`). |
|
|
110
|
+
| `exposed_databases.top_engines`, `exposed_databases.top_countries` | Up to 15 engines and 10 countries, ranked by those services. |
|
|
111
|
+
| `exposed_databases.pii_likely`, `exposed_databases.pci_likely` | Services whose schema names pass a high-confidence gate for personal data, or for payment-card data. The names are index, database, table or field names in the Shodan or LeakIX banner or in the check's response, never record values; the gate passes one unambiguous term (such as `ssn` or `cardholder`) or two distinct indicators. It is a precision-first schema gate, not a census: a service whose names do not pass it is not counted, whatever it holds, so the other services are not shown to hold no such data. |
|
|
112
|
+
| `exposed_databases.last_run_at` | A timestamp, not a count: when the radar last completed a check, a cycle in which its Shodan search, or the LeakIX fallback, answered at least one query (others may have failed), its list of services due for a re-check was read, and the scan opt-out register could be consulted. A cycle that searched nothing, or whose reads failed, does not move it. |
|
|
113
|
+
|
|
114
|
+
#### `leaked_credentials`
|
|
115
|
+
|
|
116
|
+
| Field | What it counts |
|
|
117
|
+
|---|---|
|
|
118
|
+
| `leaked_credentials.total` | (repository, secret) pairs, not distinct secrets: a secret committed to three repositories counts three times. |
|
|
119
|
+
| `leaked_credentials.distinct_secrets` | Each secret once. |
|
|
120
|
+
| `leaked_credentials.distinct_repos` | Public GitHub repositories with at least one. |
|
|
121
|
+
| `leaked_credentials.top_providers`, `leaked_credentials.top_types` | Up to 15 providers and secret types, ranked by (repository, secret) pairs. |
|
|
122
|
+
| `leaked_credentials.last_run_at` | A timestamp, not a count: when the radar last completed a check, a cycle that read the public GitHub event stream (fetches of some of the commits it lists may have failed). A cycle whose read of that stream failed does not move it. |
|
|
123
|
+
|
|
124
|
+
None of them is validated. Each is a credential-shaped string in a public commit that passed
|
|
125
|
+
EchelonGraph's filters, at most structurally checked (a checksum or format decode), and never
|
|
126
|
+
tested against its provider, so none of these is a count of working credentials.
|
|
127
|
+
|
|
128
|
+
#### `shadow_ai`
|
|
129
|
+
|
|
130
|
+
The shadow-AI radar's answer mixes numbers that count exposed services with numbers that count
|
|
131
|
+
every observation. So `exposure_radar` regroups it by what each number counts, and its note
|
|
132
|
+
labels each one. Only `shadow_ai.confirmed_exposed` counts exposed services.
|
|
133
|
+
|
|
134
|
+
| Field | What it counts |
|
|
135
|
+
|---|---|
|
|
136
|
+
| `confirmed_exposed.total` | Confirmed exposed: services EchelonGraph's probes found answering without an authentication gate (liveness `active`, or `rechecking` during a re-check). It is the sum of `confirmed_exposed.by_category`. |
|
|
137
|
+
| `confirmed_exposed.by_category` | The same services, by category. |
|
|
138
|
+
| `confirmed_exposed.last_24h` | Confirmed-exposed services first recorded in the last 24 hours. |
|
|
139
|
+
| `observed.total` | Every Certificate Transparency or Shodan observation on record, whatever its verification state: observed, not exposed. |
|
|
140
|
+
| `observed.by_category` | The same observations, by category. |
|
|
141
|
+
| `observed.last_24h` | Observations first recorded in the last 24 hours. |
|
|
142
|
+
| `observed.trend_30d` | Observations per UTC day over the last 30 days. |
|
|
143
|
+
| `observed.top_products`, `observed.top_countries`, `observed.top_issuers` | Up to ten products, countries and issuers, ranked by observations, not by exposed services. An issuer is the certificate's CA for a Certificate Transparency observation, and the hosting operator Shodan reports for a Shodan one. |
|
|
144
|
+
| `observed.last_observation` | When the latest observation was recorded. |
|
|
145
|
+
| `authentication.observed` | Observations where a probe observed an authentication gate: a 401/403, a login page or an auth marker. |
|
|
146
|
+
| `authentication.not_determined` | Observations whose service answered, but where no probe could tell whether it enforces authentication. |
|
|
147
|
+
|
|
148
|
+
Neither authentication count is part of `confirmed_exposed`, and `observed.total` minus
|
|
149
|
+
`confirmed_exposed.total` is not a count of secured services: it also holds observations not
|
|
150
|
+
yet verified, whose hostname no longer resolves, or whose service no longer answers openly.
|
|
151
|
+
|
|
152
|
+
The tool relays no number it cannot label. A stats field this version does not know is left
|
|
153
|
+
out, and the note names it; so is a count in an unexpected shape. A ranked row carries only
|
|
154
|
+
its name and its count, and a field added to one is named too.
|
|
62
155
|
|
|
63
156
|
## What a result means
|
|
64
157
|
|
|
@@ -68,9 +161,12 @@ outage for an all-clear:
|
|
|
68
161
|
- **Success** — the first text block is the API's JSON verbatim; the second is a one-line
|
|
69
162
|
note saying the call succeeded, which base URL answered, and what it found. When the
|
|
70
163
|
feed genuinely holds nothing for the query the note says so in words ("we looked and
|
|
71
|
-
found nothing … not a lookup failure"), because a measured zero is a measurement.
|
|
72
|
-
exception to verbatim
|
|
73
|
-
|
|
164
|
+
found nothing … not a lookup failure"), because a measured zero is a measurement. The
|
|
165
|
+
exception to verbatim is `exposure_radar`: each radar is cut to the fields listed above, and
|
|
166
|
+
each `kev_exposure.newest_kev` row gains an `exposure_state`. Its `shadow_ai` counts are
|
|
167
|
+
regrouped as above, and its `poller` block carries only `running` and `last_run_at`, or is dropped when it
|
|
168
|
+
carries no real completion time. The block's other fields describe the server instance that
|
|
169
|
+
answered, not the radar. Its `last_run_at` is when the radar's leader last
|
|
74
170
|
completed a Certificate Transparency (crt.sh) cycle, and its `running` is true only when
|
|
75
171
|
that was within 30 minutes of the answer. When `running` is true, the note says the
|
|
76
172
|
radar's leader last completed a cycle at that time; when it is false, the note says no
|
package/dist/index.js
CHANGED
|
@@ -10,7 +10,10 @@
|
|
|
10
10
|
// Result contract (#1874). A tool answers in exactly one of two shapes, and they never blur:
|
|
11
11
|
// - success: content[0] is the API's JSON verbatim; content[1] is a one-line note saying
|
|
12
12
|
// the call succeeded, where, and what it found — in words, including when it found
|
|
13
|
-
// nothing ("we looked and found nothing" is a measurement).
|
|
13
|
+
// nothing ("we looked and found nothing" is a measurement). One exception: exposure_radar
|
|
14
|
+
// relays each radar's answer cut to the fields it can label by what they count, and names
|
|
15
|
+
// what it left out (readKEVExposure, readExposedDatabases, readLeakedCredentials and
|
|
16
|
+
// readShadowAI; #2307, #2313).
|
|
14
17
|
// - failure (isError: true): the lookup did not complete — unreachable host, non-2xx,
|
|
15
18
|
// timeout, or a 2xx whose body is not a JSON object — named by tool, cause and base URL.
|
|
16
19
|
// Never a success with null fields: a model handed one of those tells its user "no
|
|
@@ -388,13 +391,26 @@ const RADARS = [
|
|
|
388
391
|
["leaked_credentials", "/api/v1/public/leaked-credentials/stats"],
|
|
389
392
|
["shadow_ai", "/api/v1/public/shadow-ai-radar/stats"],
|
|
390
393
|
];
|
|
391
|
-
// The shadow-AI stats answer,
|
|
394
|
+
// The shadow-AI stats answer, regrouped by what each number counts (#2307).
|
|
392
395
|
//
|
|
393
|
-
// stats
|
|
394
|
-
//
|
|
395
|
-
//
|
|
396
|
+
// GET /api/v1/public/shadow-ai-radar/stats (core-backend shadowctlog store.go Stats) mixes
|
|
397
|
+
// three kinds of number, and only one of them counts exposed services:
|
|
398
|
+
// - confirmed exposed: rows whose liveness is active or rechecking, the published exposure
|
|
399
|
+
// set (visible_by_category, last_24h_visible_count);
|
|
400
|
+
// - observed: every Certificate Transparency or Shodan observation on record, whatever its
|
|
401
|
+
// verification state (total, by_category, last_24h_count, trend_30d, and the top_products,
|
|
402
|
+
// top_countries and top_issuers rankings);
|
|
403
|
+
// - authentication state: rows where a probe observed an authentication gate (auth_confirmed,
|
|
404
|
+
// liveness authenticated) or could not tell (auth_undetermined, liveness inconclusive).
|
|
405
|
+
// On 2026-09-27 total was 36,222 against 2,000 confirmed exposed, and top_products ranked
|
|
406
|
+
// LiteLLM at 6,813 observations while 1,395 LLM-PROXY services were confirmed exposed. Relayed
|
|
407
|
+
// under the API's names, every one of those reads as an exposure count. So the tool relays them
|
|
408
|
+
// grouped as confirmed_exposed, observed and authentication, with the note labelling each, and
|
|
409
|
+
// relays nothing it cannot label: a stats field this version does not know, a field other than
|
|
410
|
+
// a ranked row's label and count, and the poller block's instance fields are left out, the
|
|
411
|
+
// first named in the note.
|
|
396
412
|
//
|
|
397
|
-
//
|
|
413
|
+
// The `poller` block's running and last_run_at describe the radar's fleet, not the instance
|
|
398
414
|
// that answered (core-backend shadowctlog Poller.Status): last_run_at is when the fleet's
|
|
399
415
|
// leader last COMPLETED a crt.sh (Certificate Transparency) cycle, and running is true only
|
|
400
416
|
// when that was within pollerSilentAfter, 30 minutes, of the answer. The API sends both or
|
|
@@ -402,52 +418,525 @@ const RADARS = [
|
|
|
402
418
|
// than that answered from the serving instance's own memory, so a follower sent running:false
|
|
403
419
|
// and last_run_at 0001-01-01T00:00:00Z, which reads as "the radar is down and has never run".
|
|
404
420
|
// So the note reads three cases: running with a real completion time, stopped since a real
|
|
405
|
-
// completion time, and unknown. Only the first two are relayed
|
|
406
|
-
// completion time and running verdict) is dropped, so no running:false
|
|
407
|
-
// the model as freshness.
|
|
421
|
+
// completion time, and unknown. Only the first two are relayed, as those two fields alone; an
|
|
422
|
+
// unknown block (no real completion time and running verdict) is dropped, so no running:false
|
|
423
|
+
// or zero time reaches the model as freshness. The block's other fields (interval_seconds,
|
|
424
|
+
// shodan_enabled, and a polling instance's last_new_inserts, skipped_as_follower,
|
|
425
|
+
// consecutive_fails and shodan_last_new) describe the instance that answered, not the fleet,
|
|
426
|
+
// and none of them counts AI services.
|
|
408
427
|
const SHADOW_AI_SILENT_AFTER = "30 minutes";
|
|
428
|
+
// The stats fields readShadowAI knows how to label. Any other is left out and named.
|
|
429
|
+
const SHADOW_AI_STATS = new Set([
|
|
430
|
+
"total",
|
|
431
|
+
"by_category",
|
|
432
|
+
"visible_by_category",
|
|
433
|
+
"last_observation",
|
|
434
|
+
"last_24h_count",
|
|
435
|
+
"last_24h_visible_count",
|
|
436
|
+
"auth_confirmed",
|
|
437
|
+
"auth_undetermined",
|
|
438
|
+
"top_products",
|
|
439
|
+
"top_countries",
|
|
440
|
+
"top_issuers",
|
|
441
|
+
"trend_30d",
|
|
442
|
+
]);
|
|
443
|
+
// The observed rankings and the daily series: API field, and the row field that names each row.
|
|
444
|
+
const SHADOW_AI_ROWS = [
|
|
445
|
+
["trend_30d", "date"],
|
|
446
|
+
["top_products", "product"],
|
|
447
|
+
["top_countries", "country"],
|
|
448
|
+
["top_issuers", "issuer"],
|
|
449
|
+
];
|
|
450
|
+
const isCount = (v) => typeof v === "number" && Number.isFinite(v);
|
|
451
|
+
const isPlainObject = (v) => v !== null && typeof v === "object" && !Array.isArray(v);
|
|
452
|
+
// A category → count map, or nothing unless every value is a count: a partial map is an
|
|
453
|
+
// undercount that reads as a measurement (store.go categoryCounts).
|
|
454
|
+
function countMap(v) {
|
|
455
|
+
if (!isPlainObject(v))
|
|
456
|
+
return undefined;
|
|
457
|
+
const entries = Object.entries(v);
|
|
458
|
+
return entries.every(([, n]) => isCount(n)) ? Object.fromEntries(entries) : undefined;
|
|
459
|
+
}
|
|
460
|
+
const isKind = (v, k) => (k === "number" ? isCount(v) : typeof v === k);
|
|
461
|
+
// A ranking, a series or a list, each row cut down to the fields its spec names, so a field the
|
|
462
|
+
// backend adds to a row later is left out rather than relayed unlabelled; extra lists those
|
|
463
|
+
// fields so the note can name them. All rows or none: a row that is not an object, lacks a
|
|
464
|
+
// required field, or carries a known field of the wrong type leaves the whole list out, since
|
|
465
|
+
// a partial list reads as a complete one. An optional field may be absent (Go's omitempty).
|
|
466
|
+
function readRows(v, spec) {
|
|
467
|
+
if (!Array.isArray(v))
|
|
468
|
+
return { extra: [] };
|
|
469
|
+
const optional = spec.optional ?? {};
|
|
470
|
+
const rows = [];
|
|
471
|
+
const extra = new Set();
|
|
472
|
+
for (const r of v) {
|
|
473
|
+
if (!isPlainObject(r))
|
|
474
|
+
return { extra: [] };
|
|
475
|
+
const row = {};
|
|
476
|
+
for (const [k, kind] of Object.entries(spec.required)) {
|
|
477
|
+
if (!isKind(r[k], kind))
|
|
478
|
+
return { extra: [] };
|
|
479
|
+
row[k] = r[k];
|
|
480
|
+
}
|
|
481
|
+
for (const [k, kind] of Object.entries(optional)) {
|
|
482
|
+
if (r[k] === undefined)
|
|
483
|
+
continue;
|
|
484
|
+
if (!isKind(r[k], kind))
|
|
485
|
+
return { extra: [] };
|
|
486
|
+
row[k] = r[k];
|
|
487
|
+
}
|
|
488
|
+
// An own-property test: `in` would also find toString and constructor on the prototype.
|
|
489
|
+
for (const k of Object.keys(r))
|
|
490
|
+
if (!Object.prototype.hasOwnProperty.call(spec.required, k) && !Object.prototype.hasOwnProperty.call(optional, k))
|
|
491
|
+
extra.add(k);
|
|
492
|
+
rows.push(row);
|
|
493
|
+
}
|
|
494
|
+
return { rows, extra: [...extra] };
|
|
495
|
+
}
|
|
496
|
+
// A ranking or a daily series whose rows are a label and a count (the shadow-AI shape).
|
|
497
|
+
const countRows = (v, label) => readRows(v, { required: { [label]: "string", count: "number" } });
|
|
498
|
+
// The note's sentences about what a radar's answer carried that the tool left out.
|
|
499
|
+
function leftOut(radar, unknown, unusable) {
|
|
500
|
+
const out = [];
|
|
501
|
+
if (unknown.length)
|
|
502
|
+
out.push(`Left out of ${radar} because this version of the tool cannot label them: ${unknown.join(", ")}.`);
|
|
503
|
+
if (unusable.length)
|
|
504
|
+
out.push(`Left out of ${radar} because they were not in the expected shape: ${unusable.join(", ")}.`);
|
|
505
|
+
return out;
|
|
506
|
+
}
|
|
507
|
+
function readRadar(body, spec) {
|
|
508
|
+
const src = isPlainObject(body) ? body : {};
|
|
509
|
+
const data = {};
|
|
510
|
+
const unknown = [];
|
|
511
|
+
const unusable = [];
|
|
512
|
+
const present = (k) => src[k] !== undefined && src[k] !== null;
|
|
513
|
+
for (const k of spec.counts) {
|
|
514
|
+
if (isCount(src[k]))
|
|
515
|
+
data[k] = src[k];
|
|
516
|
+
else if (present(k))
|
|
517
|
+
unusable.push(k);
|
|
518
|
+
}
|
|
519
|
+
for (const [k, rowSpec] of Object.entries(spec.rows)) {
|
|
520
|
+
const r = readRows(src[k], rowSpec);
|
|
521
|
+
if (r.rows) {
|
|
522
|
+
data[k] = r.rows;
|
|
523
|
+
unknown.push(...r.extra.map((x) => `${k}[].${x}`));
|
|
524
|
+
}
|
|
525
|
+
else if (present(k))
|
|
526
|
+
unusable.push(k);
|
|
527
|
+
}
|
|
528
|
+
// A timestamp that is not a real instant (the Go zero time) is not relayed as one.
|
|
529
|
+
for (const k of spec.instants) {
|
|
530
|
+
const s = strAt(src, k);
|
|
531
|
+
if (realInstant(s))
|
|
532
|
+
data[k] = s;
|
|
533
|
+
else if (present(k) && typeof src[k] !== "string")
|
|
534
|
+
unusable.push(k);
|
|
535
|
+
}
|
|
536
|
+
const known = new Set([...spec.counts, ...Object.keys(spec.rows), ...spec.instants]);
|
|
537
|
+
unknown.unshift(...Object.keys(src).filter((k) => !known.has(k)));
|
|
538
|
+
return { data, unknown, unusable };
|
|
539
|
+
}
|
|
540
|
+
// ── last_run_at: when a radar last COMPLETED a check (#2335) ──
|
|
541
|
+
//
|
|
542
|
+
// Since #2335 the kev_exposure, exposed_databases and leaked_credentials stats answers carry
|
|
543
|
+
// last_run_at (core-backend pollerlock/published.go): the radar's completion row in
|
|
544
|
+
// poller_run_state, recorded for the radar as a whole rather than for the instance that
|
|
545
|
+
// answered, UTC, truncated to the second. Each poller writes that row at the END of a cycle,
|
|
546
|
+
// and only when the cycle's reads succeeded (poller.go checkCompleted / recordCheck), so a
|
|
547
|
+
// cycle that read nothing never moves it. What counts as a completed check differs per radar,
|
|
548
|
+
// and LAST_CHECK words each. The API omits the field when no completed check is on record,
|
|
549
|
+
// when its read failed, or when its last good read is more than two minutes old; it never
|
|
550
|
+
// sends the Go zero time.
|
|
551
|
+
//
|
|
552
|
+
// It is a timestamp, not a count, and not the time of the numbers beside it: those count
|
|
553
|
+
// everything still on record, not only what the last check found. Nor is it generated_at,
|
|
554
|
+
// which is only when the API's 60 s stats cache recomputed the totals. An absent stamp (or one
|
|
555
|
+
// that is not a real instant) means the API could not tell, so the note says nothing about it:
|
|
556
|
+
// no "never", no zero.
|
|
557
|
+
const LAST_CHECK = {
|
|
558
|
+
kev_exposure: "its Shodan search answered at least one query (others may have failed) and its list of services due for a re-check was read; a cycle that skipped the search for want of Shodan query credits, or whose reads failed, does not move it",
|
|
559
|
+
exposed_databases: "its Shodan search, or the LeakIX fallback, answered at least one query (others may have failed), its list of services due for a re-check was read, and the scan opt-out register could be consulted; a cycle that searched nothing, or whose reads failed, does not move it",
|
|
560
|
+
leaked_credentials: "it read the public GitHub event stream (fetches of some of the commits it lists may have failed); a cycle whose read of that stream failed does not move it",
|
|
561
|
+
};
|
|
562
|
+
// The note's sentences for one radar's relayed last_run_at, each naming the field so that any
|
|
563
|
+
// one of them quoted alone still says what it is; none when the stamp was not relayed.
|
|
564
|
+
function lastCheck(radar, data) {
|
|
565
|
+
const at = data.last_run_at;
|
|
566
|
+
if (typeof at !== "string")
|
|
567
|
+
return [];
|
|
568
|
+
return [
|
|
569
|
+
`${radar} last completed check: ${at} (${radar}.last_run_at, a timestamp, not a count).`,
|
|
570
|
+
`${radar}.last_run_at is when the radar last finished a cycle whose reads succeeded: ${LAST_CHECK[radar]}.`,
|
|
571
|
+
`${radar}.last_run_at is not the time of every record the ${radar} numbers count, which cover everything still on record, not only what that check found; nor is it ${radar}.generated_at, when the API computed those numbers.`,
|
|
572
|
+
];
|
|
573
|
+
}
|
|
574
|
+
// The first row of a relayed ranking, for the note: "(first: http_server, 9120 services)".
|
|
575
|
+
function firstOf(rows, label, count, unit) {
|
|
576
|
+
const top = Array.isArray(rows) ? rows[0] : undefined;
|
|
577
|
+
return top === undefined ? "" : ` (first: ${String(top[label])}, ${String(top[count])} ${unit})`;
|
|
578
|
+
}
|
|
579
|
+
// "a", "a and b", "a, b and c".
|
|
580
|
+
const listed = (xs) => (xs.length < 2 ? xs.join("") : `${xs.slice(0, -1).join(", ")} and ${xs[xs.length - 1]}`);
|
|
581
|
+
const numIn = (d, k) => (isCount(d[k]) ? d[k] : undefined);
|
|
582
|
+
const be = (n) => (n === 1 ? "is" : "are");
|
|
583
|
+
// ── kev_exposure: GET /api/v1/public/kev-exposure/stats (#2313 items 6 and 7) ──
|
|
584
|
+
//
|
|
585
|
+
// core-backend/internal/kevexposure store.go Stats. Every count is over the observation
|
|
586
|
+
// table, one row per (ip:port, CVE) (Upsert keys it obsID(host, cve), the host being
|
|
587
|
+
// "ip:port" from poller.go), filtered to rows whose kev_listed flag the poller copied in at
|
|
588
|
+
// match time:
|
|
589
|
+
// distinct_hosts COUNT(DISTINCT host): distinct ip:port services, not machines
|
|
590
|
+
// kev_cves_exposed COUNT(DISTINCT cve_id): CVEs with at least one such service
|
|
591
|
+
// correlations COUNT(*): service×CVE pairs, so a service with three CVEs counts 3 times
|
|
592
|
+
// ransomware_cves COUNT(DISTINCT cve_id) FILTER (WHERE ransomware)
|
|
593
|
+
// ransomware_hosts COUNT(DISTINCT host) FILTER (WHERE ransomware): services, not machines
|
|
594
|
+
// top_products / top_cves / top_countries COUNT(DISTINCT host) per group, LIMIT 12/12/10;
|
|
595
|
+
// a top_cves row's cvss_v3_score and epss_score are MAX() over its rows
|
|
596
|
+
// trend COUNT(*) per date_trunc('week', first_seen) over 12 weeks: pairs still on
|
|
597
|
+
// record, by the week first recorded (a pruned pair is gone, so earlier
|
|
598
|
+
// weeks read low)
|
|
599
|
+
// newest_kev the 15 cves rows with kev_listed and the latest kev_added_date, LEFT JOINed
|
|
600
|
+
// to COUNT(DISTINCT host) of their KEV-flagged rows, COALESCE(e.h, 0).
|
|
601
|
+
// last_run_at the radar's last completed check, KeyKEVExposureCheckCompleted (#2335; see
|
|
602
|
+
// LAST_CHECK). Its own row: KeyKEVExposure is the Shodan-credit watermark.
|
|
603
|
+
// So newest_kev[].exposed_hosts is 0 for every CVE the radar holds no row for, whether or not
|
|
604
|
+
// it could ever hold one: the join default, not a measurement. The per-CVE endpoint decides
|
|
605
|
+
// that with `tracked` (store.go trackedVerdict), which is true ONLY when rows exist
|
|
606
|
+
// (exposedHosts > 0) and otherwise false or absent: the backend never reports a measured zero.
|
|
607
|
+
// So a 0 here is never relayed: its row says exposure_state not_assessed and carries no
|
|
608
|
+
// count. If the stats answer ever sends a row's `tracked` (the contract proposed on #2313), it
|
|
609
|
+
// is read the way cve_exposure reads it: false is not assessed whatever the count, and true
|
|
610
|
+
// with 0 is a measured zero in the radar's sample.
|
|
611
|
+
const KEV_SPEC = {
|
|
612
|
+
counts: ["distinct_hosts", "ransomware_hosts", "kev_cves_exposed", "ransomware_cves", "correlations"],
|
|
613
|
+
rows: {
|
|
614
|
+
top_products: { required: { product: "string", hosts: "number" } },
|
|
615
|
+
top_countries: { required: { country: "string", hosts: "number" } },
|
|
616
|
+
top_cves: {
|
|
617
|
+
required: { cve_id: "string", hosts: "number" },
|
|
618
|
+
optional: { severity: "string", cvss_v3_score: "number", epss_score: "number", ransomware: "boolean" },
|
|
619
|
+
},
|
|
620
|
+
trend: { required: { week: "string", new_exposures: "number" } },
|
|
621
|
+
newest_kev: {
|
|
622
|
+
required: { cve_id: "string", exposed_hosts: "number" },
|
|
623
|
+
optional: {
|
|
624
|
+
added_date: "string",
|
|
625
|
+
vuln_name: "string",
|
|
626
|
+
severity: "string",
|
|
627
|
+
cvss_v3_score: "number",
|
|
628
|
+
epss_score: "number",
|
|
629
|
+
ransomware: "boolean",
|
|
630
|
+
tracked: "boolean",
|
|
631
|
+
},
|
|
632
|
+
},
|
|
633
|
+
},
|
|
634
|
+
instants: ["generated_at", "last_run_at"],
|
|
635
|
+
};
|
|
636
|
+
// One newest_kev row as relayed: the CVE record's fields, its exposure_state, and a count only
|
|
637
|
+
// when the count is a measurement. `tracked` is read, not relayed: exposure_state says it.
|
|
638
|
+
function newestKEVRow(r) {
|
|
639
|
+
const hosts = r.exposed_hosts;
|
|
640
|
+
const tracked = r.tracked;
|
|
641
|
+
const state = tracked === false ? "not_assessed" : hosts > 0 ? "exposed" : tracked === true ? "measured_zero" : "not_assessed";
|
|
642
|
+
const row = {};
|
|
643
|
+
for (const k of ["cve_id", "added_date", "vuln_name", "severity", "cvss_v3_score", "epss_score", "ransomware"]) {
|
|
644
|
+
if (r[k] !== undefined)
|
|
645
|
+
row[k] = r[k];
|
|
646
|
+
}
|
|
647
|
+
row.exposure_state = state;
|
|
648
|
+
if (state !== "not_assessed")
|
|
649
|
+
row.exposed_hosts = hosts;
|
|
650
|
+
return { row, state, outside: tracked === false };
|
|
651
|
+
}
|
|
652
|
+
function readKEVExposure(body) {
|
|
653
|
+
const { data, unknown, unusable } = readRadar(body, KEV_SPEC);
|
|
654
|
+
const sentences = [];
|
|
655
|
+
const n = (k) => numIn(data, k);
|
|
656
|
+
const totals = [
|
|
657
|
+
n("distinct_hosts") !== undefined &&
|
|
658
|
+
`kev_exposure.distinct_hosts (${n("distinct_hosts")}) counts distinct ip:port services, not machines (a machine answering on two ports counts twice), with at least one CISA-KEV-listed CVE on record`,
|
|
659
|
+
n("ransomware_hosts") !== undefined &&
|
|
660
|
+
`kev_exposure.ransomware_hosts (${n("ransomware_hosts")}) counts the services among them with at least one ransomware-linked KEV CVE (one whose KEV entry notes known use in ransomware campaigns)`,
|
|
661
|
+
n("kev_cves_exposed") !== undefined &&
|
|
662
|
+
`kev_exposure.kev_cves_exposed (${n("kev_cves_exposed")}) counts distinct CISA-KEV-listed CVEs with at least one service on record`,
|
|
663
|
+
n("ransomware_cves") !== undefined && `kev_exposure.ransomware_cves (${n("ransomware_cves")}) counts the ransomware-linked CVEs among them`,
|
|
664
|
+
n("correlations") !== undefined &&
|
|
665
|
+
`kev_exposure.correlations (${n("correlations")}) counts service×CVE pairs, not services: a service with three KEV CVEs counts three times`,
|
|
666
|
+
].filter((p) => typeof p === "string");
|
|
667
|
+
if (totals.length)
|
|
668
|
+
sentences.push(`KEV exposure: ${totals.join("; ")}.`);
|
|
669
|
+
const rankings = [
|
|
670
|
+
data.top_products !== undefined && `kev_exposure.top_products ranks up to 12 products${firstOf(data.top_products, "product", "hosts", "services")}`,
|
|
671
|
+
data.top_countries !== undefined && `kev_exposure.top_countries ranks up to 10 countries${firstOf(data.top_countries, "country", "hosts", "services")}`,
|
|
672
|
+
data.top_cves !== undefined && `kev_exposure.top_cves ranks up to 12 CVEs${firstOf(data.top_cves, "cve_id", "hosts", "services")}`,
|
|
673
|
+
].filter((p) => typeof p === "string");
|
|
674
|
+
if (rankings.length)
|
|
675
|
+
sentences.push(`${listed(rankings)}, each by distinct ip:port services with a CISA-KEV-listed CVE on record.`);
|
|
676
|
+
if (data.top_cves !== undefined) {
|
|
677
|
+
sentences.push("kev_exposure.top_cves[].cvss_v3_score and kev_exposure.top_cves[].epss_score are the highest CVSS v3 base score and EPSS probability (0 to 1) recorded on that CVE's observations: scores, not counts.");
|
|
678
|
+
}
|
|
679
|
+
if (data.trend !== undefined) {
|
|
680
|
+
sentences.push("kev_exposure.trend counts service×CVE pairs, not services, by the week (starting Monday) in which each pair was first recorded, over the last 12 weeks; only pairs still on record are counted, so earlier weeks read low.");
|
|
681
|
+
}
|
|
682
|
+
// newest_kev: a 0 is the join's default, never a measurement (see the block comment above).
|
|
683
|
+
if (Array.isArray(data.newest_kev)) {
|
|
684
|
+
const read = data.newest_kev.map(newestKEVRow);
|
|
685
|
+
data.newest_kev = read.map((x) => x.row);
|
|
686
|
+
const ids = (pred) => read.filter(pred).map((x) => String(x.row.cve_id));
|
|
687
|
+
const exposed = read.filter((x) => x.state === "exposed");
|
|
688
|
+
const zero = ids((x) => x.state === "measured_zero");
|
|
689
|
+
const outside = ids((x) => x.state === "not_assessed" && x.outside);
|
|
690
|
+
const unsaid = ids((x) => x.state === "not_assessed" && !x.outside);
|
|
691
|
+
sentences.push(read.length
|
|
692
|
+
? `kev_exposure.newest_kev lists the ${read.length} CVEs that EchelonGraph's CVE records most recently mark as CISA-KEV-listed (by added_date), each with an exposure_state.`
|
|
693
|
+
: "kev_exposure.newest_kev is empty: EchelonGraph's CVE records returned no CISA-KEV-listed CVE with a date added.");
|
|
694
|
+
if (exposed.length) {
|
|
695
|
+
sentences.push(`${exposed.length} of them ${be(exposed.length)} exposed (exposure_state exposed): kev_exposure.newest_kev[].exposed_hosts counts distinct ip:port services on record with that CVE (${exposed.map((x) => `${String(x.row.cve_id)}: ${String(x.row.exposed_hosts)}`).join(", ")}).`);
|
|
696
|
+
}
|
|
697
|
+
if (zero.length) {
|
|
698
|
+
sentences.push(`${zero.length} ${be(zero.length)} a measured zero in the radar's sample (exposure_state measured_zero, exposed_hosts 0): the API says the radar tracks them and holds no service for them, though it reads at most the first 100 Shodan results per tracked-product query, so this is not an internet-wide zero: ${zero.join(", ")}.`);
|
|
699
|
+
}
|
|
700
|
+
if (outside.length) {
|
|
701
|
+
sentences.push(`${outside.length} ${be(outside.length)} NOT ASSESSED (exposure_state not_assessed) because the API says they are outside the radar's tracked set, so no count is relayed for them: ${outside.join(", ")}.`);
|
|
702
|
+
}
|
|
703
|
+
if (unsaid.length) {
|
|
704
|
+
sentences.push(`${unsaid.length} ${be(unsaid.length)} NOT ASSESSED (exposure_state not_assessed): ${unsaid.join(", ")}. The API answers 0 for each, and that 0 is not a measurement: it is what the stats answer gives any CVE the radar holds no service for, whether or not the radar looks for that CVE at all, so no count is relayed for them. cve_exposure says per CVE whether it is in the radar's tracked set.`);
|
|
705
|
+
}
|
|
706
|
+
sentences.push("kev_exposure.newest_kev[].cvss_v3_score and kev_exposure.newest_kev[].epss_score are the CVE record's CVSS v3 base score and EPSS probability (0 to 1), absent when the record has none: scores, not counts.");
|
|
707
|
+
}
|
|
708
|
+
sentences.push(...lastCheck("kev_exposure", data));
|
|
709
|
+
sentences.push(...leftOut("kev_exposure", unknown, unusable));
|
|
710
|
+
return { data, sentences };
|
|
711
|
+
}
|
|
712
|
+
// ── exposed_databases: GET /api/v1/public/exposed-databases/stats (#2313 item 7) ──
|
|
713
|
+
//
|
|
714
|
+
// core-backend/internal/exposeddb store.go Stats, over one row per (ip:port, engine)
|
|
715
|
+
// (obsID(host, engine); the host is "ip:port", poller.go obsFrom). A row is stored only after
|
|
716
|
+
// EchelonGraph's own check confirmed the service answering without authentication.
|
|
717
|
+
// distinct_hosts COUNT(DISTINCT host): ip:port services, not machines
|
|
718
|
+
// engines COUNT(DISTINCT engine): engine types (mongodb, redis, grafana, ...)
|
|
719
|
+
// pii_likely / pci_likely COUNT(DISTINCT host) whose data_class names pii / pci AND whose
|
|
720
|
+
// schema terms pass the high-confidence gate (2+ distinct indicators, or one
|
|
721
|
+
// tier-1 term), mirroring classify.go dataConfidence. The classifier reads
|
|
722
|
+
// NAMES only (index / database / table / field names in the banner and the
|
|
723
|
+
// check's response, classify.go), never a record, and is precision-first:
|
|
724
|
+
// a service whose names miss the gate is not counted, whatever it holds.
|
|
725
|
+
// So these are a schema gate, not a census, and the rest are not "no PII".
|
|
726
|
+
// top_engines / top_countries COUNT(DISTINCT host) per group, LIMIT 15 / 10.
|
|
727
|
+
// last_run_at the radar's last completed check, KeyExposedDBCheckCompleted (#2335; see
|
|
728
|
+
// LAST_CHECK). Its own row: KeyExposedDB is claimed at the START of a cycle.
|
|
729
|
+
const EXPOSED_DB_SPEC = {
|
|
730
|
+
counts: ["distinct_hosts", "engines", "pii_likely", "pci_likely"],
|
|
731
|
+
rows: {
|
|
732
|
+
top_engines: { required: { engine: "string", hosts: "number" } },
|
|
733
|
+
top_countries: { required: { country: "string", hosts: "number" } },
|
|
734
|
+
},
|
|
735
|
+
instants: ["generated_at", "last_run_at"],
|
|
736
|
+
};
|
|
737
|
+
function readExposedDatabases(body) {
|
|
738
|
+
const { data, unknown, unusable } = readRadar(body, EXPOSED_DB_SPEC);
|
|
739
|
+
const sentences = [];
|
|
740
|
+
const n = (k) => numIn(data, k);
|
|
741
|
+
const totals = [
|
|
742
|
+
n("distinct_hosts") !== undefined &&
|
|
743
|
+
`exposed_databases.distinct_hosts (${n("distinct_hosts")}) counts distinct ip:port services, not machines (a machine answering on two ports counts twice), that EchelonGraph's check confirmed answering without authentication, data stores and observability UIs alike`,
|
|
744
|
+
n("engines") !== undefined && `exposed_databases.engines (${n("engines")}) counts the distinct engine types among them`,
|
|
745
|
+
].filter((p) => typeof p === "string");
|
|
746
|
+
if (totals.length)
|
|
747
|
+
sentences.push(`Exposed data stores: ${totals.join("; ")}.`);
|
|
748
|
+
const rankings = [
|
|
749
|
+
data.top_engines !== undefined && `exposed_databases.top_engines ranks up to 15 engines${firstOf(data.top_engines, "engine", "hosts", "services")}`,
|
|
750
|
+
data.top_countries !== undefined && `exposed_databases.top_countries ranks up to 10 countries${firstOf(data.top_countries, "country", "hosts", "services")}`,
|
|
751
|
+
].filter((p) => typeof p === "string");
|
|
752
|
+
if (rankings.length)
|
|
753
|
+
sentences.push(`${listed(rankings)}, each by those services.`);
|
|
754
|
+
const gated = [
|
|
755
|
+
n("pii_likely") !== undefined && `exposed_databases.pii_likely (${n("pii_likely")}) counts services whose schema names pass a high-confidence gate for personal data`,
|
|
756
|
+
n("pci_likely") !== undefined && `exposed_databases.pci_likely (${n("pci_likely")}) counts services whose schema names pass a high-confidence gate for payment-card data`,
|
|
757
|
+
].filter((p) => typeof p === "string");
|
|
758
|
+
if (gated.length) {
|
|
759
|
+
sentences.push(`${gated.join("; ")}. The names are index, database, table or field names in the Shodan or LeakIX banner or in the check's response, never record values, and the gate passes one unambiguous term (such as ssn or cardholder) or two distinct indicators. It is precision-first: a service whose names do not pass it is not counted, whatever it holds, so the other services are not shown to hold no personal or card data.`);
|
|
760
|
+
}
|
|
761
|
+
sentences.push(...lastCheck("exposed_databases", data));
|
|
762
|
+
sentences.push(...leftOut("exposed_databases", unknown, unusable));
|
|
763
|
+
return { data, sentences };
|
|
764
|
+
}
|
|
765
|
+
// ── leaked_credentials: GET /api/v1/public/leaked-credentials/stats (#2313 item 7) ──
|
|
766
|
+
//
|
|
767
|
+
// core-backend/internal/leakedcreds store.go Stats, over one row per (repository, secret):
|
|
768
|
+
// obsID(repo, fingerprint), "one row per leaked secret per repo".
|
|
769
|
+
// total COUNT(*): (repository, secret) pairs, so one secret in three
|
|
770
|
+
// repositories counts three times; not distinct secrets
|
|
771
|
+
// distinct_secrets COUNT(DISTINCT fingerprint); distinct_repos COUNT(DISTINCT repo)
|
|
772
|
+
// top_providers / top_types COUNT(*) per group, LIMIT 15: pairs again
|
|
773
|
+
// None is validated: detect.go says "verified" means STRUCTURALLY verified (a checksum or format
|
|
774
|
+
// decode), never "confirmed live", and the radar never uses a credential it finds.
|
|
775
|
+
// last_run_at the radar's last completed check, its own KeyLeakedCreds row, written at
|
|
776
|
+
// the END of a cycle that read the event stream (#2335; see LAST_CHECK).
|
|
777
|
+
const LEAKED_CREDS_SPEC = {
|
|
778
|
+
counts: ["total", "distinct_secrets", "distinct_repos"],
|
|
779
|
+
rows: {
|
|
780
|
+
top_providers: { required: { provider: "string", count: "number" } },
|
|
781
|
+
top_types: { required: { secret_type: "string", count: "number" } },
|
|
782
|
+
},
|
|
783
|
+
instants: ["generated_at", "last_run_at"],
|
|
784
|
+
};
|
|
785
|
+
function readLeakedCredentials(body) {
|
|
786
|
+
const { data, unknown, unusable } = readRadar(body, LEAKED_CREDS_SPEC);
|
|
787
|
+
const sentences = [];
|
|
788
|
+
const n = (k) => numIn(data, k);
|
|
789
|
+
const totals = [
|
|
790
|
+
n("total") !== undefined &&
|
|
791
|
+
`leaked_credentials.total (${n("total")}) counts (repository, secret) pairs, not distinct secrets: a secret committed to three repositories counts three times`,
|
|
792
|
+
n("distinct_secrets") !== undefined && `leaked_credentials.distinct_secrets (${n("distinct_secrets")}) counts each secret once`,
|
|
793
|
+
n("distinct_repos") !== undefined && `leaked_credentials.distinct_repos (${n("distinct_repos")}) counts public GitHub repositories`,
|
|
794
|
+
].filter((p) => typeof p === "string");
|
|
795
|
+
if (totals.length)
|
|
796
|
+
sentences.push(`Leaked credentials: ${totals.join("; ")}.`);
|
|
797
|
+
const rankings = [
|
|
798
|
+
data.top_providers !== undefined && `leaked_credentials.top_providers ranks up to 15 providers${firstOf(data.top_providers, "provider", "count", "pairs")}`,
|
|
799
|
+
data.top_types !== undefined && `leaked_credentials.top_types ranks up to 15 secret types${firstOf(data.top_types, "secret_type", "count", "pairs")}`,
|
|
800
|
+
].filter((p) => typeof p === "string");
|
|
801
|
+
if (rankings.length)
|
|
802
|
+
sentences.push(`${listed(rankings)}, each by the same (repository, secret) pairs.`);
|
|
803
|
+
if (totals.length || rankings.length) {
|
|
804
|
+
sentences.push("None of these is validated: each is a credential-shaped string in a public commit that passed EchelonGraph's filters, at most structurally checked (a checksum or format decode) and never tested against its provider, so none of them is a count of working credentials.");
|
|
805
|
+
}
|
|
806
|
+
sentences.push(...lastCheck("leaked_credentials", data));
|
|
807
|
+
sentences.push(...leftOut("leaked_credentials", unknown, unusable));
|
|
808
|
+
return { data, sentences };
|
|
809
|
+
}
|
|
409
810
|
function readShadowAI(sa) {
|
|
410
811
|
const sentences = [];
|
|
411
|
-
const
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
812
|
+
const stats = field(sa, "stats");
|
|
813
|
+
const used = new Set();
|
|
814
|
+
const take = (key, v) => {
|
|
815
|
+
if (v !== undefined)
|
|
816
|
+
used.add(key);
|
|
817
|
+
return v;
|
|
818
|
+
};
|
|
819
|
+
// Confirmed exposed: liveness IN ('active','rechecking') (store.go Stats, visible_by_category
|
|
820
|
+
// and last_24h_visible_count).
|
|
821
|
+
const confirmed = {};
|
|
822
|
+
const visible = take("visible_by_category", countMap(field(stats, "visible_by_category")));
|
|
823
|
+
if (visible) {
|
|
824
|
+
confirmed.total = Object.values(visible).reduce((a, b) => a + b, 0);
|
|
825
|
+
confirmed.by_category = visible;
|
|
826
|
+
}
|
|
827
|
+
const last24Visible = take("last_24h_visible_count", numAt(stats, "last_24h_visible_count"));
|
|
828
|
+
if (last24Visible !== undefined)
|
|
829
|
+
confirmed.last_24h = last24Visible;
|
|
830
|
+
// Observed: every row of shadow_ct_observations, whatever its liveness.
|
|
831
|
+
const observed = {};
|
|
832
|
+
const total = take("total", numAt(stats, "total"));
|
|
833
|
+
if (total !== undefined)
|
|
834
|
+
observed.total = total;
|
|
835
|
+
const byCategory = take("by_category", countMap(field(stats, "by_category")));
|
|
836
|
+
if (byCategory)
|
|
837
|
+
observed.by_category = byCategory;
|
|
838
|
+
const last24 = take("last_24h_count", numAt(stats, "last_24h_count"));
|
|
839
|
+
if (last24 !== undefined)
|
|
840
|
+
observed.last_24h = last24;
|
|
841
|
+
const rows = {};
|
|
842
|
+
const rowExtras = [];
|
|
843
|
+
for (const [key, label] of SHADOW_AI_ROWS) {
|
|
844
|
+
const r = countRows(field(stats, key), label);
|
|
845
|
+
if (take(key, r.rows)) {
|
|
846
|
+
observed[key] = rows[key] = r.rows;
|
|
847
|
+
rowExtras.push(...r.extra.map((x) => `stats.${key}[].${x}`));
|
|
417
848
|
}
|
|
418
849
|
}
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
850
|
+
// A timestamp, not a count; one that is not a real instant is dropped (see realInstant).
|
|
851
|
+
const lastObservation = take("last_observation", strAt(stats, "last_observation"));
|
|
852
|
+
if (realInstant(lastObservation))
|
|
853
|
+
observed.last_observation = lastObservation;
|
|
854
|
+
// Authentication state: liveness 'authenticated' and 'inconclusive'.
|
|
855
|
+
const authentication = {};
|
|
856
|
+
const authSeen = take("auth_confirmed", numAt(stats, "auth_confirmed"));
|
|
857
|
+
if (authSeen !== undefined)
|
|
858
|
+
authentication.observed = authSeen;
|
|
859
|
+
const authUnknown = take("auth_undetermined", numAt(stats, "auth_undetermined"));
|
|
860
|
+
if (authUnknown !== undefined)
|
|
861
|
+
authentication.not_determined = authUnknown;
|
|
862
|
+
sentences.push(confirmed.total === undefined
|
|
863
|
+
? `Shadow AI: the answer carries no usable stats.visible_by_category, so it does not say how many services are confirmed exposed${last24Visible === undefined ? "" : `; confirmed_exposed.last_24h (${last24Visible}) counts confirmed-exposed services first recorded in the last 24 h`}.`
|
|
864
|
+
: `Shadow AI confirmed exposed: ${confirmed.total} (confirmed_exposed.total, the sum of confirmed_exposed.by_category: services EchelonGraph's probes found answering without an authentication gate, with liveness active or, during a re-check, rechecking)${last24Visible === undefined ? "" : `, ${last24Visible} of them first recorded in the last 24 h (confirmed_exposed.last_24h)`}. Of the shadow_ai numbers, only confirmed_exposed counts exposed services.`);
|
|
424
865
|
if (total !== undefined) {
|
|
425
|
-
sentences.push(`
|
|
866
|
+
sentences.push(`Shadow AI observed: ${total} (observed.total) counts every Certificate Transparency or Shodan observation on record, whatever its verification state: that many observed, not that many exposed.`);
|
|
426
867
|
}
|
|
427
|
-
|
|
868
|
+
const first = (key, label) => {
|
|
869
|
+
const top = rows[key]?.[0];
|
|
870
|
+
return top === undefined ? "" : ` (first: ${top[label]}, ${top.count} observations)`;
|
|
871
|
+
};
|
|
872
|
+
const observedParts = [
|
|
873
|
+
byCategory && "observed.by_category counts them by category",
|
|
874
|
+
last24 !== undefined && `observed.last_24h (${last24}) counts those first recorded in the last 24 h`,
|
|
875
|
+
rows.trend_30d && "observed.trend_30d counts them per UTC day over the last 30 days",
|
|
876
|
+
rows.top_products && `observed.top_products ranks up to ten products by observations${first("top_products", "product")}`,
|
|
877
|
+
rows.top_countries && `observed.top_countries ranks up to ten countries by observations${first("top_countries", "country")}`,
|
|
878
|
+
rows.top_issuers &&
|
|
879
|
+
`observed.top_issuers ranks up to ten issuers by observations, an issuer being the certificate's CA for a Certificate Transparency observation and the hosting operator Shodan reports for a Shodan one${first("top_issuers", "issuer")}`,
|
|
880
|
+
].filter((p) => typeof p === "string");
|
|
881
|
+
if (observedParts.length) {
|
|
882
|
+
sentences.push(`${total === undefined ? "Every number" : "Every other number"} under observed counts the same observations, not exposed services: ${observedParts.join("; ")}.`);
|
|
883
|
+
}
|
|
884
|
+
const authParts = [
|
|
885
|
+
authSeen !== undefined &&
|
|
886
|
+
`authentication.observed (${authSeen}) counts observations where a probe observed an authentication gate (a 401/403, a login page or an auth marker; liveness authenticated)`,
|
|
887
|
+
authUnknown !== undefined &&
|
|
888
|
+
`authentication.not_determined (${authUnknown}) counts observations whose service answered but where no probe could tell whether it enforces authentication (liveness inconclusive)`,
|
|
889
|
+
].filter((p) => typeof p === "string");
|
|
890
|
+
if (authParts.length) {
|
|
891
|
+
sentences.push(`Shadow AI authentication, over every observation on record: ${authParts.join("; ")}.`);
|
|
892
|
+
sentences.push("Neither authentication count is part of confirmed_exposed, and observed.total minus confirmed_exposed.total is not a count of secured services: it also holds observations not yet verified (liveness unverified), whose hostname no longer resolves (unreachable), or whose service no longer answers openly (resolved).");
|
|
893
|
+
}
|
|
894
|
+
// What the answer carried that the tool cannot label: named, and left out.
|
|
895
|
+
const unknown = [
|
|
896
|
+
...Object.keys(isPlainObject(sa) ? sa : {}).filter((k) => k !== "stats" && k !== "poller"),
|
|
897
|
+
...Object.keys(isPlainObject(stats) ? stats : {})
|
|
898
|
+
.filter((k) => !SHADOW_AI_STATS.has(k))
|
|
899
|
+
.map((k) => `stats.${k}`),
|
|
900
|
+
...rowExtras,
|
|
901
|
+
];
|
|
902
|
+
const unusable = Object.entries(isPlainObject(stats) ? stats : {})
|
|
903
|
+
.filter(([k, v]) => SHADOW_AI_STATS.has(k) && !used.has(k) && v !== null)
|
|
904
|
+
.map(([k]) => `stats.${k}`);
|
|
905
|
+
if (!isPlainObject(stats))
|
|
906
|
+
sentences.push("The answer carries no stats object, so it gives no shadow-AI counts.");
|
|
907
|
+
sentences.push(...leftOut("shadow_ai", unknown, unusable));
|
|
908
|
+
const data = {};
|
|
909
|
+
if (Object.keys(confirmed).length)
|
|
910
|
+
data.confirmed_exposed = confirmed;
|
|
911
|
+
if (Object.keys(observed).length)
|
|
912
|
+
data.observed = observed;
|
|
913
|
+
if (Object.keys(authentication).length)
|
|
914
|
+
data.authentication = authentication;
|
|
428
915
|
const lastRun = strAt(sa, "poller", "last_run_at");
|
|
429
916
|
const running = boolAt(sa, "poller", "running");
|
|
430
917
|
if (realInstant(lastRun) && running === true) {
|
|
918
|
+
data.poller = { running, last_run_at: lastRun };
|
|
431
919
|
sentences.push(`Shadow-AI radar freshness: the radar's leader last completed a Certificate Transparency (crt.sh) discovery cycle at ${lastRun} (poller.running is true: that was within ${SHADOW_AI_SILENT_AFTER} of the API's answer).`);
|
|
432
920
|
}
|
|
433
921
|
else if (realInstant(lastRun) && running === false) {
|
|
922
|
+
data.poller = { running, last_run_at: lastRun };
|
|
434
923
|
sentences.push(`Shadow-AI radar freshness: no Certificate Transparency (crt.sh) discovery cycle has completed since ${lastRun}, when the radar's leader last completed one (poller.running is false: that was more than ${SHADOW_AI_SILENT_AFTER} before the API's answer), so the shadow-AI figures may be stale.`);
|
|
435
924
|
}
|
|
436
925
|
else if (field(sa, "poller") === undefined) {
|
|
437
926
|
sentences.push("Shadow-AI radar freshness is unknown: the answer carries no poller block.");
|
|
438
927
|
}
|
|
439
928
|
else {
|
|
440
|
-
const rest = { ...sa };
|
|
441
|
-
delete rest.poller;
|
|
442
|
-
data = rest;
|
|
443
929
|
sentences.push("Shadow-AI radar freshness is unknown: the answer's poller block does not give both a real time at which the radar's leader last completed a cycle (poller.last_run_at) and a running verdict (poller.running), so that block was left out of the data above.");
|
|
444
930
|
}
|
|
445
|
-
const lastObservation = strAt(sa, "stats", "last_observation");
|
|
446
931
|
if (realInstant(lastObservation))
|
|
447
932
|
sentences.push(`Latest shadow-AI observation on record: ${lastObservation}.`);
|
|
448
933
|
return { data, sentences };
|
|
449
934
|
}
|
|
450
|
-
server.tool("exposure_radar", `Aggregate totals from EchelonGraph's internet-exposure radars, each refreshed on its own schedule:
|
|
935
|
+
server.tool("exposure_radar", `Aggregate totals from EchelonGraph's internet-exposure radars, each refreshed on its own schedule: internet-facing services running actively-exploited (CISA-KEV) CVEs, plus the ransomware-linked subset, derived from Shodan data; unauthenticated data stores and observability UIs, found through Shodan (LeakIX when Shodan query credits run low) and then confirmed by EchelonGraph's own identified check (not a pure read: on Redis it names its client, and on ClickHouse its query is recorded in the server's query log); leaked credentials sampled from public GitHub push events; and shadow AI services found through Certificate Transparency logs and Shodan and then checked by EchelonGraph's identified probes. Every number in the result is labelled here and in the result's note by what it counts; a field this version cannot label is left out and named in the note. ` +
|
|
936
|
+
`kev_exposure: kev_exposure.distinct_hosts counts distinct ip:port services, not machines (a machine answering on two ports counts twice), with at least one CISA-KEV-listed CVE on record, and kev_exposure.ransomware_hosts those with a ransomware-linked one; kev_exposure.kev_cves_exposed and kev_exposure.ransomware_cves count distinct CVEs with at least one such service; kev_exposure.correlations counts service×CVE pairs, not services, so a service with three KEV CVEs counts three times. kev_exposure.top_products, kev_exposure.top_countries and kev_exposure.top_cves rank up to 12 products, 10 countries and 12 CVEs by those services; kev_exposure.top_cves[].cvss_v3_score and kev_exposure.top_cves[].epss_score are the highest CVSS v3 base score and EPSS probability recorded on that CVE's observations. kev_exposure.trend counts service×CVE pairs still on record by the week in which each was first recorded, over 12 weeks, so earlier weeks read low. kev_exposure.newest_kev lists the 15 CVEs that EchelonGraph's CVE records most recently mark as CISA-KEV-listed, each with an exposure_state: exposed, where kev_exposure.newest_kev[].exposed_hosts counts distinct ip:port services on record with it; or not_assessed, where the API's answer holds no measurement for that CVE (its 0 is not one) and no count is relayed, and cve_exposure says per CVE whether the radar tracks it. kev_exposure.newest_kev[].cvss_v3_score and kev_exposure.newest_kev[].epss_score are the CVE record's CVSS v3 base score and EPSS probability. ` +
|
|
937
|
+
`exposed_databases: exposed_databases.distinct_hosts counts distinct ip:port services the check confirmed answering without authentication, and exposed_databases.engines the distinct engine types among them; exposed_databases.top_engines and exposed_databases.top_countries rank up to 15 engines and 10 countries by those services. exposed_databases.pii_likely and exposed_databases.pci_likely count services whose schema names (never record values) pass a high-confidence, precision-first gate for personal or payment-card data, so a service outside them is not shown to hold no such data. ` +
|
|
938
|
+
`leaked_credentials: leaked_credentials.total counts (repository, secret) pairs, not distinct secrets, so one secret in three repositories counts three times; leaked_credentials.distinct_secrets counts each secret once and leaked_credentials.distinct_repos counts repositories; leaked_credentials.top_providers and leaked_credentials.top_types rank up to 15 providers and secret types by those pairs. None is validated: each is a credential-shaped string that passed EchelonGraph's filters, at most structurally checked and never tested against its provider. Each of those three radars also carries generated_at, when the API computed its totals, and, when the API can tell, last_run_at. kev_exposure.last_run_at, exposed_databases.last_run_at and leaked_credentials.last_run_at are timestamps, not counts: each is when that radar last completed a check, a cycle whose reads succeeded, among them a Shodan search (for exposed_databases, or its LeakIX fallback) that answered at least one query, or for leaked_credentials a read of the public GitHub event stream. A cycle that read nothing does not move it, and it is not the time of every record a radar's numbers count, which cover everything still on record, not only what the last check found. A radar whose answer carries no last_run_at has none in the result, and the note says nothing about it. ` +
|
|
939
|
+
`shadow_ai: the result is regrouped by what each number counts. shadow_ai.confirmed_exposed counts services EchelonGraph's probes found answering without an authentication gate (liveness active, or rechecking during a re-check): confirmed_exposed.total is the sum of confirmed_exposed.by_category, and confirmed_exposed.last_24h counts those first recorded in the last 24 h. Of the shadow_ai numbers, only confirmed_exposed counts exposed services. shadow_ai.observed counts every Certificate Transparency or Shodan observation on record, whatever its verification state: its numbers are observed, not exposed. They are observed.total; observed.by_category (the same observations by category); observed.last_24h (those first recorded in the last 24 h); observed.trend_30d (observations per UTC day over the last 30 days); and observed.top_products, observed.top_countries and observed.top_issuers (up to ten products, countries and issuers ranked by observations, where an issuer is the certificate's CA for a Certificate Transparency observation and the hosting operator Shodan reports for a Shodan one). shadow_ai.authentication counts observations by probe outcome: authentication.observed where a probe observed an authentication gate (a 401/403, a login page or an auth marker), and authentication.not_determined where the service answered but no probe could tell. Neither authentication count is part of confirmed_exposed, and observed.total minus confirmed_exposed.total is not a count of secured services. The shadow-AI poller block carries only running and last_run_at: last_run_at is when the radar's leader last completed a Certificate Transparency (crt.sh) cycle, and its running is true only when that was within ${SHADOW_AI_SILENT_AFTER} of the answer. ${SHODAN_OWNERSHIP}`, {}, async () => {
|
|
451
940
|
const tool = "exposure_radar";
|
|
452
941
|
try {
|
|
453
942
|
const results = await Promise.all(RADARS.map(([, path]) => api(path)));
|
|
@@ -473,9 +962,20 @@ server.tool("exposure_radar", `Aggregate totals from EchelonGraph's internet-exp
|
|
|
473
962
|
isError: true,
|
|
474
963
|
};
|
|
475
964
|
}
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
965
|
+
// Each radar's answer is relayed cut to the fields its reader can label, in RADARS order.
|
|
966
|
+
const read = {
|
|
967
|
+
kev_exposure: readKEVExposure(data.kev_exposure),
|
|
968
|
+
exposed_databases: readExposedDatabases(data.exposed_databases),
|
|
969
|
+
leaked_credentials: readLeakedCredentials(data.leaked_credentials),
|
|
970
|
+
shadow_ai: readShadowAI(data.shadow_ai),
|
|
971
|
+
};
|
|
972
|
+
const relayed = {};
|
|
973
|
+
const sentences = [];
|
|
974
|
+
for (const [name] of RADARS) {
|
|
975
|
+
relayed[name] = read[name].data;
|
|
976
|
+
sentences.push(...read[name].sentences);
|
|
977
|
+
}
|
|
978
|
+
return succeeded(relayed, `${okHead(tool)} All ${RADARS.length} radars answered: ${RADARS.map(([name]) => name).join(", ")}. Every number relayed is labelled below by what it counts, and a field this version cannot label is left out and named. ${sentences.join(" ")} ${RADAR_SOURCES} ${SHODAN_OWNERSHIP} Each radar refreshes on its own schedule.`);
|
|
479
979
|
}
|
|
480
980
|
catch (e) {
|
|
481
981
|
return crashed(tool, e);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "echelongraph-mcp",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.4",
|
|
4
4
|
"mcpName": "io.echelongraph/echelongraph-mcp",
|
|
5
5
|
"description": "EchelonGraph CVE and internet-exposure data as an MCP server: CVEs with EchelonGraph multi-source scores, CISA-KEV and EPSS, plus a per-CVE internet-exposure footprint derived from Shodan data. Shodan data is owned by Shodan, which holds its copyright (© Shodan). Free and keyless, for Claude, Cursor and any MCP client.",
|
|
6
6
|
"type": "module",
|
|
@@ -36,7 +36,10 @@
|
|
|
36
36
|
"homepage": "https://echelongraph.io/pulse/mcp",
|
|
37
37
|
"repository": {
|
|
38
38
|
"type": "git",
|
|
39
|
-
"url": "git+https://github.com/echelongraph/
|
|
39
|
+
"url": "git+https://github.com/echelongraph/echelongraph-mcp.git"
|
|
40
|
+
},
|
|
41
|
+
"bugs": {
|
|
42
|
+
"url": "https://github.com/echelongraph/echelongraph-mcp/issues"
|
|
40
43
|
},
|
|
41
44
|
"engines": {
|
|
42
45
|
"node": ">=18"
|
package/server.json
CHANGED
|
@@ -4,12 +4,16 @@
|
|
|
4
4
|
"title": "EchelonGraph CVE & Exposure",
|
|
5
5
|
"description": "CVEs (NVD, CISA-KEV, EPSS, GHSA) and per-CVE exposure from Shodan data (© Shodan). Free, keyless.",
|
|
6
6
|
"websiteUrl": "https://echelongraph.io/pulse/mcp",
|
|
7
|
-
"
|
|
7
|
+
"repository": {
|
|
8
|
+
"url": "https://github.com/echelongraph/echelongraph-mcp",
|
|
9
|
+
"source": "github"
|
|
10
|
+
},
|
|
11
|
+
"version": "1.0.4",
|
|
8
12
|
"packages": [
|
|
9
13
|
{
|
|
10
14
|
"registryType": "npm",
|
|
11
15
|
"identifier": "echelongraph-mcp",
|
|
12
|
-
"version": "1.0.
|
|
16
|
+
"version": "1.0.4",
|
|
13
17
|
"transport": {
|
|
14
18
|
"type": "stdio"
|
|
15
19
|
},
|