prism-mcp-server 20.3.0 → 20.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +75 -6
- package/dist/config.js +4 -0
- package/dist/scholar/freeSearch.js +237 -34
- package/dist/scholar/webScholar.js +30 -4
- package/docs/prism-browser.md +86 -2
- package/package.json +1 -1
- package/scripts/dev/browse.py +1729 -622
package/README.md
CHANGED
|
@@ -61,6 +61,75 @@ features.
|
|
|
61
61
|
<details>
|
|
62
62
|
<summary>Release history (optional)</summary>
|
|
63
63
|
|
|
64
|
+
## What's New in v20.3.2
|
|
65
|
+
|
|
66
|
+
### Web Scholar: SSRF Hardening
|
|
67
|
+
|
|
68
|
+
Security release. Web Scholar scrapes article URLs that come from
|
|
69
|
+
search-engine output, so the target is attacker-influenceable through SEO
|
|
70
|
+
poisoning — and because what it scrapes is written into the memory corpus and
|
|
71
|
+
passed to the configured LLM, a redirection to a local address meant reading an
|
|
72
|
+
internal service *and* sending the result onward.
|
|
73
|
+
|
|
74
|
+
The host guard matched string prefixes instead of parsing the address, and six
|
|
75
|
+
spellings of a local address got through: `[::1]` (`URL.hostname` keeps the
|
|
76
|
+
brackets), `127.0.0.2` (only `.1` was enumerated, not all of `127.0.0.0/8`),
|
|
77
|
+
`0.0.0.0`, `[::ffff:127.0.0.1]`, `localhost.` (a trailing dot defeated every
|
|
78
|
+
suffix check at once), and `[64:ff9b::7f00:1]` (NAT64 embeds IPv4 in its low
|
|
79
|
+
bits). Host classification now parses addresses and also covers CGNAT,
|
|
80
|
+
benchmarking, multicast, reserved, and IPv6 unique-local and link-local ranges.
|
|
81
|
+
|
|
82
|
+
DNS rebinding is closed too. Every check read the URL string, so a hostname the
|
|
83
|
+
attacker controls passed all of them and could still resolve to `127.0.0.1`.
|
|
84
|
+
Targets are now resolved first, every returned address is validated, and the
|
|
85
|
+
connection is pinned to those addresses so the name is never resolved a second
|
|
86
|
+
time — which also shuts the window between the check and the connect.
|
|
87
|
+
|
|
88
|
+
Scrape failures no longer vanish into a bare `catch {}`, a run is bounded by
|
|
89
|
+
`PRISM_SCHOLAR_SCRAPE_BUDGET_MS` (default 60s) instead of stalling on a raised
|
|
90
|
+
article count, and responses are capped at 8 MiB.
|
|
91
|
+
|
|
92
|
+
This is reachable only when scholar actually runs — `scholar_research`, or the
|
|
93
|
+
background loop under `PRISM_SCHOLAR_ENABLED=true` — and when the attacker also
|
|
94
|
+
controls DNS or a search result. Upgrade if you use Web Scholar.
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## What's New in v20.3.1
|
|
99
|
+
|
|
100
|
+
### Prism Browser Reports Real Failures
|
|
101
|
+
|
|
102
|
+
`prism browser` could not fail a test. `eval 1 === 2` returned `status: ok`
|
|
103
|
+
with exit code 0, a page serving HTTP 500 reported `status: ok`, and console
|
|
104
|
+
errors and uncaught page exceptions were discarded entirely. This release adds
|
|
105
|
+
assertions — `assert-text`, `assert-visible`, `assert-count`, `assert-url`,
|
|
106
|
+
`assert-title`, `assert-eval`, `assert-no-page-errors` — that return
|
|
107
|
+
`status: failed` and a non-zero exit. `open` now reports `http_status` and
|
|
108
|
+
fails on 400 or higher, screenshots are validated rather than assumed, and
|
|
109
|
+
`eval` returns native JSON with its type instead of a Python `repr`.
|
|
110
|
+
|
|
111
|
+
The fingerprint layer had never been applied: a wrong keyword argument made
|
|
112
|
+
the stealth library throw on every launch — 1,139 failures and 0 successes
|
|
113
|
+
since April — while the runner reported it as active. It is fixed, and a layer
|
|
114
|
+
that cannot be applied now fails loudly. The headless build no longer
|
|
115
|
+
advertises itself through `navigator.userAgentData` or the `Sec-CH-UA` header,
|
|
116
|
+
and a patch that corrupted `Object.getOwnPropertyDescriptor` on every page
|
|
117
|
+
under test has been removed. These remain best-effort test aids, not a
|
|
118
|
+
guarantee against bot detection.
|
|
119
|
+
|
|
120
|
+
`--local-only` now actually isolates: WebSocket, EventSource, WebRTC and
|
|
121
|
+
`sendBeacon` egress bypass request routing and were never blocked, and service
|
|
122
|
+
workers were allowed through. `--cleanup` was a no-op in the two modes agents
|
|
123
|
+
use. Site isolation, phishing detection and popup blocking are no longer
|
|
124
|
+
disabled by default, since these profiles hold live authenticated cookies.
|
|
125
|
+
|
|
126
|
+
New for test runs: `--ephemeral-profile` and `--storage-state` for hermetic
|
|
127
|
+
authenticated flows, `pages`/`switch-page` so OAuth popups are reachable,
|
|
128
|
+
`--fail-fast`, `--fast`, `--trace`/`--video`/`--har`, and
|
|
129
|
+
`profiles --prune-older-than` for profile maintenance.
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
64
133
|
## What's New in v20.2.7
|
|
65
134
|
|
|
66
135
|
### Session Saves Survive Agent Restarts
|
|
@@ -81,12 +150,12 @@ to the patched 8.5.23 release.
|
|
|
81
150
|
|
|
82
151
|
### Hybrid Memory Search (Portal Tier)
|
|
83
152
|
|
|
84
|
-
`session_search_memory` on Synalux-backed installs now
|
|
85
|
-
similarity with exact-term lexical matching
|
|
86
|
-
fusion
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
153
|
+
`session_search_memory` on the portal tier (Synalux-backed installs) now
|
|
154
|
+
fuses semantic similarity with exact-term lexical matching via weighted
|
|
155
|
+
reciprocal-rank fusion. Measured on blind probes against a real
|
|
156
|
+
8.5k-entry corpus: fused retrieval was **never worse** than semantic
|
|
157
|
+
alone at top-5, and exact identifiers — TPNs, function names, error
|
|
158
|
+
strings — now rescue queries that embedding similarity blurs. Results say how they were found — `hybrid retrieval`
|
|
90
159
|
headers, per-hit `sem#/lex#` arms — and a lexical-only rescue is labelled
|
|
91
160
|
`exact-term match` instead of pretending to a similarity score. Local
|
|
92
161
|
SQLite installs keep pure vector search; hybrid needs the portal's
|
package/dist/config.js
CHANGED
|
@@ -202,6 +202,10 @@ if (PRISM_SCHOLAR_ENABLED && !FIRECRAWL_API_KEY) {
|
|
|
202
202
|
export const PRISM_SCHOLAR_INTERVAL_MS = parseInt(process.env.PRISM_SCHOLAR_INTERVAL_MS || "0", 10 // Default manual-only
|
|
203
203
|
);
|
|
204
204
|
export const PRISM_SCHOLAR_MAX_ARTICLES_PER_RUN = parseInt(process.env.PRISM_SCHOLAR_MAX_ARTICLES_PER_RUN || "3", 10);
|
|
205
|
+
// Wall-clock ceiling for a run's scrape loop. Scrapes are sequential with a
|
|
206
|
+
// 15s per-fetch timeout, so without this a raised article count turns into a
|
|
207
|
+
// multi-minute stall.
|
|
208
|
+
export const PRISM_SCHOLAR_SCRAPE_BUDGET_MS = parseInt(process.env.PRISM_SCHOLAR_SCRAPE_BUDGET_MS || "60000", 10);
|
|
205
209
|
export const PRISM_SCHOLAR_TOPICS = (process.env.PRISM_SCHOLAR_TOPICS || "ai,agents")
|
|
206
210
|
.split(",")
|
|
207
211
|
.map(t => t.trim());
|
|
@@ -2,6 +2,12 @@ import * as cheerio from 'cheerio';
|
|
|
2
2
|
import { JSDOM } from 'jsdom';
|
|
3
3
|
import { Readability } from '@mozilla/readability';
|
|
4
4
|
import TurndownService from 'turndown';
|
|
5
|
+
import { lookup as dnsLookupCb } from 'node:dns';
|
|
6
|
+
import http from 'node:http';
|
|
7
|
+
import https from 'node:https';
|
|
8
|
+
import net from 'node:net';
|
|
9
|
+
import { promisify } from 'node:util';
|
|
10
|
+
const dnsLookup = promisify(dnsLookupCb);
|
|
5
11
|
/**
|
|
6
12
|
* Searches Yahoo Web Search and parses the HTML results using Cheerio.
|
|
7
13
|
* Yahoo provides a reliable HTML fallback that does not block basic automated browser requests.
|
|
@@ -42,47 +48,244 @@ export async function searchYahooFree(query, limit = 5) {
|
|
|
42
48
|
});
|
|
43
49
|
return results;
|
|
44
50
|
}
|
|
51
|
+
function classifyIPv4(octets) {
|
|
52
|
+
const [a, b] = octets;
|
|
53
|
+
if (a === 127)
|
|
54
|
+
return 'loopback'; // 127.0.0.0/8 — not just 127.0.0.1
|
|
55
|
+
if (a === 0)
|
|
56
|
+
return 'loopback'; // 0.0.0.0/8 routes to the local host
|
|
57
|
+
if (a === 10)
|
|
58
|
+
return 'private'; // RFC1918
|
|
59
|
+
if (a === 172 && b >= 16 && b <= 31)
|
|
60
|
+
return 'private';
|
|
61
|
+
if (a === 192 && b === 168)
|
|
62
|
+
return 'private';
|
|
63
|
+
if (a === 169 && b === 254)
|
|
64
|
+
return 'private'; // link-local + cloud metadata
|
|
65
|
+
if (a === 100 && b >= 64 && b <= 127)
|
|
66
|
+
return 'private'; // CGNAT
|
|
67
|
+
if (a === 192 && b === 0)
|
|
68
|
+
return 'private'; // IETF protocol assignments
|
|
69
|
+
if (a === 198 && (b === 18 || b === 19))
|
|
70
|
+
return 'private'; // benchmarking
|
|
71
|
+
if (a >= 224)
|
|
72
|
+
return 'private'; // multicast + reserved
|
|
73
|
+
return 'public';
|
|
74
|
+
}
|
|
75
|
+
function parseIPv4(host) {
|
|
76
|
+
const parts = host.split('.');
|
|
77
|
+
if (parts.length !== 4)
|
|
78
|
+
return null;
|
|
79
|
+
const octets = [];
|
|
80
|
+
for (const part of parts) {
|
|
81
|
+
if (!/^\d{1,3}$/.test(part))
|
|
82
|
+
return null;
|
|
83
|
+
const value = Number(part);
|
|
84
|
+
if (value > 255)
|
|
85
|
+
return null;
|
|
86
|
+
octets.push(value);
|
|
87
|
+
}
|
|
88
|
+
return octets;
|
|
89
|
+
}
|
|
45
90
|
/**
|
|
46
|
-
*
|
|
47
|
-
*
|
|
91
|
+
* Classify a URL's host.
|
|
92
|
+
*
|
|
93
|
+
* Exported so the bypass table can be asserted directly. String-prefix checks
|
|
94
|
+
* are not sufficient here: `URL.hostname` keeps the brackets on IPv6 literals,
|
|
95
|
+
* 127.0.0.0/8 is far wider than 127.0.0.1, and 0.0.0.0 and IPv4-mapped IPv6
|
|
96
|
+
* both reach the local host.
|
|
48
97
|
*/
|
|
49
|
-
export
|
|
50
|
-
//
|
|
51
|
-
//
|
|
52
|
-
//
|
|
53
|
-
|
|
54
|
-
|
|
98
|
+
export function classifyScrapeHost(rawHost) {
|
|
99
|
+
// URL.hostname returns IPv6 literals bracketed — strip the brackets. Node
|
|
100
|
+
// keeps a trailing '.' on named hosts (localhost. still resolves to
|
|
101
|
+
// 127.0.0.1), and that dot otherwise defeats every suffix check below.
|
|
102
|
+
const host = rawHost.toLowerCase()
|
|
103
|
+
.replace(/^\[|\]$/g, '')
|
|
104
|
+
.replace(/\.+$/, '');
|
|
105
|
+
if (host === 'localhost' || host.endsWith('.localhost'))
|
|
106
|
+
return 'loopback';
|
|
107
|
+
if (host.endsWith('.internal') || host.endsWith('.local'))
|
|
108
|
+
return 'private';
|
|
109
|
+
const ipv4 = parseIPv4(host);
|
|
110
|
+
if (ipv4)
|
|
111
|
+
return classifyIPv4(ipv4);
|
|
112
|
+
if (host.includes(':')) {
|
|
113
|
+
if (host === '::1')
|
|
114
|
+
return 'loopback';
|
|
115
|
+
if (host === '::')
|
|
116
|
+
return 'loopback';
|
|
117
|
+
// Prefixes that carry an IPv4 address in their low 32 bits. Node
|
|
118
|
+
// rewrites ::ffff:127.0.0.1 to ::ffff:7f00:1, so both the dotted and
|
|
119
|
+
// hex-pair spellings have to decode.
|
|
120
|
+
const embedding = /^(?:::ffff|64:ff9b(?::1)?::?)[:]?(.+)$/.exec(host);
|
|
121
|
+
if (embedding) {
|
|
122
|
+
const suffix = embedding[1];
|
|
123
|
+
const dotted = parseIPv4(suffix);
|
|
124
|
+
if (dotted)
|
|
125
|
+
return classifyIPv4(dotted);
|
|
126
|
+
const hexPair = /^([0-9a-f]{1,4}):([0-9a-f]{1,4})$/.exec(suffix);
|
|
127
|
+
if (hexPair) {
|
|
128
|
+
const high = parseInt(hexPair[1], 16);
|
|
129
|
+
const low = parseInt(hexPair[2], 16);
|
|
130
|
+
return classifyIPv4([high >> 8, high & 0xff, low >> 8, low & 0xff]);
|
|
131
|
+
}
|
|
132
|
+
// Reserved translation prefix with an undecodable tail — not public.
|
|
133
|
+
return 'private';
|
|
134
|
+
}
|
|
135
|
+
if (/^64:ff9b:/.test(host))
|
|
136
|
+
return 'private'; // NAT64 (RFC 6052/8215)
|
|
137
|
+
if (/^f[cd][0-9a-f]{2}:/.test(host))
|
|
138
|
+
return 'private'; // fc00::/7 unique-local
|
|
139
|
+
if (/^fe[89ab][0-9a-f]:/.test(host))
|
|
140
|
+
return 'private'; // fe80::/10 link-local
|
|
141
|
+
return 'public';
|
|
142
|
+
}
|
|
143
|
+
return 'public';
|
|
144
|
+
}
|
|
145
|
+
/**
|
|
146
|
+
* Reject scrape targets that point at the local host or an internal network.
|
|
147
|
+
*
|
|
148
|
+
* The URL reaching here comes from search-engine output, so it is
|
|
149
|
+
* attacker-influenceable via SEO poisoning. A successful fetch would be
|
|
150
|
+
* persisted into the memory corpus, so this fails closed.
|
|
151
|
+
*/
|
|
152
|
+
export function assertSafeScrapeTarget(url, devMode) {
|
|
153
|
+
let parsed;
|
|
154
|
+
try {
|
|
155
|
+
parsed = new URL(url);
|
|
156
|
+
}
|
|
157
|
+
catch {
|
|
158
|
+
throw new Error('Invalid URL: could not be parsed');
|
|
159
|
+
}
|
|
160
|
+
if (parsed.protocol !== 'https:' && parsed.protocol !== 'http:') {
|
|
161
|
+
throw new Error(`Invalid URL: protocol ${parsed.protocol} is not allowed`);
|
|
162
|
+
}
|
|
163
|
+
const hostClass = classifyScrapeHost(parsed.hostname);
|
|
164
|
+
if (hostClass === 'private') {
|
|
165
|
+
// Never allowed, even in dev mode: on a shared LAN these can reach
|
|
166
|
+
// other machines, and 169.254.169.254 is cloud metadata.
|
|
167
|
+
throw new Error('Invalid URL: private network URLs not allowed');
|
|
168
|
+
}
|
|
169
|
+
if (hostClass === 'loopback' && !devMode) {
|
|
170
|
+
throw new Error('Invalid URL: loopback URLs not allowed in production (set PRISM_DEV_MODE=1 to allow)');
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
/** Max article HTML accepted, so a hostile endpoint cannot stream us to death. */
|
|
174
|
+
const MAX_ARTICLE_BYTES = 8 * 1024 * 1024;
|
|
175
|
+
/**
|
|
176
|
+
* Resolve a hostname and validate every address it answers with.
|
|
177
|
+
*
|
|
178
|
+
* The string checks above only see the hostname, so a name the attacker
|
|
179
|
+
* controls can pass them and still resolve to 127.0.0.1 (DNS rebinding).
|
|
180
|
+
* Returns the address to connect to, which the caller pins so the name is
|
|
181
|
+
* never resolved a second time.
|
|
182
|
+
*/
|
|
183
|
+
export async function resolveAndValidateHost(hostname, devMode) {
|
|
184
|
+
const bracketless = hostname.replace(/^\[|\]$/g, '');
|
|
185
|
+
// A literal address needs no lookup; assertSafeScrapeTarget already ruled on it.
|
|
186
|
+
if (net.isIP(bracketless) !== 0) {
|
|
187
|
+
return [{ address: bracketless, family: net.isIP(bracketless) }];
|
|
188
|
+
}
|
|
189
|
+
let resolved;
|
|
55
190
|
try {
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
191
|
+
resolved = await dnsLookup(hostname, { all: true, verbatim: true });
|
|
192
|
+
}
|
|
193
|
+
catch (err) {
|
|
194
|
+
throw new Error(`Invalid URL: could not resolve ${hostname}`);
|
|
195
|
+
}
|
|
196
|
+
if (resolved.length === 0) {
|
|
197
|
+
throw new Error(`Invalid URL: ${hostname} resolved to no addresses`);
|
|
198
|
+
}
|
|
199
|
+
// Reject if ANY answer is non-public. A name that resolves to both a
|
|
200
|
+
// public and a private address is the rebinding shape itself, so picking
|
|
201
|
+
// the "good" one would just make the attack racy instead of blocked.
|
|
202
|
+
for (const entry of resolved) {
|
|
203
|
+
const hostClass = classifyScrapeHost(entry.address);
|
|
204
|
+
if (hostClass === 'private') {
|
|
205
|
+
throw new Error(`Invalid URL: ${hostname} resolves to a private address (${entry.address})`);
|
|
65
206
|
}
|
|
66
|
-
if (
|
|
67
|
-
|
|
68
|
-
// can cross into other tenants' machines on a shared LAN.
|
|
69
|
-
throw new Error('Private network URLs not allowed');
|
|
207
|
+
if (hostClass === 'loopback' && !devMode) {
|
|
208
|
+
throw new Error(`Invalid URL: ${hostname} resolves to a loopback address (${entry.address})`);
|
|
70
209
|
}
|
|
71
210
|
}
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
211
|
+
// Every answer is validated, so returning all of them keeps Node's
|
|
212
|
+
// dual-stack fallback working. Pinning only the first would fail whenever
|
|
213
|
+
// a host's leading AAAA is unreachable, which plain fetch handles.
|
|
214
|
+
return resolved;
|
|
215
|
+
}
|
|
216
|
+
/**
|
|
217
|
+
* GET an article over a connection pinned to a pre-validated address.
|
|
218
|
+
*
|
|
219
|
+
* Node's fetch would resolve the name again inside the request, which
|
|
220
|
+
* reopens the rebinding window between our check and the connection
|
|
221
|
+
* (TOCTOU). Supplying `lookup` guarantees the socket goes to the address we
|
|
222
|
+
* validated, while the URL still supplies the Host header and TLS SNI.
|
|
223
|
+
*/
|
|
224
|
+
function fetchPinned(url, addresses) {
|
|
225
|
+
return new Promise((resolve, reject) => {
|
|
226
|
+
const transport = url.protocol === 'https:' ? https : http;
|
|
227
|
+
const req = transport.request(url, {
|
|
228
|
+
headers: {
|
|
229
|
+
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
|
|
230
|
+
},
|
|
231
|
+
// Node calls this in two shapes: with { all: true } it wants an
|
|
232
|
+
// array of { address, family } (autoSelectFamily, the default
|
|
233
|
+
// since Node 20), otherwise the (err, address, family) form.
|
|
234
|
+
// Answering the wrong one yields "Invalid IP address: undefined".
|
|
235
|
+
lookup: (_hostname, options, callback) => {
|
|
236
|
+
if (options && options.all) {
|
|
237
|
+
callback(null, addresses);
|
|
238
|
+
}
|
|
239
|
+
else {
|
|
240
|
+
callback(null, addresses[0].address, addresses[0].family);
|
|
241
|
+
}
|
|
242
|
+
},
|
|
243
|
+
}, (res) => {
|
|
244
|
+
const status = res.statusCode ?? 0;
|
|
245
|
+
if (status >= 300 && status < 400) {
|
|
246
|
+
// Matches the previous redirect: 'error' behaviour — a
|
|
247
|
+
// redirect is a second target that was never validated.
|
|
248
|
+
res.destroy();
|
|
249
|
+
reject(new Error(`Failed to fetch article HTML: refusing redirect (${status})`));
|
|
250
|
+
return;
|
|
251
|
+
}
|
|
252
|
+
if (status < 200 || status >= 300) {
|
|
253
|
+
res.destroy();
|
|
254
|
+
reject(new Error(`Failed to fetch article HTML: ${status} ${res.statusMessage ?? ''}`.trim()));
|
|
255
|
+
return;
|
|
256
|
+
}
|
|
257
|
+
let size = 0;
|
|
258
|
+
const chunks = [];
|
|
259
|
+
res.on('data', (chunk) => {
|
|
260
|
+
size += chunk.length;
|
|
261
|
+
if (size > MAX_ARTICLE_BYTES) {
|
|
262
|
+
res.destroy();
|
|
263
|
+
reject(new Error(`Failed to fetch article HTML: exceeded ${MAX_ARTICLE_BYTES} bytes`));
|
|
264
|
+
return;
|
|
265
|
+
}
|
|
266
|
+
chunks.push(chunk);
|
|
267
|
+
});
|
|
268
|
+
res.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
|
|
269
|
+
res.on('error', reject);
|
|
270
|
+
});
|
|
271
|
+
req.setTimeout(15_000, () => {
|
|
272
|
+
req.destroy(new Error('Failed to fetch article HTML: timed out after 15000ms'));
|
|
273
|
+
});
|
|
274
|
+
req.on('error', reject);
|
|
275
|
+
req.end();
|
|
81
276
|
});
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
277
|
+
}
|
|
278
|
+
export async function scrapeArticleLocal(url) {
|
|
279
|
+
// SSRF protection: reject private/internal URLs.
|
|
280
|
+
// Set PRISM_DEV_MODE=1 to allow loopback hosts during local dev (testing
|
|
281
|
+
// against a local docs server, internal wiki, etc.). The flag is
|
|
282
|
+
// intentionally OFF in production deploys.
|
|
283
|
+
const devMode = process.env.PRISM_DEV_MODE === '1' || process.env.NODE_ENV === 'development';
|
|
284
|
+
assertSafeScrapeTarget(url, devMode);
|
|
285
|
+
// Then resolve, validate every answer, and pin the connection to it.
|
|
286
|
+
const parsed = new URL(url);
|
|
287
|
+
const addresses = await resolveAndValidateHost(parsed.hostname, devMode);
|
|
288
|
+
const html = await fetchPinned(parsed, addresses);
|
|
86
289
|
// Create a virtual DOM for Readability to traverse
|
|
87
290
|
const doc = new JSDOM(html, { url });
|
|
88
291
|
// Extract the article content like Firefox Reader View
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { BRAVE_API_KEY, FIRECRAWL_API_KEY, GOOGLE_SEARCH_API_KEY, GOOGLE_SEARCH_CX, SEMANTIC_SCHOLAR_API_KEY, PRISM_SCHOLAR_MAX_ARTICLES_PER_RUN, PRISM_USER_ID, PRISM_SCHOLAR_TOPICS, PRISM_ENABLE_HIVEMIND, } from "../config.js";
|
|
1
|
+
import { BRAVE_API_KEY, FIRECRAWL_API_KEY, GOOGLE_SEARCH_API_KEY, GOOGLE_SEARCH_CX, SEMANTIC_SCHOLAR_API_KEY, PRISM_SCHOLAR_MAX_ARTICLES_PER_RUN, PRISM_USER_ID, PRISM_SCHOLAR_TOPICS, PRISM_ENABLE_HIVEMIND, PRISM_SCHOLAR_SCRAPE_BUDGET_MS, } from "../config.js";
|
|
2
2
|
import { getStorage } from "../storage/index.js";
|
|
3
3
|
import { debugLog } from "../utils/logger.js";
|
|
4
4
|
import { getLLMProvider } from "../utils/llm/factory.js";
|
|
@@ -238,15 +238,41 @@ export async function runWebScholar(overrideTopic, overrideProject) {
|
|
|
238
238
|
return `No articles found for "${topic}"`;
|
|
239
239
|
await hivemindHeartbeat(`Scraping ${urls.length} articles on: ${topic}`);
|
|
240
240
|
const scrapedTexts = [];
|
|
241
|
+
const scrapeFailures = [];
|
|
242
|
+
// Scrapes run sequentially with a 15s per-fetch timeout, so N URLs cost
|
|
243
|
+
// up to N * 15s. PRISM_SCHOLAR_MAX_ARTICLES_PER_RUN is env-overridable,
|
|
244
|
+
// so bound the whole loop rather than trusting the item count.
|
|
245
|
+
// A non-finite budget (bad env value, or a test that stubs the config
|
|
246
|
+
// module wholesale) must not silently disable scraping or the budget.
|
|
247
|
+
const budgetMs = Number.isFinite(PRISM_SCHOLAR_SCRAPE_BUDGET_MS) && PRISM_SCHOLAR_SCRAPE_BUDGET_MS > 0
|
|
248
|
+
? PRISM_SCHOLAR_SCRAPE_BUDGET_MS
|
|
249
|
+
: 60_000;
|
|
250
|
+
const scrapeDeadline = Date.now() + budgetMs;
|
|
241
251
|
for (const url of urls) {
|
|
252
|
+
if (Date.now() >= scrapeDeadline) {
|
|
253
|
+
scrapeFailures.push(`${url} — skipped: ${budgetMs}ms scrape budget exhausted`);
|
|
254
|
+
continue;
|
|
255
|
+
}
|
|
242
256
|
try {
|
|
243
257
|
const article = await scrapeArticleLocal(url);
|
|
244
258
|
scrapedTexts.push(`Source: ${url}\nTitle: ${article.title}\n\n${article.content.slice(0, 15_000)}`);
|
|
245
259
|
}
|
|
246
|
-
catch {
|
|
260
|
+
catch (err) {
|
|
261
|
+
// A swallowed failure makes an empty run look identical to a good one.
|
|
262
|
+
const reason = err instanceof Error ? err.message : String(err);
|
|
263
|
+
scrapeFailures.push(`${url} — ${reason}`);
|
|
264
|
+
console.error(`[Scholar] scrape failed: ${url} — ${reason}`);
|
|
265
|
+
}
|
|
266
|
+
}
|
|
267
|
+
if (scrapedTexts.length === 0) {
|
|
268
|
+
const detail = scrapeFailures.length
|
|
269
|
+
? `\n${scrapeFailures.join('\n')}`
|
|
270
|
+
: ' (no URLs attempted)';
|
|
271
|
+
return `All ${urls.length} scrape(s) failed for "${topic}":${detail}`;
|
|
272
|
+
}
|
|
273
|
+
if (scrapeFailures.length > 0) {
|
|
274
|
+
console.error(`[Scholar] ${scrapeFailures.length}/${urls.length} scrapes failed for "${topic}"`);
|
|
247
275
|
}
|
|
248
|
-
if (scrapedTexts.length === 0)
|
|
249
|
-
return "All scrapes failed";
|
|
250
276
|
await hivemindHeartbeat(`Synthesizing ${scrapedTexts.length} articles on: ${topic}`);
|
|
251
277
|
const prompt = `You are an AI research assistant. Topic: "${topic}". Read these articles and write a comprehensive report.\n\n${scrapedTexts.join("\n---\n")}`;
|
|
252
278
|
const llm = getLLMProvider();
|
package/docs/prism-browser.md
CHANGED
|
@@ -18,16 +18,100 @@ so a separate app or DMG is not required.
|
|
|
18
18
|
scripts, allowing deterministic feature flags, fixtures, capability shims,
|
|
19
19
|
or instrumentation for localhost tests.
|
|
20
20
|
- **A constrained injection boundary.** Injection requires `--local-only`,
|
|
21
|
-
which rejects public navigation
|
|
21
|
+
which rejects public navigation, non-loopback subrequests, service workers,
|
|
22
|
+
and WebSocket/EventSource/WebRTC egress.
|
|
22
23
|
- **Private audit records.** The runner stores a local audit trail with private
|
|
23
24
|
filesystem permissions and removes URL credentials, query strings,
|
|
24
|
-
fragments, common email/phone patterns,
|
|
25
|
+
fragments, common email/phone patterns, record identifiers in URL paths, and
|
|
26
|
+
injected source text.
|
|
27
|
+
- **Failure signals instead of silence.** Console errors, uncaught page
|
|
28
|
+
exceptions, and failed requests are captured and attached to command output.
|
|
29
|
+
An HTTP status of 400 or higher, an empty screenshot frame, and a failed
|
|
30
|
+
assertion each produce a non-zero exit.
|
|
25
31
|
|
|
26
32
|
These are orchestration and safety benefits. Prism Browser does not replace
|
|
27
33
|
Playwright Test: use raw Playwright when you need its complete fixture,
|
|
28
34
|
assertion, trace, project, or parallel-worker APIs. Compatibility patches are
|
|
29
35
|
best effort and are not a CAPTCHA-bypass guarantee.
|
|
30
36
|
|
|
37
|
+
## Assertions and exit codes
|
|
38
|
+
|
|
39
|
+
A command reports `status: "ok"` only when it actually succeeded. Anything else
|
|
40
|
+
(`failed`, `error`, `timeout`) marks the run, and `pipe` exits non-zero.
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
printf '%s\n' \
|
|
44
|
+
'open http://127.0.0.1:3000' \
|
|
45
|
+
'assert-title Dashboard' \
|
|
46
|
+
'assert-text #status Ready' \
|
|
47
|
+
'assert-count [data-row] 12' \
|
|
48
|
+
'assert-no-page-errors' \
|
|
49
|
+
| prism browser --headless --fast --fail-fast pipe
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
`eval` returns the native JSON value with its type, so results are machine
|
|
53
|
+
readable:
|
|
54
|
+
|
|
55
|
+
```json
|
|
56
|
+
{"status": "ok", "result": {"a": 1}, "type": "dict", "serializable": true}
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
`assert-eval` is the assertion form — `eval` alone never fails a run, because a
|
|
60
|
+
falsy expression is a legitimate result. Use `--fail-fast` for test runs so a
|
|
61
|
+
failed navigation cannot be followed by commands that silently target the
|
|
62
|
+
previous page.
|
|
63
|
+
|
|
64
|
+
A trailing backslash continues a command onto the next line, which makes
|
|
65
|
+
multi-line JavaScript and multi-line input text expressible. Quoted arguments
|
|
66
|
+
are parsed with shell-style quoting, so `type "div > .cell" "two words"` works.
|
|
67
|
+
|
|
68
|
+
## Hermetic runs
|
|
69
|
+
|
|
70
|
+
Persistent profiles are convenient for interactive work and wrong for tests:
|
|
71
|
+
state carries between runs and makes them order dependent. For test runs use
|
|
72
|
+
|
|
73
|
+
- `--ephemeral-profile` — a throwaway profile directory, removed on exit.
|
|
74
|
+
- `--storage-state PATH` — seed cookies and localStorage from a Playwright
|
|
75
|
+
storage-state file, so authenticated flows skip the login UI.
|
|
76
|
+
- `save-storage PATH` — write the current state back out.
|
|
77
|
+
- `--fast` — skip the human-latency emulation (per-character typing and
|
|
78
|
+
inter-action pauses), which otherwise dominates a test's wall clock.
|
|
79
|
+
|
|
80
|
+
`--trace PATH`, `--video DIR`, and `--har PATH` record Playwright artifacts for
|
|
81
|
+
a failing run.
|
|
82
|
+
|
|
83
|
+
## Multiple pages
|
|
84
|
+
|
|
85
|
+
A popup (OAuth, payment, print preview) opens a new page. `pages` lists them,
|
|
86
|
+
`switch-page N` makes one active, and `close-page` disposes of it. Without
|
|
87
|
+
switching, commands continue to target the original page.
|
|
88
|
+
|
|
89
|
+
## Fingerprint patches
|
|
90
|
+
|
|
91
|
+
`--stealth full` applies playwright-stealth, a CDP
|
|
92
|
+
`Emulation.setUserAgentOverride` carrying full `userAgentMetadata`, and a
|
|
93
|
+
supplementary init script. The CDP override is what keeps `navigator.userAgent`,
|
|
94
|
+
`navigator.userAgentData`, and the `Sec-CH-UA` request headers consistent;
|
|
95
|
+
rewriting headers through request interception does not work, because Chromium
|
|
96
|
+
re-adds client hints after interception.
|
|
97
|
+
|
|
98
|
+
Applied layers are verified rather than assumed. The runner probes the page for
|
|
99
|
+
UA/brand/platform/webdriver agreement at startup and again after the first real
|
|
100
|
+
navigation, and `--stealth full` fails with an actionable error when a requested
|
|
101
|
+
layer cannot be applied. Pass `--allow-degraded-stealth` to proceed anyway, or
|
|
102
|
+
`--stealth light` to skip the library layer deliberately. The `fingerprint`
|
|
103
|
+
command reports the current state.
|
|
104
|
+
|
|
105
|
+
These patches do not disable site isolation, client-side phishing detection, or
|
|
106
|
+
popup blocking. Profiles hold live authenticated cookies, and trading away those
|
|
107
|
+
mitigations for a fingerprint delta is the wrong exchange.
|
|
108
|
+
|
|
109
|
+
## Profile maintenance
|
|
110
|
+
|
|
111
|
+
Persistent profiles accumulate. `prism browser profiles` lists them by size and
|
|
112
|
+
age; `--prune-older-than DAYS` reports what is stale and deletes it only when
|
|
113
|
+
`--yes` is also passed.
|
|
114
|
+
|
|
31
115
|
## Install the local runtime
|
|
32
116
|
|
|
33
117
|
```bash
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "prism-mcp-server",
|
|
3
|
-
"version": "20.3.
|
|
3
|
+
"version": "20.3.2",
|
|
4
4
|
"mcpName": "io.github.dcostenco/prism-coder",
|
|
5
5
|
"description": "Prism Coder — Cognitive memory + tool-calling intelligence for AI agents. Mind Palace persistent memory (BFCL Gold Certified, 100% Tool-Call Accuracy, 114 Agent Skills, PHI Guard, Tier Enforcement, Prompt-Based Skill Routing, Zero-Search HDC/HRR retrieval, HRR Semantic Drift Detection across BCBA/Coding/AAC domains, HIPAA-hardened local or subscription-gated Synalux storage, SLERP-optimized GRPO alignment) plus the prism-coder 1.7B–32B open-weights LLM fleet.",
|
|
6
6
|
"module": "index.ts",
|