@fffangchenghao/dsh-web-search 0.13.8 → 0.13.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/SECURITY.md +28 -13
- package/lib/fetch.js +1 -1
- package/lib/proxy.js +19 -3
- package/lib/render.d.ts +1 -1
- package/lib/render.js +7 -7
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,27 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.13.10
|
|
4
|
+
|
|
5
|
+
### Security
|
|
6
|
+
- The macOS system-proxy probe now spawns `/usr/sbin/scutil` by absolute path
|
|
7
|
+
instead of resolving `scutil` through `PATH`, so a hijacked `PATH` entry cannot
|
|
8
|
+
receive that spawn. This also makes the process list in `SECURITY.md` exact.
|
|
9
|
+
|
|
10
|
+
## 0.13.9
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
- **`SECURITY.md` wrongly claimed the plugin spawns nothing.** It does: on macOS
|
|
14
|
+
the system-proxy probe runs `scutil --proxy`. The page now lists that single
|
|
15
|
+
process, the renderer's headless browser, and the `shell` return field that
|
|
16
|
+
the store's scanner had flagged as a shell-mode call site.
|
|
17
|
+
- **The proxy probe had no deadline.** A wedged `scutil` could hold a search
|
|
18
|
+
open; it is now capped at 2 s and the child is killed on expiry.
|
|
19
|
+
- **The store's fixed-source scanner read an internal field name as a
|
|
20
|
+
capability.** `RenderReport.shell` (a boolean: "this page needs rendering") is
|
|
21
|
+
now `shellPage`, so `shell: true` no longer matches the scanner's shell-mode
|
|
22
|
+
pattern. Behaviour is unchanged; the store's capability list should now
|
|
23
|
+
contain only real signals.
|
|
24
|
+
|
|
3
25
|
## 0.13.8
|
|
4
26
|
|
|
5
27
|
### Fixed
|
package/SECURITY.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Security and permissions
|
|
2
2
|
|
|
3
|
-
This page states what the plugin can reach, what it
|
|
4
|
-
It is written for a reviewer deciding whether to install it, and
|
|
5
|
-
auditing the fixed commit that DSH STORE scans.
|
|
3
|
+
This page states what the plugin can reach, what it runs, what it writes, and
|
|
4
|
+
where it fails. It is written for a reviewer deciding whether to install it, and
|
|
5
|
+
for anyone auditing the fixed commit that DSH STORE scans.
|
|
6
6
|
|
|
7
7
|
## Dependencies
|
|
8
8
|
|
|
@@ -21,7 +21,7 @@ There are no lifecycle scripts (`preinstall`, `install`, `postinstall`,
|
|
|
21
21
|
|
|
22
22
|
## Network
|
|
23
23
|
|
|
24
|
-
Outbound HTTPS only
|
|
24
|
+
Outbound TCP/HTTPS only, plus one local system query described below.
|
|
25
25
|
|
|
26
26
|
| host | purpose |
|
|
27
27
|
|---|---|
|
|
@@ -29,16 +29,37 @@ Outbound HTTPS only. The default search set is:
|
|
|
29
29
|
| `html.duckduckgo.com` | key-free HTML search |
|
|
30
30
|
| `www.so.com` | key-free HTML search, fallback tier, usually bot-walled |
|
|
31
31
|
| `api.anysearch.com` | optional: search and page extraction, **only when an AnySearch key is configured** |
|
|
32
|
+
| your configured proxy | HTTP `CONNECT` tunnel for the hosts above |
|
|
32
33
|
|
|
33
34
|
`web_fetch` and `read_page` reach **any** HTTP(S) URL the model asks for; that is
|
|
34
35
|
their purpose. With `useProxy` enabled they speak to the configured proxy
|
|
35
|
-
(default `127.0.0.1:7890`, auto-detected) using HTTP `CONNECT
|
|
36
|
-
|
|
36
|
+
(default `127.0.0.1:7890`, auto-detected) using HTTP `CONNECT`; a 400 ms TCP
|
|
37
|
+
connect first checks that the proxy port is listening so a stale setting cannot
|
|
38
|
+
stall a search. The plugin never contacts an analytics, telemetry, or update
|
|
39
|
+
endpoint.
|
|
37
40
|
|
|
38
41
|
What leaves the machine: the query text goes to each enabled search engine; a
|
|
39
42
|
URL goes to AnySearch extraction only when a key is configured and the earlier
|
|
40
43
|
rungs failed. Nothing else is transmitted.
|
|
41
44
|
|
|
45
|
+
## Processes
|
|
46
|
+
|
|
47
|
+
The plugin spawns exactly one helper process:
|
|
48
|
+
|
|
49
|
+
- `/usr/sbin/scutil --proxy` — macOS only, when `useProxy` is on, no explicit
|
|
50
|
+
proxy is configured, and the system proxy has to be discovered. The argv is
|
|
51
|
+
fixed (no shell, no interpolation, no model- or user-controlled arguments),
|
|
52
|
+
stdout is parsed locally with fixed regexes, stderr is discarded, and the
|
|
53
|
+
probe is capped at 2 s before the child is killed. It reads nothing but the
|
|
54
|
+
system proxy settings.
|
|
55
|
+
|
|
56
|
+
The optional renderer is a **library**, not a command: when the user installed
|
|
57
|
+
`playwright` or `playwright-core` and enabled `renderJs`, the plugin imports it
|
|
58
|
+
dynamically and calls `chromium.launch({ headless: true })`, which starts a
|
|
59
|
+
headless browser process. That browser is the only third-party code the plugin
|
|
60
|
+
executes, and only under those two user choices. No shell command is ever
|
|
61
|
+
constructed from fetched content or model output.
|
|
62
|
+
|
|
42
63
|
## Credentials
|
|
43
64
|
|
|
44
65
|
The only credential is an optional AnySearch key. It lives where you put it —
|
|
@@ -57,13 +78,6 @@ credentials, and it writes no credential anywhere.
|
|
|
57
78
|
The cache holds queries and result rows only — no page bodies, no headers, no
|
|
58
79
|
cookies.
|
|
59
80
|
|
|
60
|
-
## Commands
|
|
61
|
-
|
|
62
|
-
The plugin executes **no** shell command and spawns **no** process. The optional
|
|
63
|
-
renderer launches a headless browser through the Playwright library the user
|
|
64
|
-
installed; that is the only way it runs third-party code, and only when the user
|
|
65
|
-
both installed Playwright and enabled `renderJs`.
|
|
66
|
-
|
|
67
81
|
## Failure bounds
|
|
68
82
|
|
|
69
83
|
| operation | bound |
|
|
@@ -74,6 +88,7 @@ both installed Playwright and enabled `renderJs`.
|
|
|
74
88
|
| response body | 2 MiB read cap before decoding, 20 000 characters returned by default |
|
|
75
89
|
| `read_page` | 2 000 000 characters per page, 8-page LRU, chunked reads |
|
|
76
90
|
| cache | TTL and entry count are bounded and both configurable, `0` disables |
|
|
91
|
+
| system proxy probe | 2 s, then the child is killed |
|
|
77
92
|
|
|
78
93
|
Failures are reported as structured codes (`SEARCH_THROTTLED`, `SEARCH_BLOCKED`,
|
|
79
94
|
`SEARCH_TIMEOUT`, `SEARCH_NETWORK`, `SEARCH_PARSE`, `SEARCH_EMPTY`,
|
package/lib/fetch.js
CHANGED
|
@@ -310,7 +310,7 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
310
310
|
}
|
|
311
311
|
if (candidate !== undefined && mode === 'auto') {
|
|
312
312
|
const verdict = detectShellPage(candidate);
|
|
313
|
-
if (!verdict.
|
|
313
|
+
if (!verdict.shellPage) {
|
|
314
314
|
return { done: true, outcome: candidate, report: makeReport(mode, false, false, verdict.reason) };
|
|
315
315
|
}
|
|
316
316
|
}
|
package/lib/proxy.js
CHANGED
|
@@ -189,12 +189,28 @@ export async function proxyFromSystem() {
|
|
|
189
189
|
return undefined;
|
|
190
190
|
const output = await new Promise((resolve) => {
|
|
191
191
|
let text = '';
|
|
192
|
-
|
|
192
|
+
let settled = false;
|
|
193
|
+
const finish = (value) => {
|
|
194
|
+
if (settled)
|
|
195
|
+
return;
|
|
196
|
+
settled = true;
|
|
197
|
+
clearTimeout(timer);
|
|
198
|
+
resolve(value);
|
|
199
|
+
};
|
|
200
|
+
// Absolute path: a hijacked PATH entry must not be able to receive this
|
|
201
|
+
// spawn. macOS ships scutil at /usr/sbin/scutil.
|
|
202
|
+
const child = spawn('/usr/sbin/scutil', ['--proxy'], { stdio: ['ignore', 'pipe', 'ignore'] });
|
|
203
|
+
// `scutil` answers in milliseconds; a wedged system call must not hold a
|
|
204
|
+
// search open, so the probe is capped and the child is killed on expiry.
|
|
205
|
+
const timer = setTimeout(() => {
|
|
206
|
+
child.kill();
|
|
207
|
+
finish('');
|
|
208
|
+
}, 2_000);
|
|
193
209
|
child.stdout.on('data', (chunk) => {
|
|
194
210
|
text += chunk.toString();
|
|
195
211
|
});
|
|
196
|
-
child.on('error', () =>
|
|
197
|
-
child.on('close', () =>
|
|
212
|
+
child.on('error', () => finish(''));
|
|
213
|
+
child.on('close', () => finish(text));
|
|
198
214
|
});
|
|
199
215
|
if (!/HTTPSEnable\s*:\s*1|HTTPEnable\s*:\s*1/.test(output))
|
|
200
216
|
return undefined;
|
package/lib/render.d.ts
CHANGED
|
@@ -50,7 +50,7 @@ export interface ShellInput {
|
|
|
50
50
|
/** A shell decision plus the numbers and reason behind it. */
|
|
51
51
|
export interface ShellVerdict {
|
|
52
52
|
/** Whether a browser render is worth attempting. */
|
|
53
|
-
readonly
|
|
53
|
+
readonly shellPage: boolean;
|
|
54
54
|
/** Characters of visible text, tags and script bodies removed. */
|
|
55
55
|
readonly readableChars: number;
|
|
56
56
|
/** Short, content-free explanation, safe to surface in a report. */
|
package/lib/render.js
CHANGED
|
@@ -94,27 +94,27 @@ export function readableTextLength(html) {
|
|
|
94
94
|
*/
|
|
95
95
|
export function detectShellPage(page) {
|
|
96
96
|
if (page.body.kind !== 'html') {
|
|
97
|
-
return {
|
|
97
|
+
return { shellPage: false, readableChars: 0, reason: 'the body is not HTML' };
|
|
98
98
|
}
|
|
99
99
|
const readable = readableTextLength(page.body.content);
|
|
100
100
|
const status = page.statusCode;
|
|
101
101
|
const renderableStatus = (status >= 200 && status < 300) || status === 403 || status === 429;
|
|
102
102
|
if (!renderableStatus) {
|
|
103
|
-
return {
|
|
103
|
+
return { shellPage: false, readableChars: readable, reason: `HTTP ${status} is an error response, not a page shell` };
|
|
104
104
|
}
|
|
105
105
|
if (readable === 0) {
|
|
106
|
-
return {
|
|
106
|
+
return { shellPage: true, readableChars: 0, reason: 'the HTML carries no readable text' };
|
|
107
107
|
}
|
|
108
108
|
if (status === 403 || status === 429) {
|
|
109
109
|
return {
|
|
110
|
-
|
|
110
|
+
shellPage: true,
|
|
111
111
|
readableChars: readable,
|
|
112
112
|
reason: `HTTP ${status} still answered with ${readable} readable characters`,
|
|
113
113
|
};
|
|
114
114
|
}
|
|
115
115
|
if (readable < SHELL_READABLE_LIMIT) {
|
|
116
116
|
return {
|
|
117
|
-
|
|
117
|
+
shellPage: true,
|
|
118
118
|
readableChars: readable,
|
|
119
119
|
reason: `only ${readable} readable characters, below the ${SHELL_READABLE_LIMIT}-character shell limit`,
|
|
120
120
|
};
|
|
@@ -124,13 +124,13 @@ export function detectShellPage(page) {
|
|
|
124
124
|
const marker = SHELL_MARKERS.find((candidate) => haystack.includes(candidate));
|
|
125
125
|
if (marker !== undefined) {
|
|
126
126
|
return {
|
|
127
|
-
|
|
127
|
+
shellPage: true,
|
|
128
128
|
readableChars: readable,
|
|
129
129
|
reason: `${readable} readable characters with the client-render marker ${marker}`,
|
|
130
130
|
};
|
|
131
131
|
}
|
|
132
132
|
}
|
|
133
|
-
return {
|
|
133
|
+
return { shellPage: false, readableChars: readable, reason: `${readable} readable characters look like real content` };
|
|
134
134
|
}
|
|
135
135
|
/**
|
|
136
136
|
* Scrub a diagnostic string before it leaves this module.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@fffangchenghao/dsh-web-search",
|
|
3
|
-
"version": "0.13.
|
|
3
|
+
"version": "0.13.10",
|
|
4
4
|
"description": "DSH bundle: a key-free web search provider. Leads with the anonymous AnySearch API and falls back to public result pages (DuckDuckGo, Bing, 360), merged by reciprocal-rank fusion with a bilingual region policy, an engine cooldown and a TTL cache.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "lib/index.js",
|