dsh-web-fetch-playwright 0.2.20 → 0.2.23
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -17
- package/README.zh-CN.md +19 -17
- package/cordis.patch.yml +0 -4
- package/lib/client.js +18 -68
- package/lib/index.js +24049 -22754
- package/lib/types/autoscroll.d.ts +46 -10
- package/lib/types/candidates.d.ts +55 -0
- package/lib/types/challenge.d.ts +2 -2
- package/lib/types/client/controller.d.ts +0 -14
- package/lib/types/client/locales.d.ts +1 -1
- package/lib/types/config.d.ts +80 -30
- package/lib/types/index.d.ts +1 -1
- package/lib/types/provider.d.ts +101 -9
- package/lib/types/types.d.ts +26 -6
- package/package.json +1 -1
- package/src/autoscroll.ts +459 -17
- package/src/block-model.ts +1 -1
- package/src/candidates.ts +184 -35
- package/src/challenge.ts +2 -2
- package/src/client/card.tsx +0 -26
- package/src/client/controller.ts +0 -18
- package/src/client/locales.ts +0 -14
- package/src/config.ts +97 -37
- package/src/index.ts +4 -1
- package/src/markdown.ts +13 -2
- package/src/provider.ts +482 -56
- package/src/selector-model.ts +1 -1
- package/src/selector.ts +18 -7
- package/src/types.ts +27 -7
|
@@ -60,7 +60,8 @@
|
|
|
60
60
|
* collecting after each, until the walk can hold AT THE BOTTOM through a
|
|
61
61
|
* {@link SCROLL_BOTTOM_PATIENCE_MS} window with the height unchanged
|
|
62
62
|
* (growth restarts the clock — that is what pages load-more sentinels),
|
|
63
|
-
* or the step/budget caps close it.
|
|
63
|
+
* or the step/budget caps close it. A page that never reacted and issued
|
|
64
|
+
* no fetch/XHR holds only {@link SCROLL_INERT_PATIENCE_MS}.
|
|
64
65
|
*
|
|
65
66
|
* A MutationObserver watches for the walk's whole reason to exist: the
|
|
66
67
|
* page REACTING to scroll by adding or removing elements. Most of the web
|
|
@@ -71,6 +72,14 @@
|
|
|
71
72
|
* jumped over may hold content that never rendered, so the walk rewinds
|
|
72
73
|
* to where it jumped from and finishes carefully. Correctness never
|
|
73
74
|
* rests on the warm-up guess; only the cost of a scroll-inert page does.
|
|
75
|
+
*
|
|
76
|
+
* The same observer HARVESTS UNKEYED ROWS — the third shape: react-window
|
|
77
|
+
* lists and X-style timelines give rows no DOM identity at all (the
|
|
78
|
+
* framework key never reaches the DOM), and framework-generated ids
|
|
79
|
+
* (`ember123`, React useId's `:r1:`) change per render so they are not
|
|
80
|
+
* keys either. When the engine removes such a row, the record still
|
|
81
|
+
* holds it whole plus the sibling nodes it sat between; that is the row's
|
|
82
|
+
* stash and its splice anchor, with no layout read at all.
|
|
74
83
|
* 4. MERGE into a structural clone (`document.cloneNode(true)`: no
|
|
75
84
|
* serialize/re-parse round trip — a runtime-rewritten data script with a
|
|
76
85
|
* literal `</script>` in its text cannot corrupt it — and, having no
|
|
@@ -84,11 +93,21 @@
|
|
|
84
93
|
* spliced parent is already present and skipped. Placement is
|
|
85
94
|
* structural, so it is right whatever order the walk SAW rows in (an
|
|
86
95
|
* anchored open starts mid-document).
|
|
96
|
+
* - RESTORE (unkeyed rows): in every list the engine both removed text-
|
|
97
|
+
* bearing rows from and added to during the walk, each removed row the
|
|
98
|
+
* list no longer holds goes back beside the sibling it was removed
|
|
99
|
+
* next to (chains through removed anchors; rows with no surviving
|
|
100
|
+
* anchor at all — a wholesale innerHTML re-render — go above the final
|
|
101
|
+
* window, which is the only place a top-down walk loses them). Rows
|
|
102
|
+
* still present are recognised by a per-list identity attribute
|
|
103
|
+
* (data-index, aria-rowindex, …) or by text; in-place stubs are
|
|
104
|
+
* absorbed.
|
|
87
105
|
* - SWAP: every keyed element emptier than its snapshot (fewer child
|
|
88
106
|
* elements, or meaningfully less text — the Discourse placeholder) is
|
|
89
107
|
* replaced by that snapshot, except containers (ancestors of another
|
|
90
108
|
* element being swapped), which are assembled from those descendants
|
|
91
|
-
* instead: a container's own snapshot is one window's partial view.
|
|
109
|
+
* instead: a container's own snapshot is one window's partial view. A
|
|
110
|
+
* restored list and its ancestors are containers too.
|
|
92
111
|
* The clone is then stripped of scripting elements and serialized; the
|
|
93
112
|
* provider reads that string instead of the live page. The live document
|
|
94
113
|
* is never rewritten, so no late scroll/rAF/timer reaction can touch the
|
|
@@ -98,13 +117,16 @@
|
|
|
98
117
|
* error, or an engine that rejects the walk just keeps the legacy
|
|
99
118
|
* single-instant capture — a walk can only add content, never fail a fetch.
|
|
100
119
|
* A walk that restored nothing returns no HTML, and the provider reads the
|
|
101
|
-
* live page exactly as before
|
|
120
|
+
* live page exactly as before — scrolled back to where it opened when the
|
|
121
|
+
* page reacted, so a virtualizer the merge could not rebuild hands over its
|
|
122
|
+
* opening window rather than the walk's last one.
|
|
102
123
|
*
|
|
103
124
|
* The walk runs in BOTH denoise modes (the raw-HTML path serializes the same
|
|
104
125
|
* cloaked placeholders) but never for non-HTML documents. It is bounded by
|
|
105
126
|
* the `scrollBudgetMs` setting (`0` restores the pre-walk legacy behavior)
|
|
106
|
-
* and always by the remaining
|
|
107
|
-
*
|
|
127
|
+
* and always by the fetch's remaining waiting budget: a walk cut short there
|
|
128
|
+
* still merges and returns what it stashed, and the content read plus decode
|
|
129
|
+
* run in the provider's extraction grace window after it.
|
|
108
130
|
*
|
|
109
131
|
* @module dsh-web-fetch-playwright/autoscroll
|
|
110
132
|
*/
|
|
@@ -135,6 +157,16 @@ export declare const SCROLL_BOTTOM_POLL_MS = 450;
|
|
|
135
157
|
* two 450ms checks), so this is sized over a slow round trip.
|
|
136
158
|
*/
|
|
137
159
|
export declare const SCROLL_BOTTOM_PATIENCE_MS = 2500;
|
|
160
|
+
/**
|
|
161
|
+
* Bottom hold for a page that has given no sign of loading more: it never
|
|
162
|
+
* added or removed an element in response to the walk's scrolling, and it
|
|
163
|
+
* issued no fetch/XHR since the walk began. The full
|
|
164
|
+
* {@link SCROLL_BOTTOM_PATIENCE_MS} exists for feeds whose next chunk is in
|
|
165
|
+
* flight; holding it on every static article made the walk cost ~4s on
|
|
166
|
+
* pages it could never help. Any reaction or data request (even one that
|
|
167
|
+
* lands during this shorter hold) restores the full patience.
|
|
168
|
+
*/
|
|
169
|
+
export declare const SCROLL_INERT_PATIENCE_MS = 900;
|
|
138
170
|
/**
|
|
139
171
|
* Careful steps without any DOM reaction before the walk treats the page as
|
|
140
172
|
* scroll-inert and probes the rest with a single jump (rewinding if the
|
|
@@ -162,11 +194,6 @@ export declare const SCROLL_STASH_MAX_CHARS = 2000000;
|
|
|
162
194
|
* rehydrate. Such keys are remembered and never serialized again.
|
|
163
195
|
*/
|
|
164
196
|
export declare const SCROLL_STASH_MAX_ENTRY_CHARS = 64000;
|
|
165
|
-
/**
|
|
166
|
-
* Tail of the fetch budget the walk never spends, keeping room for the
|
|
167
|
-
* content read and the synchronous denoise decode.
|
|
168
|
-
*/
|
|
169
|
-
export declare const SCROLL_FINISH_RESERVE_MS = 4000;
|
|
170
197
|
/**
|
|
171
198
|
* A walk budget below this cannot complete even the first window plus one
|
|
172
199
|
* stability check, so the walk is skipped outright.
|
|
@@ -195,6 +222,14 @@ export interface DeepCaptureStats {
|
|
|
195
222
|
rehydrated: number;
|
|
196
223
|
/** Removed keyed elements spliced back (container walks only). */
|
|
197
224
|
spliced?: number;
|
|
225
|
+
/** Unkeyed rows a virtualizer removed, restored into their list. */
|
|
226
|
+
restored?: number;
|
|
227
|
+
/**
|
|
228
|
+
* True when nothing could be restored on a page that reacted, so the walk
|
|
229
|
+
* scrolled back to where the page opened before handing it to the
|
|
230
|
+
* provider's live read.
|
|
231
|
+
*/
|
|
232
|
+
rewound?: boolean;
|
|
198
233
|
/**
|
|
199
234
|
* Whether the page reacted to scrolling (added/removed elements). False
|
|
200
235
|
* means the walk took the scroll-inert fast path to the bottom.
|
|
@@ -224,6 +259,7 @@ export interface DeepCaptureParams {
|
|
|
224
259
|
stepSettleMs: number;
|
|
225
260
|
bottomPollMs: number;
|
|
226
261
|
bottomPatienceMs: number;
|
|
262
|
+
inertPatienceMs: number;
|
|
227
263
|
inertSteps: number;
|
|
228
264
|
maxSteps: number;
|
|
229
265
|
stashMaxEntries: number;
|
|
@@ -38,6 +38,15 @@ export interface RawCandidate {
|
|
|
38
38
|
kind: CandidateKind;
|
|
39
39
|
/** HTML of the extraction; sanitized by the caller. */
|
|
40
40
|
html: string;
|
|
41
|
+
/**
|
|
42
|
+
* The extraction as a live DOM node, when the generator already holds one
|
|
43
|
+
* (`doc`/`docc`/`ns_doc` serialize `body`, `main` the container itself).
|
|
44
|
+
* DOMPurify sanitizes a node by importing a copy — the caller's re-parse
|
|
45
|
+
* of `html` (hundreds of ms on large pages, measured on the Angular docs
|
|
46
|
+
* badcase: 555ms for the whole-body candidate alone) is skipped; `html`
|
|
47
|
+
* stays the identity for the dedup below.
|
|
48
|
+
*/
|
|
49
|
+
node?: Element;
|
|
41
50
|
/** Title to prepend when the body does not open with a heading. */
|
|
42
51
|
title?: string;
|
|
43
52
|
/** Whether the 0.2.18 pipeline would have returned exactly this candidate. */
|
|
@@ -109,6 +118,13 @@ export declare function childElements(element: Element): Element[];
|
|
|
109
118
|
export declare function unwrapNoscripts(document: Document): number;
|
|
110
119
|
/** Visible text inside `<noscript>` elements (parsed either way). */
|
|
111
120
|
export declare function noscriptTextLength(document: Document): number;
|
|
121
|
+
/**
|
|
122
|
+
* Cookie/consent banner wording (EN/DE/FR/ES/IT/NL/ZH), as a regex source
|
|
123
|
+
* shared with the selector's `consent_hits` feature so both read the same
|
|
124
|
+
* languages. Chinese banners say "Cookie" or 隐私; 同意 ("agree") alone is
|
|
125
|
+
* everyday prose and only counts as the banner's ACTION below.
|
|
126
|
+
*/
|
|
127
|
+
export declare const CONSENT_TERMS: string;
|
|
112
128
|
/**
|
|
113
129
|
* Strip page chrome from a cloned body: chrome-named/roled subtrees, hidden
|
|
114
130
|
* subtrees, and consent banners recognised by their wording. A subtree is
|
|
@@ -146,7 +162,46 @@ export interface GenerationFacts {
|
|
|
146
162
|
hasPreloaded: boolean;
|
|
147
163
|
/** The semantic main container, when found. */
|
|
148
164
|
main: Element | null;
|
|
165
|
+
/**
|
|
166
|
+
* True when the Readability family was SKIPPED because nearly all body
|
|
167
|
+
* text is custom-wrapped ({@link customWrappedTextShare}) — every
|
|
168
|
+
* extraction that would carry it sanitizes to nothing. The orchestrator
|
|
169
|
+
* then seeds the selector with the empty-`art` reference stats such a run
|
|
170
|
+
* would have measured, keeping the surviving candidates' features
|
|
171
|
+
* (`words_vs_art_log`, the block diffs) exactly what a dead Readability
|
|
172
|
+
* pick yields.
|
|
173
|
+
*/
|
|
174
|
+
readabilitySkipped?: boolean;
|
|
149
175
|
}
|
|
176
|
+
/**
|
|
177
|
+
* At or above this share of custom-wrapped body text the Readability family
|
|
178
|
+
* is skipped (see {@link customWrappedTextShare}).
|
|
179
|
+
*/
|
|
180
|
+
export declare const CUSTOM_WRAP_SKIP_SHARE = 0.9;
|
|
181
|
+
/**
|
|
182
|
+
* Share of the body's visible text that sits inside custom-element subtrees
|
|
183
|
+
* (tag names containing `-`: `<ms-app>`, `<my-doc>` — the HTML spec reserves
|
|
184
|
+
* them for custom elements; no standard HTML or MathML tag and no tag the
|
|
185
|
+
* denoise pipeline's DOMPurify step keeps contains a dash). DOMPurify's
|
|
186
|
+
* allowlist removes such tags wholesale — `KEEP_CONTENT: false` takes their
|
|
187
|
+
* subtrees with them — so once nearly ALL the page's text is custom-wrapped,
|
|
188
|
+
* every extraction that would carry it sanitizes to a corpse. Measured on
|
|
189
|
+
* the badcase (aistudio.google.com/docs/computer-use, an Angular app whose
|
|
190
|
+
* entire DOM lives under `<ms-app>`): 99% of body text custom-wrapped;
|
|
191
|
+
* Readability's pick sanitized to a root with ZERO words, the whole-document
|
|
192
|
+
* candidate to 39 of ~4.5k words — while the three Readability passes over
|
|
193
|
+
* the 1.1MB document cost ~6.5s of pure latency on candidates that could
|
|
194
|
+
* never win. The DOM-backed fallbacks, the noscript paths and the block
|
|
195
|
+
* classifier — the candidates that survive DOMPurify on this shape, and the
|
|
196
|
+
* ones the selector picks on such pages anyway — carry the extraction.
|
|
197
|
+
*
|
|
198
|
+
* Noise subtrees (script/style/nav/…) count as neither wrapped nor unwrapped;
|
|
199
|
+
* the share is over CONTENT-bearing text only. A partial share (a web
|
|
200
|
+
* component widget inside a normal article) stays below the threshold and
|
|
201
|
+
* keeps the full pipeline: only the shape where the family is provably dead
|
|
202
|
+
* pays for it.
|
|
203
|
+
*/
|
|
204
|
+
export declare function customWrappedTextShare(body: Element): number;
|
|
150
205
|
/**
|
|
151
206
|
* Propose every candidate extraction for one parsed page.
|
|
152
207
|
* @param document - the parsed page (data URIs already elided); not mutated.
|
package/lib/types/challenge.d.ts
CHANGED
|
@@ -41,8 +41,8 @@
|
|
|
41
41
|
export declare const CHALLENGE_POLL_INTERVAL_MS = 500;
|
|
42
42
|
/**
|
|
43
43
|
* Tail of the fetch budget held back from the challenge wait so a cleared
|
|
44
|
-
* challenge still has time for its settle (
|
|
45
|
-
* content read and the denoise pipeline.
|
|
44
|
+
* challenge still has time for its settle (the `DEFAULT_SETTLE_MS` cap)
|
|
45
|
+
* plus the content read and the denoise pipeline.
|
|
46
46
|
*/
|
|
47
47
|
export declare const CHALLENGE_FINISH_RESERVE_MS = 8000;
|
|
48
48
|
/**
|
|
@@ -37,16 +37,6 @@ export interface PlaywrightSettings {
|
|
|
37
37
|
denoise?: boolean;
|
|
38
38
|
/** How many fetches may render at once (1–200); blank = backend default. */
|
|
39
39
|
maxConcurrency?: number;
|
|
40
|
-
/**
|
|
41
|
-
* Bounded wait (ms) for a Cloudflare challenge to clear naturally in the
|
|
42
|
-
* same tab (0–60000); 0 = off (return the first response as-is).
|
|
43
|
-
*/
|
|
44
|
-
challengeWaitMs?: number;
|
|
45
|
-
/**
|
|
46
|
-
* Bounded budget (ms) for the deep-capture scroll walk (0–30000); 0 = off
|
|
47
|
-
* (capture the document as it rests, the legacy single-instant behavior).
|
|
48
|
-
*/
|
|
49
|
-
scrollBudgetMs?: number;
|
|
50
40
|
}
|
|
51
41
|
/** What the Playwright card renders. */
|
|
52
42
|
export interface PlaywrightCardState extends CardShell {
|
|
@@ -63,10 +53,6 @@ export interface PlaywrightCardState extends CardShell {
|
|
|
63
53
|
denoise: CardFieldState;
|
|
64
54
|
/** Concurrency input (draft decimal integer). */
|
|
65
55
|
maxConcurrency: CardFieldState;
|
|
66
|
-
/** Challenge wait input (draft decimal integer of milliseconds). */
|
|
67
|
-
challengeWaitMs: CardFieldState;
|
|
68
|
-
/** Scroll-walk budget input (draft decimal integer of milliseconds). */
|
|
69
|
-
scrollBudgetMs: CardFieldState;
|
|
70
56
|
}
|
|
71
57
|
/** The registration-side face the card's slot entry injects. */
|
|
72
58
|
export interface PlaywrightCardFace extends CardActions {
|
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
* @module dsh-web-fetch-playwright/client/locales
|
|
6
6
|
*/
|
|
7
7
|
/** Locale keys this card renders. */
|
|
8
|
-
export type PlaywrightCardLocaleKey = 'description' | 'backendLabel' | 'backendLocal' | 'backendLocalHint' | 'backendCdp' | 'backendCdpHint' | 'playwrightPath' | 'playwrightPathHint' | 'playwrightPathPlaceholder' | 'cdpEndpoint' | 'cdpEndpointHint' | 'shareBrowserContext' | 'shareBrowserContextHint' | 'contextMode' | 'isolateBrowserContext' | 'isolateBrowserContextHint' | 'contextIsolationScope' | 'contextScopeSession' | 'contextScopeSessionHint' | 'contextScopeGlobal' | 'contextScopeGlobalHint' | 'denoise' | 'denoiseHint' | 'maxConcurrency' | 'maxConcurrencyHint' | 'maxConcurrencyPlaceholder' | '
|
|
8
|
+
export type PlaywrightCardLocaleKey = 'description' | 'backendLabel' | 'backendLocal' | 'backendLocalHint' | 'backendCdp' | 'backendCdpHint' | 'playwrightPath' | 'playwrightPathHint' | 'playwrightPathPlaceholder' | 'cdpEndpoint' | 'cdpEndpointHint' | 'shareBrowserContext' | 'shareBrowserContextHint' | 'contextMode' | 'isolateBrowserContext' | 'isolateBrowserContextHint' | 'contextIsolationScope' | 'contextScopeSession' | 'contextScopeSessionHint' | 'contextScopeGlobal' | 'contextScopeGlobalHint' | 'denoise' | 'denoiseHint' | 'maxConcurrency' | 'maxConcurrencyHint' | 'maxConcurrencyPlaceholder' | 'overridden' | 'reset' | 'readOnly' | 'save' | 'saving' | 'discard' | 'saveFailed' | 'invalidText';
|
|
9
9
|
/** This plugin's dictionary namespace, merged into the locale key map. */
|
|
10
10
|
declare module '@deepseek-ai/dsh-client-ui-slots' {
|
|
11
11
|
interface LocaleNamespaceMap {
|
package/lib/types/config.d.ts
CHANGED
|
@@ -15,16 +15,24 @@
|
|
|
15
15
|
import type { Volatile } from '@deepseek-ai/cordis';
|
|
16
16
|
import z from '@deepseek-ai/schemastery';
|
|
17
17
|
/**
|
|
18
|
-
* Default
|
|
19
|
-
*
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
*
|
|
18
|
+
* Default CAP (ms) on the deep-capture scroll walk (`src/autoscroll.ts`).
|
|
19
|
+
* It bounds only pages that keep reacting to scrolling — virtualized feeds
|
|
20
|
+
* and load-more streams — where 20s covers ~25kpx at the careful cadence
|
|
21
|
+
* (`SCROLL_STEP_MOTION_MS` + `SCROLL_STEP_SETTLE_MS` per ~0.7-viewport
|
|
22
|
+
* step). It is not what a typical page pays: a page that never reacts takes
|
|
23
|
+
* `SCROLL_INERT_STEPS` careful steps, one jump to the bottom and a
|
|
24
|
+
* `SCROLL_INERT_PATIENCE_MS` hold, and a page that does not scroll is
|
|
25
|
+
* skipped outright.
|
|
24
26
|
*/
|
|
25
27
|
export declare const DEFAULT_SCROLL_BUDGET_MS = 20000;
|
|
26
|
-
/**
|
|
27
|
-
|
|
28
|
+
/**
|
|
29
|
+
* Default cap (ms) on the post-navigation settle — the wait for SPA
|
|
30
|
+
* rendering to finish once the document has loaded (see the settle in
|
|
31
|
+
* `src/provider.ts`). A CAP, not a fixed cost: the settle ends as soon as
|
|
32
|
+
* the page's network goes quiet. Code-level only (tests, demo scripts):
|
|
33
|
+
* not part of the settings schema. `0` disables the settle entirely.
|
|
34
|
+
*/
|
|
35
|
+
export declare const DEFAULT_SETTLE_MS = 5000;
|
|
28
36
|
/** Default CDP endpoint when the settings section leaves it blank. */
|
|
29
37
|
export declare const DEFAULT_CDP_ENDPOINT = "127.0.0.1:9222";
|
|
30
38
|
/**
|
|
@@ -45,16 +53,24 @@ export declare const MAX_CONCURRENCY_CEILING = 200;
|
|
|
45
53
|
/**
|
|
46
54
|
* Default bounded wait (ms) for a Cloudflare challenge to clear naturally —
|
|
47
55
|
* the user's real browser passes the verification on its own while the fetch
|
|
48
|
-
* holds the same page and context.
|
|
49
|
-
*
|
|
56
|
+
* holds the same page and context. Bounded further by the 25s per-fetch
|
|
57
|
+
* waiting budget, which keeps a settle/decode tail after the wait.
|
|
50
58
|
*/
|
|
51
59
|
export declare const DEFAULT_CHALLENGE_WAIT_MS = 15000;
|
|
52
|
-
/** Ceiling the schema accepts for `challengeWaitMs`. */
|
|
53
|
-
export declare const MAX_CHALLENGE_WAIT_MS = 60000;
|
|
54
60
|
/** Default same-page re-navigation attempts after a challenge wait runs out. */
|
|
55
61
|
export declare const DEFAULT_CHALLENGE_RETRIES = 1;
|
|
56
62
|
/** Ceiling the schema accepts for `challengeRetries`. */
|
|
57
63
|
export declare const MAX_CHALLENGE_RETRIES = 3;
|
|
64
|
+
/**
|
|
65
|
+
* Default for `challengeHttpFallback`: when the browser cannot clear a
|
|
66
|
+
* challenge inside its window (the interactive checkbox was never offered,
|
|
67
|
+
* or verification refused), refetch the URL once through the page's
|
|
68
|
+
* plain-HTTP request context — the same cookie jar, no render decision —
|
|
69
|
+
* because many Cloudflare edges serve the real document to non-browser
|
|
70
|
+
* clients while withholding the interactive challenge from the browser
|
|
71
|
+
* session (measured live: v2ex.com from a datacenter IP).
|
|
72
|
+
*/
|
|
73
|
+
export declare const DEFAULT_CHALLENGE_HTTP_FALLBACK = true;
|
|
58
74
|
/** Which browser backend serves a fetch. */
|
|
59
75
|
export type PlaywrightBackend = 'local' | 'cdp';
|
|
60
76
|
/**
|
|
@@ -98,7 +114,9 @@ export interface Config {
|
|
|
98
114
|
* Bounded wait (ms) for a Cloudflare challenge to clear naturally inside
|
|
99
115
|
* the same page/context. `0` (or any config leaving this at 0) restores the
|
|
100
116
|
* legacy behavior: the first response is final, no waiting — the A/B
|
|
101
|
-
* baseline.
|
|
117
|
+
* baseline. Code-level only (tests, demo scripts): not part of the
|
|
118
|
+
* settings schema, so users cannot configure it. Default:
|
|
119
|
+
* {@link DEFAULT_CHALLENGE_WAIT_MS}.
|
|
102
120
|
*/
|
|
103
121
|
challengeWaitMs?: number;
|
|
104
122
|
/**
|
|
@@ -107,15 +125,32 @@ export interface Config {
|
|
|
107
125
|
* Schema default: {@link DEFAULT_CHALLENGE_RETRIES}.
|
|
108
126
|
*/
|
|
109
127
|
challengeRetries?: number;
|
|
128
|
+
/**
|
|
129
|
+
* When the browser cannot clear a challenge inside its window (no
|
|
130
|
+
* checkbox offered, or verification refused), refetch the URL once
|
|
131
|
+
* through the page's plain-HTTP request context (same cookie jar) and
|
|
132
|
+
* return that document when it is not itself a challenge. `false`
|
|
133
|
+
* restores the browser-only posture.
|
|
134
|
+
*/
|
|
135
|
+
challengeHttpFallback?: boolean;
|
|
110
136
|
/**
|
|
111
137
|
* Bounded budget (ms) for the deep-capture scroll walk — the in-page pass
|
|
112
138
|
* that defeats viewport virtualization on feed-shaped pages (Discourse
|
|
113
139
|
* topic streams: replies outside the render window are empty height-only
|
|
114
140
|
* placeholders carrying the post's id; see `src/autoscroll.ts`). `0`
|
|
115
|
-
* restores the legacy single-instant capture.
|
|
116
|
-
*
|
|
141
|
+
* restores the legacy single-instant capture. Code-level only (tests,
|
|
142
|
+
* demo scripts): not part of the settings schema, so users cannot
|
|
143
|
+
* configure it. Default: {@link DEFAULT_SCROLL_BUDGET_MS}.
|
|
117
144
|
*/
|
|
118
145
|
scrollBudgetMs?: number;
|
|
146
|
+
/**
|
|
147
|
+
* Cap (ms) on the post-navigation settle (load baseline + bounded
|
|
148
|
+
* network-quiet window with stale-request discount). Code-level only
|
|
149
|
+
* (tests, demo scripts): not part of the settings schema, so users cannot
|
|
150
|
+
* configure it. `0` disables the settle. Default:
|
|
151
|
+
* {@link DEFAULT_SETTLE_MS}.
|
|
152
|
+
*/
|
|
153
|
+
settleMs?: number;
|
|
119
154
|
/**
|
|
120
155
|
* How many fetches may render at once. Blank = backend default (4 for
|
|
121
156
|
* local — each slot launches a browser; 50 for CDP — each slot is a tab in
|
|
@@ -131,9 +166,8 @@ export declare const Config: z<Schemastery.ObjectS<NoInfer<{
|
|
|
131
166
|
contextIsolationScope: z<"session" | "global", "session" | "global", "volatile-defined">;
|
|
132
167
|
denoise: z<boolean, boolean, "volatile-defined">;
|
|
133
168
|
maxConcurrency: z<number, number, "volatile">;
|
|
134
|
-
challengeWaitMs: z<number, number, "volatile-defined">;
|
|
135
169
|
challengeRetries: z<number, number, "volatile-defined">;
|
|
136
|
-
|
|
170
|
+
challengeHttpFallback: z<boolean, boolean, "volatile-defined">;
|
|
137
171
|
}>>, Schemastery.ObjectT<NoInfer<{
|
|
138
172
|
backend: z<"local" | "cdp", "local" | "cdp", "volatile-defined">;
|
|
139
173
|
playwrightPath: z<string, string, "volatile-defined">;
|
|
@@ -142,9 +176,8 @@ export declare const Config: z<Schemastery.ObjectS<NoInfer<{
|
|
|
142
176
|
contextIsolationScope: z<"session" | "global", "session" | "global", "volatile-defined">;
|
|
143
177
|
denoise: z<boolean, boolean, "volatile-defined">;
|
|
144
178
|
maxConcurrency: z<number, number, "volatile">;
|
|
145
|
-
challengeWaitMs: z<number, number, "volatile-defined">;
|
|
146
179
|
challengeRetries: z<number, number, "volatile-defined">;
|
|
147
|
-
|
|
180
|
+
challengeHttpFallback: z<boolean, boolean, "volatile-defined">;
|
|
148
181
|
}>>, "plain">;
|
|
149
182
|
/**
|
|
150
183
|
* The config `apply` receives once the schema resolves: every volatile field
|
|
@@ -169,12 +202,10 @@ export interface PlaywrightPluginConfig {
|
|
|
169
202
|
* field owns no default — the backend resolves one).
|
|
170
203
|
*/
|
|
171
204
|
maxConcurrency: Volatile<number | undefined>;
|
|
172
|
-
/** Challenge wait budget (ms) as a live reference. */
|
|
173
|
-
challengeWaitMs: Volatile<number>;
|
|
174
205
|
/** Challenge retry count as a live reference. */
|
|
175
206
|
challengeRetries: Volatile<number>;
|
|
176
|
-
/**
|
|
177
|
-
|
|
207
|
+
/** Challenge dead-end plain-HTTP refetch toggle as a live reference. */
|
|
208
|
+
challengeHttpFallback: Volatile<boolean>;
|
|
178
209
|
}
|
|
179
210
|
/**
|
|
180
211
|
* Read the live references `apply` holds into one plain input for
|
|
@@ -198,14 +229,16 @@ export declare function snapshotsOf(config: PlaywrightPluginConfig): Config;
|
|
|
198
229
|
export declare function resolveConfig(config: Partial<Config>): ResolvedConfig;
|
|
199
230
|
/**
|
|
200
231
|
* Complete config after schemastery applies the field defaults it owns.
|
|
201
|
-
* `maxConcurrency`
|
|
202
|
-
*
|
|
203
|
-
*
|
|
204
|
-
*
|
|
232
|
+
* `maxConcurrency` stays optional because its effective default depends on
|
|
233
|
+
* `backend`; `challengeWaitMs`, `scrollBudgetMs` and `settleMs` stay
|
|
234
|
+
* optional because they are code-level overrides outside the schema —
|
|
235
|
+
* absent, the effective* helpers supply the built-in defaults.
|
|
205
236
|
*/
|
|
206
|
-
export type ResolvedConfig = Omit<Required<Config>, 'maxConcurrency' | 'scrollBudgetMs'> & {
|
|
237
|
+
export type ResolvedConfig = Omit<Required<Config>, 'maxConcurrency' | 'challengeWaitMs' | 'scrollBudgetMs' | 'settleMs'> & {
|
|
207
238
|
maxConcurrency?: number;
|
|
239
|
+
challengeWaitMs?: number;
|
|
208
240
|
scrollBudgetMs?: number;
|
|
241
|
+
settleMs?: number;
|
|
209
242
|
};
|
|
210
243
|
/**
|
|
211
244
|
* The concurrency limit a fetch actually runs with: an explicit setting
|
|
@@ -218,7 +251,7 @@ export type ResolvedConfig = Omit<Required<Config>, 'maxConcurrency' | 'scrollBu
|
|
|
218
251
|
export declare function effectiveMaxConcurrency(config: Pick<Config, 'backend' | 'maxConcurrency'>): number;
|
|
219
252
|
/**
|
|
220
253
|
* The challenge wait budget a fetch actually runs with: an explicit setting
|
|
221
|
-
* wins, else the
|
|
254
|
+
* wins, else the built-in default. `0` means the challenge path is off.
|
|
222
255
|
*
|
|
223
256
|
* @param config - the resolved settings section (or any partial of it).
|
|
224
257
|
* @returns the bounded wait in milliseconds.
|
|
@@ -232,14 +265,31 @@ export declare function effectiveChallengeWaitMs(config: Pick<Config, 'challenge
|
|
|
232
265
|
* @returns retries after a wait window runs out.
|
|
233
266
|
*/
|
|
234
267
|
export declare function effectiveChallengeRetries(config: Pick<Config, 'challengeRetries'>): number;
|
|
268
|
+
/**
|
|
269
|
+
* Whether a challenge dead-end may refetch through plain HTTP: an explicit
|
|
270
|
+
* setting wins (absent = on, the default).
|
|
271
|
+
*
|
|
272
|
+
* @param config - the resolved settings section (or any partial of it).
|
|
273
|
+
* @returns true when the plain-HTTP fallback may run.
|
|
274
|
+
*/
|
|
275
|
+
export declare function effectiveChallengeHttpFallback(config: Pick<Config, 'challengeHttpFallback'>): boolean;
|
|
235
276
|
/**
|
|
236
277
|
* The scroll-walk budget a fetch actually runs with: an explicit setting
|
|
237
|
-
* wins, else the
|
|
278
|
+
* wins, else the built-in default. `0` means the deep-capture walk is off.
|
|
238
279
|
*
|
|
239
280
|
* @param config - the resolved settings section (or any partial of it).
|
|
240
281
|
* @returns the bounded walk budget in milliseconds.
|
|
241
282
|
*/
|
|
242
283
|
export declare function effectiveScrollBudgetMs(config: Pick<Config, 'scrollBudgetMs'>): number;
|
|
284
|
+
/**
|
|
285
|
+
* The settle cap a fetch actually runs with: an explicit value wins, else
|
|
286
|
+
* the built-in default. `0` means the settle is off (domcontentloaded
|
|
287
|
+
* content is read as-is).
|
|
288
|
+
*
|
|
289
|
+
* @param config - the resolved settings section (or any partial of it).
|
|
290
|
+
* @returns the bounded settle cap in milliseconds.
|
|
291
|
+
*/
|
|
292
|
+
export declare function effectiveSettleMs(config: Pick<Config, 'settleMs'>): number;
|
|
243
293
|
/**
|
|
244
294
|
* The context mode a fetch actually runs with: profile (a tab in the remote
|
|
245
295
|
* browser's real profile) requires the CDP backend — a local launch has no
|
package/lib/types/index.d.ts
CHANGED
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
*/
|
|
16
16
|
import type { Context } from '@deepseek-ai/cordis';
|
|
17
17
|
import type { PlaywrightPluginConfig } from './config.ts';
|
|
18
|
-
export { Config, DEFAULT_CDP_ENDPOINT, DEFAULT_CHALLENGE_RETRIES, DEFAULT_CHALLENGE_WAIT_MS, DEFAULT_SCROLL_BUDGET_MS,
|
|
18
|
+
export { Config, DEFAULT_CDP_ENDPOINT, DEFAULT_CHALLENGE_HTTP_FALLBACK, DEFAULT_CHALLENGE_RETRIES, DEFAULT_CHALLENGE_WAIT_MS, DEFAULT_SCROLL_BUDGET_MS, DEFAULT_SETTLE_MS, effectiveChallengeHttpFallback, effectiveChallengeRetries, effectiveChallengeWaitMs, effectiveContextMode, effectiveScrollBudgetMs, effectiveSettleMs, normalizeCdpEndpoint, resolveConfig, snapshotsOf, } from './config.ts';
|
|
19
19
|
export type { Config as PlaywrightFetchConfig, CdpContextMode, ContextIsolationScope, PlaywrightBackend, PlaywrightPluginConfig, ResolvedConfig, } from './config.ts';
|
|
20
20
|
export { CdpConnectionPool } from './cdp-pool.ts';
|
|
21
21
|
export type { CdpAcquireMode, CdpConnect, CdpLease } from './cdp-pool.ts';
|
package/lib/types/provider.d.ts
CHANGED
|
@@ -18,6 +18,21 @@
|
|
|
18
18
|
* refetch through the page's API request context
|
|
19
19
|
* ({@link recoverDownloadResponse}) — instead of a `WEB_PROVIDER_ERROR`.
|
|
20
20
|
*
|
|
21
|
+
* Transient transport failures (issue: `net::ERR_NETWORK_CHANGED` and its
|
|
22
|
+
* family) are retried, not terminal: Chromium's NetworkChangeNotifier kills
|
|
23
|
+
* every in-flight request with `ERR_NETWORK_CHANGED` the moment the HOST's
|
|
24
|
+
* network configuration shifts under the browser — a route/interface/DNS/
|
|
25
|
+
* proxy change, a VPN or DHCP flap, a Docker embedded-DNS rewrite; the page
|
|
26
|
+
* and its target are perfectly fine, only that one navigation died, and the
|
|
27
|
+
* very next `goto` usually succeeds (measured live: source.android.com
|
|
28
|
+
* failed once mid-navigation in one DSH session and fetched clean from the
|
|
29
|
+
* same host seconds later). Each navigation therefore retries the
|
|
30
|
+
* transport-blip family ({@link isTransientNavigationFailure} — deliberately
|
|
31
|
+
* NOT `ERR_ABORTED`, which is overloaded with the deterministic shapes the
|
|
32
|
+
* recoveries above already own) a bounded number of times with a short
|
|
33
|
+
* backoff, inside the same page and the remaining waiting budget; a blip
|
|
34
|
+
* that survives the attempt cap still fails as before.
|
|
35
|
+
*
|
|
21
36
|
* Lifecycle: the local backend launches a browser per fetch and closes it —
|
|
22
37
|
* nothing outlives the call. CDP session isolation keeps one connection per
|
|
23
38
|
* DSH session until its context becomes idle; this also isolates gateways that
|
|
@@ -55,9 +70,14 @@
|
|
|
55
70
|
* behavior). The wait tracks the LAST main-frame navigation response (the
|
|
56
71
|
* real page reloads in after the challenge clears) and watches the live DOM
|
|
57
72
|
* so SPA-style clears are caught too. It never clicks, never injects
|
|
58
|
-
* answers, never touches cookies itself; when the budget runs out
|
|
59
|
-
*
|
|
60
|
-
*
|
|
73
|
+
* answers, never touches cookies itself; when the budget runs out, ONE
|
|
74
|
+
* plain-HTTP refetch of the same URL through the page's request context
|
|
75
|
+
* (same cookie jar, the browser's UA, none of Chromium's render decisions
|
|
76
|
+
* — `challengeHttpFallback`, default on) tries to retrieve the real
|
|
77
|
+
* document, gated by the same response-level and content-level challenge
|
|
78
|
+
* classifiers so an interstitial can never pass as content. When that too
|
|
79
|
+
* fails, the fetch fails with {@link WEB_FETCH_CHALLENGE_CODE} instead of
|
|
80
|
+
* returning the interstitial as content.
|
|
61
81
|
*
|
|
62
82
|
* Deep capture (viewport-virtualized feeds and app-shell documents): feed-
|
|
63
83
|
* shaped pages — Discourse topic streams, comment threads, timelines —
|
|
@@ -84,6 +104,17 @@
|
|
|
84
104
|
* degrades to the legacy capture (see `src/autoscroll.ts` for the measured
|
|
85
105
|
* linux.do and Feishu badcases).
|
|
86
106
|
*
|
|
107
|
+
* Deadline (soft, then hard): each fetch has a {@link DEFAULT_TIMEOUT_MS}
|
|
108
|
+
* budget for WAITING — navigation, settle, challenge wait, scroll walk. When
|
|
109
|
+
* it runs out the page is NOT torn down: every wait is bounded by the same
|
|
110
|
+
* budget and simply ends, a navigation still loading is stopped
|
|
111
|
+
* (`window.stop()`) once its main-frame response has arrived, and whatever
|
|
112
|
+
* the page holds by then goes through the normal extraction — a slow or
|
|
113
|
+
* endless page still hands the agent the content it already rendered
|
|
114
|
+
* (flagged `truncated` when the document never finished loading). Only when
|
|
115
|
+
* nothing arrived at all, or extraction itself hangs past
|
|
116
|
+
* {@link EXTRACT_GRACE_MS}, does the fetch fail with `WEB_FETCH_TIMEOUT`.
|
|
117
|
+
*
|
|
87
118
|
* @module dsh-web-fetch-playwright/provider
|
|
88
119
|
*/
|
|
89
120
|
import type { WebFetchProvider, WebFetchRequest, WebFetchResult } from '@deepseek-ai/dsh-web';
|
|
@@ -119,19 +150,25 @@ export interface BrowserSession {
|
|
|
119
150
|
release?: () => Promise<void>;
|
|
120
151
|
}
|
|
121
152
|
/**
|
|
122
|
-
*
|
|
123
|
-
*
|
|
124
|
-
*
|
|
153
|
+
* The fetch's two-stage deadline composed with the caller's signal. The SOFT
|
|
154
|
+
* stage (`timeoutMs`) only bounds waiting: {@link remainingMs} counts down to
|
|
155
|
+
* it and {@link expired} reports it, so every wait ends there and the fetch
|
|
156
|
+
* extracts what the page holds. The HARD stage (`timeoutMs + graceMs`) aborts
|
|
157
|
+
* {@link signal}, which closes the page. The timeout reason is kept apart
|
|
158
|
+
* from an outer abort so the error translation can pick `WEB_FETCH_TIMEOUT`
|
|
159
|
+
* over `WEB_ABORTED`.
|
|
125
160
|
*/
|
|
126
161
|
declare class Deadline {
|
|
127
162
|
readonly signal: AbortSignal;
|
|
128
163
|
private readonly controller;
|
|
129
164
|
private readonly expiresAt;
|
|
130
165
|
private timedOut;
|
|
131
|
-
constructor(outer: AbortSignal | undefined, timeoutMs: number);
|
|
132
|
-
/** Whether OUR timer fired (vs an outer cancellation). */
|
|
166
|
+
constructor(outer: AbortSignal | undefined, timeoutMs: number, graceMs?: number);
|
|
167
|
+
/** Whether OUR hard timer fired (vs an outer cancellation). */
|
|
133
168
|
get isTimeout(): boolean;
|
|
134
|
-
/**
|
|
169
|
+
/** Whether the soft (waiting) budget has run out. */
|
|
170
|
+
get expired(): boolean;
|
|
171
|
+
/** Milliseconds left on the waiting budget, floored at 1 for Playwright options. */
|
|
135
172
|
remainingMs(): number;
|
|
136
173
|
}
|
|
137
174
|
/**
|
|
@@ -144,6 +181,16 @@ export declare class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
144
181
|
private readonly sessionIdSource;
|
|
145
182
|
readonly id = "playwright";
|
|
146
183
|
private readonly semaphore;
|
|
184
|
+
/** Waiting budget per fetch (ms); code-level, overridable by subclasses (tests). */
|
|
185
|
+
protected readonly fetchTimeoutMs: number;
|
|
186
|
+
/** Extraction grace after the waiting budget (ms); code-level, overridable by subclasses. */
|
|
187
|
+
protected readonly extractGraceMs: number;
|
|
188
|
+
/** Settle quiet window (ms); code-level, overridable by subclasses (tests). */
|
|
189
|
+
protected readonly settleQuietMs: number;
|
|
190
|
+
/** Request age (ms) past which it stops gating the settle; overridable by subclasses (tests). */
|
|
191
|
+
protected readonly settleStaleMs: number;
|
|
192
|
+
/** Settle re-check cadence (ms); overridable by subclasses (tests). */
|
|
193
|
+
protected readonly settlePollMs: number;
|
|
147
194
|
/** Shared CDP connection; injectable so the suite can substitute a fake. */
|
|
148
195
|
protected readonly cdpPool: CdpConnectionPool;
|
|
149
196
|
/**
|
|
@@ -174,6 +221,17 @@ export declare class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
174
221
|
* budget before the final document is read (see the class docs).
|
|
175
222
|
*/
|
|
176
223
|
private retrieve;
|
|
224
|
+
/**
|
|
225
|
+
* The challenge dead-end's plain-HTTP refetch: one request through the
|
|
226
|
+
* page's own request context (the SAME cookie jar — any clearance the
|
|
227
|
+
* profile holds rides along — but none of Chromium's render decisions),
|
|
228
|
+
* wearing the browser's own User-Agent. Returns the finished result when
|
|
229
|
+
* the answer is a NON-challenge HTML document, undefined otherwise (the
|
|
230
|
+
* caller throws the honest WEB_FETCH_CHALLENGE). A challenge served to
|
|
231
|
+
* this request too, a non-HTML body, or any transport failure keeps the
|
|
232
|
+
* browser-only verdict.
|
|
233
|
+
*/
|
|
234
|
+
private challengeHttpFallback;
|
|
177
235
|
/**
|
|
178
236
|
* One tracked `goto`: the navigation response on success, or — when
|
|
179
237
|
* Chromium failed the navigation for a reason that is not a transport
|
|
@@ -182,6 +240,11 @@ export declare class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
182
240
|
* status-only 204/205 — {@link recoverUnrenderedStatus}), and the response
|
|
183
241
|
* being a download the browser refuses to commit as a document
|
|
184
242
|
* (`Content-Disposition: attachment` — {@link recoverDownloadResponse}).
|
|
243
|
+
* A TRANSIENT transport failure (`net::ERR_NETWORK_CHANGED` and family)
|
|
244
|
+
* is neither: it carries no response and no render decision, only a
|
|
245
|
+
* momentary host-network blip that killed this one navigation — so it is
|
|
246
|
+
* retried on the SAME page, bounded by {@link MAX_TRANSIENT_NAV_RETRIES}
|
|
247
|
+
* attempts, a short backoff, and the remaining waiting budget.
|
|
185
248
|
*
|
|
186
249
|
* @returns the response `goto` resolved with, plus `recovered` when the
|
|
187
250
|
* navigation failure was recovered into a finished result the caller must
|
|
@@ -207,5 +270,34 @@ export declare class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
207
270
|
* @returns true when the challenge cleared; false when the budget ran out.
|
|
208
271
|
*/
|
|
209
272
|
private waitForChallengeClear;
|
|
273
|
+
/**
|
|
274
|
+
* Best-effort settle for client-rendered content — a Lighthouse-style
|
|
275
|
+
* bounded network-quiet window replacing Playwright's `networkidle`
|
|
276
|
+
* (officially discouraged): worker-script fetches and cross-origin auth
|
|
277
|
+
* frames hold one request open on the page session FOREVER (their
|
|
278
|
+
* completion events are attributed to other CDP targets), so strict
|
|
279
|
+
* networkidle never arrives on exactly the modern pages the settle exists
|
|
280
|
+
* for, and the legacy wait burned its whole cap on them every fetch.
|
|
281
|
+
*
|
|
282
|
+
* Sequence: a `load` baseline (usually already past — domcontentloaded
|
|
283
|
+
* resolved earlier), then poll until no request has STARTED within
|
|
284
|
+
* {@link settleQuietMs} and every still-open request is older than
|
|
285
|
+
* {@link settleStaleMs} — a request that old is an attribution artifact
|
|
286
|
+
* or a long-poll whose completion no longer gates content. All waits
|
|
287
|
+
* share the {@link effectiveSettleMs} cap and the fetch deadline; a page
|
|
288
|
+
* that exposes no request events degrades to the legacy bounded
|
|
289
|
+
* networkidle wait. Never throws: any exit path keeps what the document
|
|
290
|
+
* already holds.
|
|
291
|
+
*/
|
|
292
|
+
private settleForContent;
|
|
293
|
+
/**
|
|
294
|
+
* Settle outcome observation hook — silent in production by design (the
|
|
295
|
+
* settle is best-effort), but subclasses (tests, diagnostics) override it
|
|
296
|
+
* to see which exit path every fetch took: `off` (config), `quiet`
|
|
297
|
+
* (network went quiet), `stale-discounted` (only stale requests remained
|
|
298
|
+
* open), `cap` (window exhausted), `fallback` (no request events — the
|
|
299
|
+
* legacy networkidle wait ran), `aborted` (fetch deadline fired).
|
|
300
|
+
*/
|
|
301
|
+
protected noteSettleOutcome(outcome: 'off' | 'quiet' | 'stale-discounted' | 'cap' | 'fallback' | 'aborted', elapsedMs: number): void;
|
|
210
302
|
}
|
|
211
303
|
export {};
|