@isdk/web-fetcher 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.action.cn.md +469 -0
- package/README.action.md +452 -0
- package/README.cn.md +147 -0
- package/README.engine.cn.md +262 -0
- package/README.engine.md +262 -0
- package/README.md +147 -0
- package/dist/index.d.mts +1603 -0
- package/dist/index.d.ts +1603 -0
- package/dist/index.js +1 -0
- package/dist/index.mjs +1 -0
- package/docs/README.md +151 -0
- package/docs/_media/LICENSE-MIT +22 -0
- package/docs/_media/README.action.md +452 -0
- package/docs/_media/README.cn.md +147 -0
- package/docs/_media/README.engine.md +262 -0
- package/docs/classes/CheerioFetchEngine.md +1447 -0
- package/docs/classes/ClickAction.md +533 -0
- package/docs/classes/ExtractAction.md +533 -0
- package/docs/classes/FetchAction.md +444 -0
- package/docs/classes/FetchEngine.md +1230 -0
- package/docs/classes/FetchSession.md +111 -0
- package/docs/classes/FillAction.md +533 -0
- package/docs/classes/GetContentAction.md +533 -0
- package/docs/classes/GotoAction.md +537 -0
- package/docs/classes/PauseAction.md +533 -0
- package/docs/classes/PlaywrightFetchEngine.md +1437 -0
- package/docs/classes/SubmitAction.md +533 -0
- package/docs/classes/WaitForAction.md +533 -0
- package/docs/classes/WebFetcher.md +85 -0
- package/docs/enumerations/FetchActionResultStatus.md +40 -0
- package/docs/functions/fetchWeb.md +43 -0
- package/docs/globals.md +72 -0
- package/docs/interfaces/BaseFetchActionProperties.md +83 -0
- package/docs/interfaces/BaseFetchCollectorActionProperties.md +145 -0
- package/docs/interfaces/BaseFetcherProperties.md +206 -0
- package/docs/interfaces/Cookie.md +142 -0
- package/docs/interfaces/DispatchedEngineAction.md +60 -0
- package/docs/interfaces/ExtractActionProperties.md +113 -0
- package/docs/interfaces/FetchActionInContext.md +149 -0
- package/docs/interfaces/FetchActionProperties.md +125 -0
- package/docs/interfaces/FetchActionResult.md +55 -0
- package/docs/interfaces/FetchContext.md +424 -0
- package/docs/interfaces/FetchEngineContext.md +328 -0
- package/docs/interfaces/FetchMetadata.md +73 -0
- package/docs/interfaces/FetchResponse.md +105 -0
- package/docs/interfaces/FetchReturnTypeRegistry.md +57 -0
- package/docs/interfaces/FetchSite.md +320 -0
- package/docs/interfaces/FetcherOptions.md +300 -0
- package/docs/interfaces/GotoActionOptions.md +66 -0
- package/docs/interfaces/PendingEngineRequest.md +51 -0
- package/docs/interfaces/SubmitActionOptions.md +23 -0
- package/docs/interfaces/WaitForActionOptions.md +39 -0
- package/docs/type-aliases/BaseFetchActionOptions.md +11 -0
- package/docs/type-aliases/BaseFetchCollectorOptions.md +11 -0
- package/docs/type-aliases/BrowserEngine.md +11 -0
- package/docs/type-aliases/FetchActionCapabilities.md +11 -0
- package/docs/type-aliases/FetchActionCapabilityMode.md +11 -0
- package/docs/type-aliases/FetchActionOptions.md +11 -0
- package/docs/type-aliases/FetchEngineAction.md +18 -0
- package/docs/type-aliases/FetchEngineType.md +11 -0
- package/docs/type-aliases/FetchReturnType.md +11 -0
- package/docs/type-aliases/FetchReturnTypeFor.md +17 -0
- package/docs/type-aliases/OnFetchPauseCallback.md +23 -0
- package/docs/type-aliases/ResourceType.md +11 -0
- package/docs/variables/DefaultFetcherProperties.md +11 -0
- package/package.json +90 -0
|
@@ -0,0 +1,1230 @@
|
|
|
1
|
+
[**@isdk/web-fetcher**](../README.md)
|
|
2
|
+
|
|
3
|
+
***
|
|
4
|
+
|
|
5
|
+
[@isdk/web-fetcher](../globals.md) / FetchEngine
|
|
6
|
+
|
|
7
|
+
# Abstract Class: FetchEngine\<TContext, TCrawler, TOptions\>
|
|
8
|
+
|
|
9
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:139](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L139)
|
|
10
|
+
|
|
11
|
+
## Extended by
|
|
12
|
+
|
|
13
|
+
- [`CheerioFetchEngine`](CheerioFetchEngine.md)
|
|
14
|
+
- [`PlaywrightFetchEngine`](PlaywrightFetchEngine.md)
|
|
15
|
+
|
|
16
|
+
## Type Parameters
|
|
17
|
+
|
|
18
|
+
### TContext
|
|
19
|
+
|
|
20
|
+
`TContext` *extends* `CrawlingContext` = `any`
|
|
21
|
+
|
|
22
|
+
### TCrawler
|
|
23
|
+
|
|
24
|
+
`TCrawler` *extends* `BasicCrawler`\<`TContext`\> = `any`
|
|
25
|
+
|
|
26
|
+
### TOptions
|
|
27
|
+
|
|
28
|
+
`TOptions` *extends* `BasicCrawlerOptions`\<`TContext`\> = `any`
|
|
29
|
+
|
|
30
|
+
## Constructors
|
|
31
|
+
|
|
32
|
+
### Constructor
|
|
33
|
+
|
|
34
|
+
> **new FetchEngine**\<`TContext`, `TCrawler`, `TOptions`\>(): `FetchEngine`\<`TContext`, `TCrawler`, `TOptions`\>
|
|
35
|
+
|
|
36
|
+
#### Returns
|
|
37
|
+
|
|
38
|
+
`FetchEngine`\<`TContext`, `TCrawler`, `TOptions`\>
|
|
39
|
+
|
|
40
|
+
## Properties
|
|
41
|
+
|
|
42
|
+
### actionEmitter
|
|
43
|
+
|
|
44
|
+
> `protected` **actionEmitter**: `EventEmitter`
|
|
45
|
+
|
|
46
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:235](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L235)
|
|
47
|
+
|
|
48
|
+
***
|
|
49
|
+
|
|
50
|
+
### blockedTypes
|
|
51
|
+
|
|
52
|
+
> `protected` **blockedTypes**: `Set`\<`string`\>
|
|
53
|
+
|
|
54
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:239](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L239)
|
|
55
|
+
|
|
56
|
+
***
|
|
57
|
+
|
|
58
|
+
### crawler?
|
|
59
|
+
|
|
60
|
+
> `protected` `optional` **crawler**: `TCrawler`
|
|
61
|
+
|
|
62
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:227](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L227)
|
|
63
|
+
|
|
64
|
+
***
|
|
65
|
+
|
|
66
|
+
### ctx?
|
|
67
|
+
|
|
68
|
+
> `protected` `optional` **ctx**: [`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
69
|
+
|
|
70
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:225](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L225)
|
|
71
|
+
|
|
72
|
+
***
|
|
73
|
+
|
|
74
|
+
### hdrs
|
|
75
|
+
|
|
76
|
+
> `protected` **hdrs**: `Record`\<`string`, `string`\> = `{}`
|
|
77
|
+
|
|
78
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:231](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L231)
|
|
79
|
+
|
|
80
|
+
***
|
|
81
|
+
|
|
82
|
+
### isCrawlerReady?
|
|
83
|
+
|
|
84
|
+
> `protected` `optional` **isCrawlerReady**: `boolean`
|
|
85
|
+
|
|
86
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:228](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L228)
|
|
87
|
+
|
|
88
|
+
***
|
|
89
|
+
|
|
90
|
+
### isPageActive
|
|
91
|
+
|
|
92
|
+
> `protected` **isPageActive**: `boolean` = `false`
|
|
93
|
+
|
|
94
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:236](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L236)
|
|
95
|
+
|
|
96
|
+
***
|
|
97
|
+
|
|
98
|
+
### jar
|
|
99
|
+
|
|
100
|
+
> `protected` **jar**: [`Cookie`](../interfaces/Cookie.md)[] = `[]`
|
|
101
|
+
|
|
102
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:232](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L232)
|
|
103
|
+
|
|
104
|
+
***
|
|
105
|
+
|
|
106
|
+
### lastResponse?
|
|
107
|
+
|
|
108
|
+
> `protected` `optional` **lastResponse**: [`FetchResponse`](../interfaces/FetchResponse.md)
|
|
109
|
+
|
|
110
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:238](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L238)
|
|
111
|
+
|
|
112
|
+
***
|
|
113
|
+
|
|
114
|
+
### navigationLock
|
|
115
|
+
|
|
116
|
+
> `protected` **navigationLock**: `PromiseLock`
|
|
117
|
+
|
|
118
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:237](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L237)
|
|
119
|
+
|
|
120
|
+
***
|
|
121
|
+
|
|
122
|
+
### opts?
|
|
123
|
+
|
|
124
|
+
> `protected` `optional` **opts**: [`BaseFetcherProperties`](../interfaces/BaseFetcherProperties.md)
|
|
125
|
+
|
|
126
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:226](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L226)
|
|
127
|
+
|
|
128
|
+
***
|
|
129
|
+
|
|
130
|
+
### pendingRequests
|
|
131
|
+
|
|
132
|
+
> `protected` **pendingRequests**: `Map`\<`string`, [`PendingEngineRequest`](../interfaces/PendingEngineRequest.md)\>
|
|
133
|
+
|
|
134
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:233](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L233)
|
|
135
|
+
|
|
136
|
+
***
|
|
137
|
+
|
|
138
|
+
### requestCounter
|
|
139
|
+
|
|
140
|
+
> `protected` **requestCounter**: `number` = `0`
|
|
141
|
+
|
|
142
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:234](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L234)
|
|
143
|
+
|
|
144
|
+
***
|
|
145
|
+
|
|
146
|
+
### requestQueue?
|
|
147
|
+
|
|
148
|
+
> `protected` `optional` **requestQueue**: `RequestQueue`
|
|
149
|
+
|
|
150
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:229](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L229)
|
|
151
|
+
|
|
152
|
+
***
|
|
153
|
+
|
|
154
|
+
### id
|
|
155
|
+
|
|
156
|
+
> `readonly` `static` **id**: `string`
|
|
157
|
+
|
|
158
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:215](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L215)
|
|
159
|
+
|
|
160
|
+
Unique identifier for the engine implementation.
|
|
161
|
+
|
|
162
|
+
#### Remarks
|
|
163
|
+
|
|
164
|
+
Must be defined by concrete implementations. Used for registration and lookup in engine registry.
|
|
165
|
+
|
|
166
|
+
***
|
|
167
|
+
|
|
168
|
+
### mode
|
|
169
|
+
|
|
170
|
+
> `readonly` `static` **mode**: [`FetchEngineType`](../type-aliases/FetchEngineType.md)
|
|
171
|
+
|
|
172
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:223](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L223)
|
|
173
|
+
|
|
174
|
+
Execution mode of the engine (`'http'` or `'browser'`).
|
|
175
|
+
|
|
176
|
+
#### Remarks
|
|
177
|
+
|
|
178
|
+
Must be defined by concrete implementations. Indicates whether engine operates at HTTP level or uses full browser.
|
|
179
|
+
|
|
180
|
+
## Accessors
|
|
181
|
+
|
|
182
|
+
### context
|
|
183
|
+
|
|
184
|
+
#### Get Signature
|
|
185
|
+
|
|
186
|
+
> **get** **context**(): `undefined` \| [`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
187
|
+
|
|
188
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:473](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L473)
|
|
189
|
+
|
|
190
|
+
Gets the fetch engine context associated with this instance.
|
|
191
|
+
|
|
192
|
+
##### Returns
|
|
193
|
+
|
|
194
|
+
`undefined` \| [`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
195
|
+
|
|
196
|
+
***
|
|
197
|
+
|
|
198
|
+
### id
|
|
199
|
+
|
|
200
|
+
#### Get Signature
|
|
201
|
+
|
|
202
|
+
> **get** **id**(): `string`
|
|
203
|
+
|
|
204
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:459](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L459)
|
|
205
|
+
|
|
206
|
+
Gets the unique identifier of this engine implementation.
|
|
207
|
+
|
|
208
|
+
##### Returns
|
|
209
|
+
|
|
210
|
+
`string`
|
|
211
|
+
|
|
212
|
+
***
|
|
213
|
+
|
|
214
|
+
### mode
|
|
215
|
+
|
|
216
|
+
#### Get Signature
|
|
217
|
+
|
|
218
|
+
> **get** **mode**(): [`FetchEngineType`](../type-aliases/FetchEngineType.md)
|
|
219
|
+
|
|
220
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:466](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L466)
|
|
221
|
+
|
|
222
|
+
Gets the execution mode of this engine (`'http'` or `'browser'`).
|
|
223
|
+
|
|
224
|
+
##### Returns
|
|
225
|
+
|
|
226
|
+
[`FetchEngineType`](../type-aliases/FetchEngineType.md)
|
|
227
|
+
|
|
228
|
+
## Methods
|
|
229
|
+
|
|
230
|
+
### \_cleanup()?
|
|
231
|
+
|
|
232
|
+
> `protected` `optional` **\_cleanup**(): `Promise`\<`void`\>
|
|
233
|
+
|
|
234
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:241](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L241)
|
|
235
|
+
|
|
236
|
+
#### Returns
|
|
237
|
+
|
|
238
|
+
`Promise`\<`void`\>
|
|
239
|
+
|
|
240
|
+
***
|
|
241
|
+
|
|
242
|
+
### \_commonCleanup()
|
|
243
|
+
|
|
244
|
+
> `protected` **\_commonCleanup**(): `Promise`\<`void`\>
|
|
245
|
+
|
|
246
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:656](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L656)
|
|
247
|
+
|
|
248
|
+
#### Returns
|
|
249
|
+
|
|
250
|
+
`Promise`\<`void`\>
|
|
251
|
+
|
|
252
|
+
***
|
|
253
|
+
|
|
254
|
+
### \_createCrawler()
|
|
255
|
+
|
|
256
|
+
> `abstract` `protected` **\_createCrawler**(`options`): `TCrawler`
|
|
257
|
+
|
|
258
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:296](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L296)
|
|
259
|
+
|
|
260
|
+
**`Internal`**
|
|
261
|
+
|
|
262
|
+
Creates the crawler instance for the specific engine implementation.
|
|
263
|
+
|
|
264
|
+
#### Parameters
|
|
265
|
+
|
|
266
|
+
##### options
|
|
267
|
+
|
|
268
|
+
`TOptions`
|
|
269
|
+
|
|
270
|
+
The final crawler options.
|
|
271
|
+
|
|
272
|
+
#### Returns
|
|
273
|
+
|
|
274
|
+
`TCrawler`
|
|
275
|
+
|
|
276
|
+
***
|
|
277
|
+
|
|
278
|
+
### \_executePendingActions()
|
|
279
|
+
|
|
280
|
+
> `protected` **\_executePendingActions**(`context`): `Promise`\<`void`\>
|
|
281
|
+
|
|
282
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:572](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L572)
|
|
283
|
+
|
|
284
|
+
**`Internal`**
|
|
285
|
+
|
|
286
|
+
Executes all pending fetch engine actions within the current Crawlee request handler context.
|
|
287
|
+
|
|
288
|
+
**Critical Execution Constraint**: This method **MUST** be awaited within the synchronous execution path
|
|
289
|
+
of Crawlee's [requestHandler](https://crawlee.dev/js/api/basic-crawler) (before any `await` that yields control back to the event loop).
|
|
290
|
+
|
|
291
|
+
### Why This Constraint Exists
|
|
292
|
+
- Crawlee's page context ([PlaywrightCrawler](https://crawlee.dev/js/api/playwright-crawler)'s `page` or [CheerioCrawler](https://crawlee.dev/js/api/cheerio-crawler)'s `$`)
|
|
293
|
+
is **only valid during the synchronous execution phase** of the request handler
|
|
294
|
+
- After any `await` (e.g., `await page.goto()`), the page context may be destroyed
|
|
295
|
+
due to Crawlee's internal resource management
|
|
296
|
+
|
|
297
|
+
### How It Works
|
|
298
|
+
1. Processes all actions queued via [dispatchAction](#dispatchaction) (click, fill, submit, etc.)
|
|
299
|
+
2. Maintains the page context validity window via [isPageActive](#ispageactive) lifecycle flag
|
|
300
|
+
3. Automatically cleans up event listeners upon completion
|
|
301
|
+
|
|
302
|
+
Usage see [\_sharedRequestHandler](#_sharedrequesthandler)
|
|
303
|
+
|
|
304
|
+
#### Parameters
|
|
305
|
+
|
|
306
|
+
##### context
|
|
307
|
+
|
|
308
|
+
`TContext`
|
|
309
|
+
|
|
310
|
+
The active Crawlee crawling context containing the page/$ object
|
|
311
|
+
|
|
312
|
+
#### Returns
|
|
313
|
+
|
|
314
|
+
`Promise`\<`void`\>
|
|
315
|
+
|
|
316
|
+
#### See
|
|
317
|
+
|
|
318
|
+
[\_sharedRequestHandler](#_sharedrequesthandler)
|
|
319
|
+
|
|
320
|
+
#### Throws
|
|
321
|
+
|
|
322
|
+
If called outside valid page context window (`!this.isPageActive`)
|
|
323
|
+
Engine infrastructure method - not for direct consumer use
|
|
324
|
+
|
|
325
|
+
***
|
|
326
|
+
|
|
327
|
+
### \_extract()
|
|
328
|
+
|
|
329
|
+
> `protected` **\_extract**(`schema`, `context`): `Promise`\<`any`\>
|
|
330
|
+
|
|
331
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:246](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L246)
|
|
332
|
+
|
|
333
|
+
#### Parameters
|
|
334
|
+
|
|
335
|
+
##### schema
|
|
336
|
+
|
|
337
|
+
`ExtractSchema`
|
|
338
|
+
|
|
339
|
+
##### context
|
|
340
|
+
|
|
341
|
+
`any`
|
|
342
|
+
|
|
343
|
+
#### Returns
|
|
344
|
+
|
|
345
|
+
`Promise`\<`any`\>
|
|
346
|
+
|
|
347
|
+
***
|
|
348
|
+
|
|
349
|
+
### \_extractValue()
|
|
350
|
+
|
|
351
|
+
> `abstract` `protected` **\_extractValue**(`schema`, `context`): `Promise`\<`any`\>
|
|
352
|
+
|
|
353
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:244](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L244)
|
|
354
|
+
|
|
355
|
+
#### Parameters
|
|
356
|
+
|
|
357
|
+
##### schema
|
|
358
|
+
|
|
359
|
+
`ExtractValueSchema`
|
|
360
|
+
|
|
361
|
+
##### context
|
|
362
|
+
|
|
363
|
+
`any`
|
|
364
|
+
|
|
365
|
+
#### Returns
|
|
366
|
+
|
|
367
|
+
`Promise`\<`any`\>
|
|
368
|
+
|
|
369
|
+
***
|
|
370
|
+
|
|
371
|
+
### \_getSpecificCrawlerOptions()
|
|
372
|
+
|
|
373
|
+
> `abstract` `protected` **\_getSpecificCrawlerOptions**(`ctx`): `Partial`\<`TOptions`\> \| `Promise`\<`Partial`\<`TOptions`\>\>
|
|
374
|
+
|
|
375
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:303](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L303)
|
|
376
|
+
|
|
377
|
+
**`Internal`**
|
|
378
|
+
|
|
379
|
+
Gets the crawler-specific options from the subclass.
|
|
380
|
+
|
|
381
|
+
#### Parameters
|
|
382
|
+
|
|
383
|
+
##### ctx
|
|
384
|
+
|
|
385
|
+
[`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
386
|
+
|
|
387
|
+
The fetch engine context.
|
|
388
|
+
|
|
389
|
+
#### Returns
|
|
390
|
+
|
|
391
|
+
`Partial`\<`TOptions`\> \| `Promise`\<`Partial`\<`TOptions`\>\>
|
|
392
|
+
|
|
393
|
+
***
|
|
394
|
+
|
|
395
|
+
### \_normalizeSchema()
|
|
396
|
+
|
|
397
|
+
> `protected` **\_normalizeSchema**(`schema`): `ExtractSchema`
|
|
398
|
+
|
|
399
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:415](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L415)
|
|
400
|
+
|
|
401
|
+
#### Parameters
|
|
402
|
+
|
|
403
|
+
##### schema
|
|
404
|
+
|
|
405
|
+
`ExtractSchema`
|
|
406
|
+
|
|
407
|
+
#### Returns
|
|
408
|
+
|
|
409
|
+
`ExtractSchema`
|
|
410
|
+
|
|
411
|
+
***
|
|
412
|
+
|
|
413
|
+
### \_querySelectorAll()
|
|
414
|
+
|
|
415
|
+
> `abstract` `protected` **\_querySelectorAll**(`context`, `selector`): `Promise`\<`any`[]\>
|
|
416
|
+
|
|
417
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:243](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L243)
|
|
418
|
+
|
|
419
|
+
#### Parameters
|
|
420
|
+
|
|
421
|
+
##### context
|
|
422
|
+
|
|
423
|
+
`any`
|
|
424
|
+
|
|
425
|
+
##### selector
|
|
426
|
+
|
|
427
|
+
`string`
|
|
428
|
+
|
|
429
|
+
#### Returns
|
|
430
|
+
|
|
431
|
+
`Promise`\<`any`[]\>
|
|
432
|
+
|
|
433
|
+
***
|
|
434
|
+
|
|
435
|
+
### \_sharedFailedRequestHandler()
|
|
436
|
+
|
|
437
|
+
> `protected` **\_sharedFailedRequestHandler**(`context`, `error?`): `Promise`\<`void`\>
|
|
438
|
+
|
|
439
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:623](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L623)
|
|
440
|
+
|
|
441
|
+
#### Parameters
|
|
442
|
+
|
|
443
|
+
##### context
|
|
444
|
+
|
|
445
|
+
`TContext`
|
|
446
|
+
|
|
447
|
+
##### error?
|
|
448
|
+
|
|
449
|
+
`Error`
|
|
450
|
+
|
|
451
|
+
#### Returns
|
|
452
|
+
|
|
453
|
+
`Promise`\<`void`\>
|
|
454
|
+
|
|
455
|
+
***
|
|
456
|
+
|
|
457
|
+
### \_sharedRequestHandler()
|
|
458
|
+
|
|
459
|
+
> `protected` **\_sharedRequestHandler**(`context`): `Promise`\<`void`\>
|
|
460
|
+
|
|
461
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:595](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L595)
|
|
462
|
+
|
|
463
|
+
#### Parameters
|
|
464
|
+
|
|
465
|
+
##### context
|
|
466
|
+
|
|
467
|
+
`TContext`
|
|
468
|
+
|
|
469
|
+
#### Returns
|
|
470
|
+
|
|
471
|
+
`Promise`\<`void`\>
|
|
472
|
+
|
|
473
|
+
***
|
|
474
|
+
|
|
475
|
+
### blockResources()
|
|
476
|
+
|
|
477
|
+
> **blockResources**(`types`, `overwrite?`): `Promise`\<`number`\>
|
|
478
|
+
|
|
479
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:700](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L700)
|
|
480
|
+
|
|
481
|
+
Blocks specified resource types from loading.
|
|
482
|
+
|
|
483
|
+
#### Parameters
|
|
484
|
+
|
|
485
|
+
##### types
|
|
486
|
+
|
|
487
|
+
`string`[]
|
|
488
|
+
|
|
489
|
+
Resource types to block
|
|
490
|
+
|
|
491
|
+
##### overwrite?
|
|
492
|
+
|
|
493
|
+
`boolean`
|
|
494
|
+
|
|
495
|
+
Whether to replace existing blocked types
|
|
496
|
+
|
|
497
|
+
#### Returns
|
|
498
|
+
|
|
499
|
+
`Promise`\<`number`\>
|
|
500
|
+
|
|
501
|
+
Number of blocked resource types
|
|
502
|
+
|
|
503
|
+
#### Example
|
|
504
|
+
|
|
505
|
+
```ts
|
|
506
|
+
await engine.blockResources(['image', 'stylesheet']);
|
|
507
|
+
await engine.blockResources(['script'], true); // Replace existing
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
***
|
|
511
|
+
|
|
512
|
+
### buildResponse()
|
|
513
|
+
|
|
514
|
+
> `abstract` `protected` **buildResponse**(`context`): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
515
|
+
|
|
516
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:314](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L314)
|
|
517
|
+
|
|
518
|
+
**`Internal`**
|
|
519
|
+
|
|
520
|
+
Abstract method for building standard [FetchResponse] from Crawlee context.
|
|
521
|
+
|
|
522
|
+
#### Parameters
|
|
523
|
+
|
|
524
|
+
##### context
|
|
525
|
+
|
|
526
|
+
`TContext`
|
|
527
|
+
|
|
528
|
+
Crawlee crawling context
|
|
529
|
+
|
|
530
|
+
#### Returns
|
|
531
|
+
|
|
532
|
+
`Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
533
|
+
|
|
534
|
+
Promise resolving to [FetchResponse] object
|
|
535
|
+
|
|
536
|
+
#### Remarks
|
|
537
|
+
|
|
538
|
+
Converts implementation-specific context (Playwright `page` or Cheerio `$`) to standardized response.
|
|
539
|
+
|
|
540
|
+
***
|
|
541
|
+
|
|
542
|
+
### cleanup()
|
|
543
|
+
|
|
544
|
+
> **cleanup**(): `Promise`\<`void`\>
|
|
545
|
+
|
|
546
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:536](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L536)
|
|
547
|
+
|
|
548
|
+
#### Returns
|
|
549
|
+
|
|
550
|
+
`Promise`\<`void`\>
|
|
551
|
+
|
|
552
|
+
***
|
|
553
|
+
|
|
554
|
+
### click()
|
|
555
|
+
|
|
556
|
+
> **click**(`selector`): `Promise`\<`void`\>
|
|
557
|
+
|
|
558
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:365](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L365)
|
|
559
|
+
|
|
560
|
+
Clicks on element matching selector.
|
|
561
|
+
|
|
562
|
+
#### Parameters
|
|
563
|
+
|
|
564
|
+
##### selector
|
|
565
|
+
|
|
566
|
+
`string`
|
|
567
|
+
|
|
568
|
+
CSS selector of element to click
|
|
569
|
+
|
|
570
|
+
#### Returns
|
|
571
|
+
|
|
572
|
+
`Promise`\<`void`\>
|
|
573
|
+
|
|
574
|
+
Promise resolving when click is processed
|
|
575
|
+
|
|
576
|
+
#### Throws
|
|
577
|
+
|
|
578
|
+
When no active page context exists
|
|
579
|
+
|
|
580
|
+
***
|
|
581
|
+
|
|
582
|
+
### cookies()
|
|
583
|
+
|
|
584
|
+
#### Call Signature
|
|
585
|
+
|
|
586
|
+
> **cookies**(): `Promise`\<[`Cookie`](../interfaces/Cookie.md)[]\>
|
|
587
|
+
|
|
588
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:811](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L811)
|
|
589
|
+
|
|
590
|
+
Manages cookies for current session with multiple overloads.
|
|
591
|
+
|
|
592
|
+
Gets all cookies.
|
|
593
|
+
|
|
594
|
+
##### Returns
|
|
595
|
+
|
|
596
|
+
`Promise`\<[`Cookie`](../interfaces/Cookie.md)[]\>
|
|
597
|
+
|
|
598
|
+
Array of cookies
|
|
599
|
+
|
|
600
|
+
Sets cookies for session.
|
|
601
|
+
|
|
602
|
+
##### Example
|
|
603
|
+
|
|
604
|
+
```ts
|
|
605
|
+
const cookies = await engine.cookies();
|
|
606
|
+
await engine.cookies([{ name: 'session', value: '123' }]);
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
#### Call Signature
|
|
610
|
+
|
|
611
|
+
> **cookies**(`cookies`): `Promise`\<`boolean`\>
|
|
612
|
+
|
|
613
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:812](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L812)
|
|
614
|
+
|
|
615
|
+
Manages cookies for current session with multiple overloads.
|
|
616
|
+
|
|
617
|
+
Gets all cookies.
|
|
618
|
+
|
|
619
|
+
##### Parameters
|
|
620
|
+
|
|
621
|
+
###### cookies
|
|
622
|
+
|
|
623
|
+
[`Cookie`](../interfaces/Cookie.md)[]
|
|
624
|
+
|
|
625
|
+
Cookies to set
|
|
626
|
+
|
|
627
|
+
##### Returns
|
|
628
|
+
|
|
629
|
+
`Promise`\<`boolean`\>
|
|
630
|
+
|
|
631
|
+
Array of cookies
|
|
632
|
+
|
|
633
|
+
Sets cookies for session.
|
|
634
|
+
|
|
635
|
+
##### Example
|
|
636
|
+
|
|
637
|
+
```ts
|
|
638
|
+
const cookies = await engine.cookies();
|
|
639
|
+
await engine.cookies([{ name: 'session', value: '123' }]);
|
|
640
|
+
```
|
|
641
|
+
|
|
642
|
+
***
|
|
643
|
+
|
|
644
|
+
### dispatchAction()
|
|
645
|
+
|
|
646
|
+
> `protected` **dispatchAction**\<`T`\>(`action`): `Promise`\<`T`\>
|
|
647
|
+
|
|
648
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:639](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L639)
|
|
649
|
+
|
|
650
|
+
#### Type Parameters
|
|
651
|
+
|
|
652
|
+
##### T
|
|
653
|
+
|
|
654
|
+
`T`
|
|
655
|
+
|
|
656
|
+
#### Parameters
|
|
657
|
+
|
|
658
|
+
##### action
|
|
659
|
+
|
|
660
|
+
[`FetchEngineAction`](../type-aliases/FetchEngineAction.md)
|
|
661
|
+
|
|
662
|
+
#### Returns
|
|
663
|
+
|
|
664
|
+
`Promise`\<`T`\>
|
|
665
|
+
|
|
666
|
+
***
|
|
667
|
+
|
|
668
|
+
### dispose()
|
|
669
|
+
|
|
670
|
+
> **dispose**(): `Promise`\<`void`\>
|
|
671
|
+
|
|
672
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:829](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L829)
|
|
673
|
+
|
|
674
|
+
Disposes of engine, cleaning up all resources.
|
|
675
|
+
|
|
676
|
+
#### Returns
|
|
677
|
+
|
|
678
|
+
`Promise`\<`void`\>
|
|
679
|
+
|
|
680
|
+
Promise resolving when disposal completes
|
|
681
|
+
|
|
682
|
+
***
|
|
683
|
+
|
|
684
|
+
### executeAction()
|
|
685
|
+
|
|
686
|
+
> `abstract` `protected` **executeAction**(`context`, `action`): `Promise`\<`any`\>
|
|
687
|
+
|
|
688
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:326](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L326)
|
|
689
|
+
|
|
690
|
+
**`Internal`**
|
|
691
|
+
|
|
692
|
+
Abstract method for executing action within current page context.
|
|
693
|
+
|
|
694
|
+
#### Parameters
|
|
695
|
+
|
|
696
|
+
##### context
|
|
697
|
+
|
|
698
|
+
`TContext`
|
|
699
|
+
|
|
700
|
+
Crawlee crawling context
|
|
701
|
+
|
|
702
|
+
##### action
|
|
703
|
+
|
|
704
|
+
[`FetchEngineAction`](../type-aliases/FetchEngineAction.md)
|
|
705
|
+
|
|
706
|
+
Action to execute
|
|
707
|
+
|
|
708
|
+
#### Returns
|
|
709
|
+
|
|
710
|
+
`Promise`\<`any`\>
|
|
711
|
+
|
|
712
|
+
Promise resolving to action result
|
|
713
|
+
|
|
714
|
+
#### Remarks
|
|
715
|
+
|
|
716
|
+
Handles specific user interactions using underlying technology (Playwright/Cheerio).
|
|
717
|
+
|
|
718
|
+
***
|
|
719
|
+
|
|
720
|
+
### extract()
|
|
721
|
+
|
|
722
|
+
> **extract**\<`T`\>(`schema`): `Promise`\<`T`\>
|
|
723
|
+
|
|
724
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:410](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L410)
|
|
725
|
+
|
|
726
|
+
Extracts structured data from the current page content.
|
|
727
|
+
|
|
728
|
+
#### Type Parameters
|
|
729
|
+
|
|
730
|
+
##### T
|
|
731
|
+
|
|
732
|
+
`T`
|
|
733
|
+
|
|
734
|
+
#### Parameters
|
|
735
|
+
|
|
736
|
+
##### schema
|
|
737
|
+
|
|
738
|
+
`ExtractSchema`
|
|
739
|
+
|
|
740
|
+
An object defining the data to extract.
|
|
741
|
+
|
|
742
|
+
#### Returns
|
|
743
|
+
|
|
744
|
+
`Promise`\<`T`\>
|
|
745
|
+
|
|
746
|
+
A promise that resolves to an object with the extracted data.
|
|
747
|
+
|
|
748
|
+
***
|
|
749
|
+
|
|
750
|
+
### fill()
|
|
751
|
+
|
|
752
|
+
> **fill**(`selector`, `value`): `Promise`\<`void`\>
|
|
753
|
+
|
|
754
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:377](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L377)
|
|
755
|
+
|
|
756
|
+
Fills input element with specified value.
|
|
757
|
+
|
|
758
|
+
#### Parameters
|
|
759
|
+
|
|
760
|
+
##### selector
|
|
761
|
+
|
|
762
|
+
`string`
|
|
763
|
+
|
|
764
|
+
CSS selector of input element
|
|
765
|
+
|
|
766
|
+
##### value
|
|
767
|
+
|
|
768
|
+
`string`
|
|
769
|
+
|
|
770
|
+
Value to fill
|
|
771
|
+
|
|
772
|
+
#### Returns
|
|
773
|
+
|
|
774
|
+
`Promise`\<`void`\>
|
|
775
|
+
|
|
776
|
+
Promise resolving when fill operation completes
|
|
777
|
+
|
|
778
|
+
#### Throws
|
|
779
|
+
|
|
780
|
+
When no active page context exists
|
|
781
|
+
|
|
782
|
+
***
|
|
783
|
+
|
|
784
|
+
### getContent()
|
|
785
|
+
|
|
786
|
+
> **getContent**(): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
787
|
+
|
|
788
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:714](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L714)
|
|
789
|
+
|
|
790
|
+
Gets content of current page.
|
|
791
|
+
|
|
792
|
+
#### Returns
|
|
793
|
+
|
|
794
|
+
`Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
795
|
+
|
|
796
|
+
Promise resolving to fetch response
|
|
797
|
+
|
|
798
|
+
#### Throws
|
|
799
|
+
|
|
800
|
+
When no content has been fetched yet
|
|
801
|
+
|
|
802
|
+
***
|
|
803
|
+
|
|
804
|
+
### goto()
|
|
805
|
+
|
|
806
|
+
> `abstract` **goto**(`url`, `params?`): `Promise`\<`void` \| [`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
807
|
+
|
|
808
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:340](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L340)
|
|
809
|
+
|
|
810
|
+
Navigates to the specified URL.
|
|
811
|
+
|
|
812
|
+
#### Parameters
|
|
813
|
+
|
|
814
|
+
##### url
|
|
815
|
+
|
|
816
|
+
`string`
|
|
817
|
+
|
|
818
|
+
Target URL
|
|
819
|
+
|
|
820
|
+
##### params?
|
|
821
|
+
|
|
822
|
+
[`GotoActionOptions`](../interfaces/GotoActionOptions.md)
|
|
823
|
+
|
|
824
|
+
Navigation options
|
|
825
|
+
|
|
826
|
+
#### Returns
|
|
827
|
+
|
|
828
|
+
`Promise`\<`void` \| [`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
829
|
+
|
|
830
|
+
Promise resolving when navigation completes
|
|
831
|
+
|
|
832
|
+
#### Example
|
|
833
|
+
|
|
834
|
+
```ts
|
|
835
|
+
await engine.goto('https://example.com');
|
|
836
|
+
```
|
|
837
|
+
|
|
838
|
+
***
|
|
839
|
+
|
|
840
|
+
### headers()
|
|
841
|
+
|
|
842
|
+
#### Call Signature
|
|
843
|
+
|
|
844
|
+
> **headers**(): `Promise`\<`Record`\<`string`, `string`\>\>
|
|
845
|
+
|
|
846
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:753](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L753)
|
|
847
|
+
|
|
848
|
+
Manages HTTP headers for requests with multiple overloads.
|
|
849
|
+
|
|
850
|
+
Gets all headers.
|
|
851
|
+
|
|
852
|
+
##### Returns
|
|
853
|
+
|
|
854
|
+
`Promise`\<`Record`\<`string`, `string`\>\>
|
|
855
|
+
|
|
856
|
+
All headers as record
|
|
857
|
+
|
|
858
|
+
Gets specific header value.
|
|
859
|
+
|
|
860
|
+
##### Example
|
|
861
|
+
|
|
862
|
+
```ts
|
|
863
|
+
const allHeaders = await engine.headers();
|
|
864
|
+
const userAgent = await engine.headers('user-agent');
|
|
865
|
+
await engine.headers({ 'x-custom': 'value' });
|
|
866
|
+
await engine.headers('auth', 'token');
|
|
867
|
+
```
|
|
868
|
+
|
|
869
|
+
#### Call Signature
|
|
870
|
+
|
|
871
|
+
> **headers**(`name`): `Promise`\<`string`\>
|
|
872
|
+
|
|
873
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:754](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L754)
|
|
874
|
+
|
|
875
|
+
Manages HTTP headers for requests with multiple overloads.
|
|
876
|
+
|
|
877
|
+
Gets all headers.
|
|
878
|
+
|
|
879
|
+
##### Parameters
|
|
880
|
+
|
|
881
|
+
###### name
|
|
882
|
+
|
|
883
|
+
`string`
|
|
884
|
+
|
|
885
|
+
Header name
|
|
886
|
+
|
|
887
|
+
##### Returns
|
|
888
|
+
|
|
889
|
+
`Promise`\<`string`\>
|
|
890
|
+
|
|
891
|
+
All headers as record
|
|
892
|
+
|
|
893
|
+
Gets specific header value.
|
|
894
|
+
|
|
895
|
+
##### Example
|
|
896
|
+
|
|
897
|
+
```ts
|
|
898
|
+
const allHeaders = await engine.headers();
|
|
899
|
+
const userAgent = await engine.headers('user-agent');
|
|
900
|
+
await engine.headers({ 'x-custom': 'value' });
|
|
901
|
+
await engine.headers('auth', 'token');
|
|
902
|
+
```
|
|
903
|
+
|
|
904
|
+
#### Call Signature
|
|
905
|
+
|
|
906
|
+
> **headers**(`headers`, `replaced?`): `Promise`\<`boolean`\>
|
|
907
|
+
|
|
908
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:755](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L755)
|
|
909
|
+
|
|
910
|
+
Manages HTTP headers for requests with multiple overloads.
|
|
911
|
+
|
|
912
|
+
Gets all headers.
|
|
913
|
+
|
|
914
|
+
##### Parameters
|
|
915
|
+
|
|
916
|
+
###### headers
|
|
917
|
+
|
|
918
|
+
`Record`\<`string`, `string`\>
|
|
919
|
+
|
|
920
|
+
Headers to set
|
|
921
|
+
|
|
922
|
+
###### replaced?
|
|
923
|
+
|
|
924
|
+
`boolean`
|
|
925
|
+
|
|
926
|
+
Whether to replace all existing headers
|
|
927
|
+
|
|
928
|
+
##### Returns
|
|
929
|
+
|
|
930
|
+
`Promise`\<`boolean`\>
|
|
931
|
+
|
|
932
|
+
All headers as record
|
|
933
|
+
|
|
934
|
+
Gets specific header value.
|
|
935
|
+
|
|
936
|
+
##### Example
|
|
937
|
+
|
|
938
|
+
```ts
|
|
939
|
+
const allHeaders = await engine.headers();
|
|
940
|
+
const userAgent = await engine.headers('user-agent');
|
|
941
|
+
await engine.headers({ 'x-custom': 'value' });
|
|
942
|
+
await engine.headers('auth', 'token');
|
|
943
|
+
```
|
|
944
|
+
|
|
945
|
+
#### Call Signature
|
|
946
|
+
|
|
947
|
+
> **headers**(`name`, `value`): `Promise`\<`boolean`\>
|
|
948
|
+
|
|
949
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:756](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L756)
|
|
950
|
+
|
|
951
|
+
Manages HTTP headers for requests with multiple overloads.
|
|
952
|
+
|
|
953
|
+
Gets all headers.
|
|
954
|
+
|
|
955
|
+
##### Parameters
|
|
956
|
+
|
|
957
|
+
###### name
|
|
958
|
+
|
|
959
|
+
`string`
|
|
960
|
+
|
|
961
|
+
Header name
|
|
962
|
+
|
|
963
|
+
###### value
|
|
964
|
+
|
|
965
|
+
Header value or `null` to remove
|
|
966
|
+
|
|
967
|
+
`null` | `string`
|
|
968
|
+
|
|
969
|
+
##### Returns
|
|
970
|
+
|
|
971
|
+
`Promise`\<`boolean`\>
|
|
972
|
+
|
|
973
|
+
All headers as record
|
|
974
|
+
|
|
975
|
+
Gets specific header value.
|
|
976
|
+
|
|
977
|
+
##### Example
|
|
978
|
+
|
|
979
|
+
```ts
|
|
980
|
+
const allHeaders = await engine.headers();
|
|
981
|
+
const userAgent = await engine.headers('user-agent');
|
|
982
|
+
await engine.headers({ 'x-custom': 'value' });
|
|
983
|
+
await engine.headers('auth', 'token');
|
|
984
|
+
```
|
|
985
|
+
|
|
986
|
+
***
|
|
987
|
+
|
|
988
|
+
### initialize()
|
|
989
|
+
|
|
990
|
+
> **initialize**(`context`, `options?`): `Promise`\<`void`\>
|
|
991
|
+
|
|
992
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:488](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L488)
|
|
993
|
+
|
|
994
|
+
Initializes the fetch engine with provided context and options.
|
|
995
|
+
|
|
996
|
+
#### Parameters
|
|
997
|
+
|
|
998
|
+
##### context
|
|
999
|
+
|
|
1000
|
+
[`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
1001
|
+
|
|
1002
|
+
Fetch engine context
|
|
1003
|
+
|
|
1004
|
+
##### options?
|
|
1005
|
+
|
|
1006
|
+
[`BaseFetcherProperties`](../interfaces/BaseFetcherProperties.md)
|
|
1007
|
+
|
|
1008
|
+
Configuration options
|
|
1009
|
+
|
|
1010
|
+
#### Returns
|
|
1011
|
+
|
|
1012
|
+
`Promise`\<`void`\>
|
|
1013
|
+
|
|
1014
|
+
Promise resolving when initialization completes
|
|
1015
|
+
|
|
1016
|
+
#### Remarks
|
|
1017
|
+
|
|
1018
|
+
Sets up internal state and calls implementation-specific [_initialize](file:///home/riceball/Documents/mywork/public/@isdk/ai-tools/packages/web-fetcher/src/engine/cheerio.ts#L169-L204) method.
|
|
1019
|
+
Automatically called when creating engine via `FetchEngine.create()`.
|
|
1020
|
+
|
|
1021
|
+
***
|
|
1022
|
+
|
|
1023
|
+
### pause()
|
|
1024
|
+
|
|
1025
|
+
> **pause**(`message?`): `Promise`\<`void`\>
|
|
1026
|
+
|
|
1027
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:400](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L400)
|
|
1028
|
+
|
|
1029
|
+
Pauses execution, allowing for manual intervention or inspection.
|
|
1030
|
+
|
|
1031
|
+
#### Parameters
|
|
1032
|
+
|
|
1033
|
+
##### message?
|
|
1034
|
+
|
|
1035
|
+
`string`
|
|
1036
|
+
|
|
1037
|
+
Optional message to display during pause
|
|
1038
|
+
|
|
1039
|
+
#### Returns
|
|
1040
|
+
|
|
1041
|
+
`Promise`\<`void`\>
|
|
1042
|
+
|
|
1043
|
+
Promise resolving when execution is resumed
|
|
1044
|
+
|
|
1045
|
+
#### Throws
|
|
1046
|
+
|
|
1047
|
+
When no active page context exists
|
|
1048
|
+
|
|
1049
|
+
***
|
|
1050
|
+
|
|
1051
|
+
### submit()
|
|
1052
|
+
|
|
1053
|
+
> **submit**(`selector?`, `options?`): `Promise`\<`void`\>
|
|
1054
|
+
|
|
1055
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:389](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L389)
|
|
1056
|
+
|
|
1057
|
+
Submits a form.
|
|
1058
|
+
|
|
1059
|
+
#### Parameters
|
|
1060
|
+
|
|
1061
|
+
##### selector?
|
|
1062
|
+
|
|
1063
|
+
`any`
|
|
1064
|
+
|
|
1065
|
+
Optional form/submit button selector
|
|
1066
|
+
|
|
1067
|
+
##### options?
|
|
1068
|
+
|
|
1069
|
+
[`SubmitActionOptions`](../interfaces/SubmitActionOptions.md)
|
|
1070
|
+
|
|
1071
|
+
Submission options
|
|
1072
|
+
|
|
1073
|
+
#### Returns
|
|
1074
|
+
|
|
1075
|
+
`Promise`\<`void`\>
|
|
1076
|
+
|
|
1077
|
+
Promise resolving when form is submitted
|
|
1078
|
+
|
|
1079
|
+
#### Throws
|
|
1080
|
+
|
|
1081
|
+
When no active page context exists
|
|
1082
|
+
|
|
1083
|
+
***
|
|
1084
|
+
|
|
1085
|
+
### waitFor()
|
|
1086
|
+
|
|
1087
|
+
> **waitFor**(`params?`): `Promise`\<`void`\>
|
|
1088
|
+
|
|
1089
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:354](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L354)
|
|
1090
|
+
|
|
1091
|
+
Waits for specified condition before continuing.
|
|
1092
|
+
|
|
1093
|
+
#### Parameters
|
|
1094
|
+
|
|
1095
|
+
##### params?
|
|
1096
|
+
|
|
1097
|
+
[`WaitForActionOptions`](../interfaces/WaitForActionOptions.md)
|
|
1098
|
+
|
|
1099
|
+
Wait conditions
|
|
1100
|
+
|
|
1101
|
+
#### Returns
|
|
1102
|
+
|
|
1103
|
+
`Promise`\<`void`\>
|
|
1104
|
+
|
|
1105
|
+
Promise resolving when wait condition is met
|
|
1106
|
+
|
|
1107
|
+
#### Example
|
|
1108
|
+
|
|
1109
|
+
```ts
|
|
1110
|
+
await engine.waitFor({ ms: 1000 }); // Wait 1 second
|
|
1111
|
+
await engine.waitFor({ selector: '#content' }); // Wait for element
|
|
1112
|
+
```
|
|
1113
|
+
|
|
1114
|
+
***
|
|
1115
|
+
|
|
1116
|
+
### create()
|
|
1117
|
+
|
|
1118
|
+
> `static` **create**(`ctx`, `options?`): `Promise`\<`undefined` \| `AnyFetchEngine`\>
|
|
1119
|
+
|
|
1120
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:198](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L198)
|
|
1121
|
+
|
|
1122
|
+
Factory method to create and initialize a fetch engine instance.
|
|
1123
|
+
|
|
1124
|
+
#### Parameters
|
|
1125
|
+
|
|
1126
|
+
##### ctx
|
|
1127
|
+
|
|
1128
|
+
[`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
1129
|
+
|
|
1130
|
+
Fetch engine context
|
|
1131
|
+
|
|
1132
|
+
##### options?
|
|
1133
|
+
|
|
1134
|
+
[`BaseFetcherProperties`](../interfaces/BaseFetcherProperties.md)
|
|
1135
|
+
|
|
1136
|
+
Configuration options
|
|
1137
|
+
|
|
1138
|
+
#### Returns
|
|
1139
|
+
|
|
1140
|
+
`Promise`\<`undefined` \| `AnyFetchEngine`\>
|
|
1141
|
+
|
|
1142
|
+
Initialized fetch engine instance
|
|
1143
|
+
|
|
1144
|
+
#### Throws
|
|
1145
|
+
|
|
1146
|
+
When no suitable engine implementation is found
|
|
1147
|
+
|
|
1148
|
+
#### Remarks
|
|
1149
|
+
|
|
1150
|
+
Primary entry point for engine creation. Selects appropriate implementation based on `engine` name of the option or context.
|
|
1151
|
+
|
|
1152
|
+
***
|
|
1153
|
+
|
|
1154
|
+
### get()
|
|
1155
|
+
|
|
1156
|
+
> `static` **get**(`id`): `undefined` \| `AnyFetchEngineCtor`
|
|
1157
|
+
|
|
1158
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:171](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L171)
|
|
1159
|
+
|
|
1160
|
+
Retrieves a fetch engine implementation by its unique ID.
|
|
1161
|
+
|
|
1162
|
+
#### Parameters
|
|
1163
|
+
|
|
1164
|
+
##### id
|
|
1165
|
+
|
|
1166
|
+
`string`
|
|
1167
|
+
|
|
1168
|
+
The ID of the engine to retrieve
|
|
1169
|
+
|
|
1170
|
+
#### Returns
|
|
1171
|
+
|
|
1172
|
+
`undefined` \| `AnyFetchEngineCtor`
|
|
1173
|
+
|
|
1174
|
+
Engine class if found, otherwise `undefined`
|
|
1175
|
+
|
|
1176
|
+
***
|
|
1177
|
+
|
|
1178
|
+
### getByMode()
|
|
1179
|
+
|
|
1180
|
+
> `static` **getByMode**(`mode`): `undefined` \| `AnyFetchEngineCtor`
|
|
1181
|
+
|
|
1182
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:181](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L181)
|
|
1183
|
+
|
|
1184
|
+
Retrieves a fetch engine implementation by execution mode.
|
|
1185
|
+
|
|
1186
|
+
#### Parameters
|
|
1187
|
+
|
|
1188
|
+
##### mode
|
|
1189
|
+
|
|
1190
|
+
[`FetchEngineType`](../type-aliases/FetchEngineType.md)
|
|
1191
|
+
|
|
1192
|
+
Execution mode (`'http'` or `'browser'`)
|
|
1193
|
+
|
|
1194
|
+
#### Returns
|
|
1195
|
+
|
|
1196
|
+
`undefined` \| `AnyFetchEngineCtor`
|
|
1197
|
+
|
|
1198
|
+
Engine class if found, otherwise `undefined`
|
|
1199
|
+
|
|
1200
|
+
***
|
|
1201
|
+
|
|
1202
|
+
### register()
|
|
1203
|
+
|
|
1204
|
+
> `static` **register**(`engineClass`): `void`
|
|
1205
|
+
|
|
1206
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:158](https://github.com/isdk/web-fetcher.js/blob/ef3a4e527bf3ca75213d55b02c017b9a30a036fd/src/engine/base.ts#L158)
|
|
1207
|
+
|
|
1208
|
+
Registers a fetch engine implementation with the global registry.
|
|
1209
|
+
|
|
1210
|
+
#### Parameters
|
|
1211
|
+
|
|
1212
|
+
##### engineClass
|
|
1213
|
+
|
|
1214
|
+
`AnyFetchEngineCtor`
|
|
1215
|
+
|
|
1216
|
+
The engine class to register
|
|
1217
|
+
|
|
1218
|
+
#### Returns
|
|
1219
|
+
|
|
1220
|
+
`void`
|
|
1221
|
+
|
|
1222
|
+
#### Throws
|
|
1223
|
+
|
|
1224
|
+
When engine class lacks static `id` or ID is already registered
|
|
1225
|
+
|
|
1226
|
+
#### Example
|
|
1227
|
+
|
|
1228
|
+
```ts
|
|
1229
|
+
FetchEngine.register(CheerioFetchEngine);
|
|
1230
|
+
```
|