@isdk/web-fetcher 0.2.1 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.action.cn.md +32 -24
- package/README.action.md +14 -4
- package/README.cn.md +10 -2
- package/README.hackernews.md +52 -0
- package/README.md +10 -2
- package/dist/index.d.mts +5 -3
- package/dist/index.d.ts +5 -3
- package/dist/index.js +1 -1
- package/dist/index.mjs +1 -1
- package/docs/README.md +10 -2
- package/docs/_media/README.action.md +14 -4
- package/docs/_media/README.cn.md +10 -2
- package/docs/classes/CheerioFetchEngine.md +91 -69
- package/docs/classes/ClickAction.md +23 -23
- package/docs/classes/ExtractAction.md +23 -23
- package/docs/classes/FetchAction.md +23 -23
- package/docs/classes/FetchEngine.md +87 -69
- package/docs/classes/FetchSession.md +8 -8
- package/docs/classes/FillAction.md +23 -23
- package/docs/classes/GetContentAction.md +23 -23
- package/docs/classes/GotoAction.md +23 -23
- package/docs/classes/PauseAction.md +23 -23
- package/docs/classes/PlaywrightFetchEngine.md +91 -69
- package/docs/classes/SubmitAction.md +23 -23
- package/docs/classes/WaitForAction.md +23 -23
- package/docs/classes/WebFetcher.md +5 -5
- package/docs/enumerations/FetchActionResultStatus.md +4 -4
- package/docs/functions/fetchWeb.md +2 -2
- package/docs/interfaces/BaseFetchActionProperties.md +9 -9
- package/docs/interfaces/BaseFetchCollectorActionProperties.md +13 -13
- package/docs/interfaces/BaseFetcherProperties.md +29 -21
- package/docs/interfaces/DispatchedEngineAction.md +4 -4
- package/docs/interfaces/ExtractActionProperties.md +9 -9
- package/docs/interfaces/FetchActionInContext.md +13 -13
- package/docs/interfaces/FetchActionProperties.md +10 -10
- package/docs/interfaces/FetchActionResult.md +6 -6
- package/docs/interfaces/FetchContext.md +43 -31
- package/docs/interfaces/FetchEngineContext.md +38 -26
- package/docs/interfaces/FetchMetadata.md +5 -5
- package/docs/interfaces/FetchResponse.md +13 -13
- package/docs/interfaces/FetchReturnTypeRegistry.md +7 -7
- package/docs/interfaces/FetchSite.md +36 -24
- package/docs/interfaces/FetcherOptions.md +35 -23
- package/docs/interfaces/GotoActionOptions.md +6 -6
- package/docs/interfaces/PendingEngineRequest.md +3 -3
- package/docs/interfaces/SubmitActionOptions.md +2 -2
- package/docs/interfaces/WaitForActionOptions.md +4 -4
- package/docs/type-aliases/BaseFetchActionOptions.md +1 -1
- package/docs/type-aliases/BaseFetchCollectorOptions.md +1 -1
- package/docs/type-aliases/BrowserEngine.md +1 -1
- package/docs/type-aliases/FetchActionCapabilities.md +1 -1
- package/docs/type-aliases/FetchActionCapabilityMode.md +1 -1
- package/docs/type-aliases/FetchActionOptions.md +1 -1
- package/docs/type-aliases/FetchEngineAction.md +1 -1
- package/docs/type-aliases/FetchEngineType.md +1 -1
- package/docs/type-aliases/FetchReturnType.md +1 -1
- package/docs/type-aliases/FetchReturnTypeFor.md +1 -1
- package/docs/type-aliases/OnFetchPauseCallback.md +1 -1
- package/docs/type-aliases/ResourceType.md +1 -1
- package/docs/variables/DefaultFetcherProperties.md +1 -1
- package/package.json +2 -1
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Class: CheerioFetchEngine
|
|
8
8
|
|
|
9
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:20](https://github.com/isdk/web-fetcher.js/blob/
|
|
9
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:20](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L20)
|
|
10
10
|
|
|
11
11
|
## Extends
|
|
12
12
|
|
|
@@ -32,7 +32,7 @@ Defined in: [packages/web-fetcher/src/engine/cheerio.ts:20](https://github.com/i
|
|
|
32
32
|
|
|
33
33
|
> `protected` **actionEmitter**: `EventEmitter`
|
|
34
34
|
|
|
35
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:235](https://github.com/isdk/web-fetcher.js/blob/
|
|
35
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:235](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L235)
|
|
36
36
|
|
|
37
37
|
#### Inherited from
|
|
38
38
|
|
|
@@ -44,7 +44,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:235](https://github.com/isd
|
|
|
44
44
|
|
|
45
45
|
> `protected` **blockedTypes**: `Set`\<`string`\>
|
|
46
46
|
|
|
47
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:239](https://github.com/isdk/web-fetcher.js/blob/
|
|
47
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:239](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L239)
|
|
48
48
|
|
|
49
49
|
#### Inherited from
|
|
50
50
|
|
|
@@ -56,7 +56,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:239](https://github.com/isd
|
|
|
56
56
|
|
|
57
57
|
> `protected` `optional` **crawler**: `CheerioCrawler`
|
|
58
58
|
|
|
59
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:227](https://github.com/isdk/web-fetcher.js/blob/
|
|
59
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:227](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L227)
|
|
60
60
|
|
|
61
61
|
#### Inherited from
|
|
62
62
|
|
|
@@ -68,7 +68,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:227](https://github.com/isd
|
|
|
68
68
|
|
|
69
69
|
> `protected` `optional` **ctx**: [`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
70
70
|
|
|
71
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:225](https://github.com/isdk/web-fetcher.js/blob/
|
|
71
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:225](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L225)
|
|
72
72
|
|
|
73
73
|
#### Inherited from
|
|
74
74
|
|
|
@@ -80,7 +80,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:225](https://github.com/isd
|
|
|
80
80
|
|
|
81
81
|
> `protected` **hdrs**: `Record`\<`string`, `string`\> = `{}`
|
|
82
82
|
|
|
83
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:231](https://github.com/isdk/web-fetcher.js/blob/
|
|
83
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:231](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L231)
|
|
84
84
|
|
|
85
85
|
#### Inherited from
|
|
86
86
|
|
|
@@ -92,7 +92,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:231](https://github.com/isd
|
|
|
92
92
|
|
|
93
93
|
> `protected` `optional` **isCrawlerReady**: `boolean`
|
|
94
94
|
|
|
95
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:228](https://github.com/isdk/web-fetcher.js/blob/
|
|
95
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:228](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L228)
|
|
96
96
|
|
|
97
97
|
#### Inherited from
|
|
98
98
|
|
|
@@ -104,7 +104,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:228](https://github.com/isd
|
|
|
104
104
|
|
|
105
105
|
> `protected` **isPageActive**: `boolean` = `false`
|
|
106
106
|
|
|
107
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:236](https://github.com/isdk/web-fetcher.js/blob/
|
|
107
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:236](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L236)
|
|
108
108
|
|
|
109
109
|
#### Inherited from
|
|
110
110
|
|
|
@@ -116,7 +116,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:236](https://github.com/isd
|
|
|
116
116
|
|
|
117
117
|
> `protected` **jar**: [`Cookie`](../interfaces/Cookie.md)[] = `[]`
|
|
118
118
|
|
|
119
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:232](https://github.com/isdk/web-fetcher.js/blob/
|
|
119
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:232](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L232)
|
|
120
120
|
|
|
121
121
|
#### Inherited from
|
|
122
122
|
|
|
@@ -128,7 +128,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:232](https://github.com/isd
|
|
|
128
128
|
|
|
129
129
|
> `protected` `optional` **lastResponse**: [`FetchResponse`](../interfaces/FetchResponse.md)
|
|
130
130
|
|
|
131
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:238](https://github.com/isdk/web-fetcher.js/blob/
|
|
131
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:238](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L238)
|
|
132
132
|
|
|
133
133
|
#### Inherited from
|
|
134
134
|
|
|
@@ -140,7 +140,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:238](https://github.com/isd
|
|
|
140
140
|
|
|
141
141
|
> `protected` **navigationLock**: `PromiseLock`
|
|
142
142
|
|
|
143
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:237](https://github.com/isdk/web-fetcher.js/blob/
|
|
143
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:237](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L237)
|
|
144
144
|
|
|
145
145
|
#### Inherited from
|
|
146
146
|
|
|
@@ -152,7 +152,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:237](https://github.com/isd
|
|
|
152
152
|
|
|
153
153
|
> `protected` `optional` **opts**: [`BaseFetcherProperties`](../interfaces/BaseFetcherProperties.md)
|
|
154
154
|
|
|
155
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:226](https://github.com/isdk/web-fetcher.js/blob/
|
|
155
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:226](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L226)
|
|
156
156
|
|
|
157
157
|
#### Inherited from
|
|
158
158
|
|
|
@@ -164,7 +164,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:226](https://github.com/isd
|
|
|
164
164
|
|
|
165
165
|
> `protected` **pendingRequests**: `Map`\<`string`, [`PendingEngineRequest`](../interfaces/PendingEngineRequest.md)\>
|
|
166
166
|
|
|
167
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:233](https://github.com/isdk/web-fetcher.js/blob/
|
|
167
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:233](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L233)
|
|
168
168
|
|
|
169
169
|
#### Inherited from
|
|
170
170
|
|
|
@@ -176,7 +176,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:233](https://github.com/isd
|
|
|
176
176
|
|
|
177
177
|
> `protected` **requestCounter**: `number` = `0`
|
|
178
178
|
|
|
179
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:234](https://github.com/isdk/web-fetcher.js/blob/
|
|
179
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:234](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L234)
|
|
180
180
|
|
|
181
181
|
#### Inherited from
|
|
182
182
|
|
|
@@ -188,7 +188,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:234](https://github.com/isd
|
|
|
188
188
|
|
|
189
189
|
> `protected` `optional` **requestQueue**: `RequestQueue`
|
|
190
190
|
|
|
191
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:229](https://github.com/isdk/web-fetcher.js/blob/
|
|
191
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:229](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L229)
|
|
192
192
|
|
|
193
193
|
#### Inherited from
|
|
194
194
|
|
|
@@ -200,7 +200,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:229](https://github.com/isd
|
|
|
200
200
|
|
|
201
201
|
> `readonly` `static` **id**: `"cheerio"` = `'cheerio'`
|
|
202
202
|
|
|
203
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:25](https://github.com/isdk/web-fetcher.js/blob/
|
|
203
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:25](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L25)
|
|
204
204
|
|
|
205
205
|
Unique identifier for the engine implementation.
|
|
206
206
|
|
|
@@ -218,7 +218,7 @@ Must be defined by concrete implementations. Used for registration and lookup in
|
|
|
218
218
|
|
|
219
219
|
> `readonly` `static` **mode**: `"http"` = `'http'`
|
|
220
220
|
|
|
221
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:26](https://github.com/isdk/web-fetcher.js/blob/
|
|
221
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:26](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L26)
|
|
222
222
|
|
|
223
223
|
Execution mode of the engine (`'http'` or `'browser'`).
|
|
224
224
|
|
|
@@ -238,7 +238,7 @@ Must be defined by concrete implementations. Indicates whether engine operates a
|
|
|
238
238
|
|
|
239
239
|
> **get** **context**(): `undefined` \| [`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
240
240
|
|
|
241
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
241
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:480](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L480)
|
|
242
242
|
|
|
243
243
|
Gets the fetch engine context associated with this instance.
|
|
244
244
|
|
|
@@ -258,7 +258,7 @@ Gets the fetch engine context associated with this instance.
|
|
|
258
258
|
|
|
259
259
|
> **get** **id**(): `string`
|
|
260
260
|
|
|
261
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
261
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:466](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L466)
|
|
262
262
|
|
|
263
263
|
Gets the unique identifier of this engine implementation.
|
|
264
264
|
|
|
@@ -278,7 +278,7 @@ Gets the unique identifier of this engine implementation.
|
|
|
278
278
|
|
|
279
279
|
> **get** **mode**(): [`FetchEngineType`](../type-aliases/FetchEngineType.md)
|
|
280
280
|
|
|
281
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
281
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:473](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L473)
|
|
282
282
|
|
|
283
283
|
Gets the execution mode of this engine (`'http'` or `'browser'`).
|
|
284
284
|
|
|
@@ -292,11 +292,45 @@ Gets the execution mode of this engine (`'http'` or `'browser'`).
|
|
|
292
292
|
|
|
293
293
|
## Methods
|
|
294
294
|
|
|
295
|
+
### \_buildResponse()
|
|
296
|
+
|
|
297
|
+
> `protected` **\_buildResponse**(`context`): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
298
|
+
|
|
299
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:28](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L28)
|
|
300
|
+
|
|
301
|
+
**`Internal`**
|
|
302
|
+
|
|
303
|
+
Abstract method for building standard [FetchResponse] from Crawlee context.
|
|
304
|
+
|
|
305
|
+
#### Parameters
|
|
306
|
+
|
|
307
|
+
##### context
|
|
308
|
+
|
|
309
|
+
`CheerioCrawlingContext`
|
|
310
|
+
|
|
311
|
+
Crawlee crawling context
|
|
312
|
+
|
|
313
|
+
#### Returns
|
|
314
|
+
|
|
315
|
+
`Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
316
|
+
|
|
317
|
+
Promise resolving to [FetchResponse] object
|
|
318
|
+
|
|
319
|
+
#### Remarks
|
|
320
|
+
|
|
321
|
+
Converts implementation-specific context (Playwright `page` or Cheerio `$`) to standardized response.
|
|
322
|
+
|
|
323
|
+
#### Overrides
|
|
324
|
+
|
|
325
|
+
[`FetchEngine`](FetchEngine.md).[`_buildResponse`](FetchEngine.md#_buildresponse)
|
|
326
|
+
|
|
327
|
+
***
|
|
328
|
+
|
|
295
329
|
### \_cleanup()?
|
|
296
330
|
|
|
297
331
|
> `protected` `optional` **\_cleanup**(): `Promise`\<`void`\>
|
|
298
332
|
|
|
299
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:241](https://github.com/isdk/web-fetcher.js/blob/
|
|
333
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:241](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L241)
|
|
300
334
|
|
|
301
335
|
#### Returns
|
|
302
336
|
|
|
@@ -312,7 +346,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:241](https://github.com/isd
|
|
|
312
346
|
|
|
313
347
|
> `protected` **\_commonCleanup**(): `Promise`\<`void`\>
|
|
314
348
|
|
|
315
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
349
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:664](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L664)
|
|
316
350
|
|
|
317
351
|
#### Returns
|
|
318
352
|
|
|
@@ -328,7 +362,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:656](https://github.com/isd
|
|
|
328
362
|
|
|
329
363
|
> `protected` **\_createCrawler**(`options`): `CheerioCrawler`
|
|
330
364
|
|
|
331
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:
|
|
365
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:246](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L246)
|
|
332
366
|
|
|
333
367
|
**`Internal`**
|
|
334
368
|
|
|
@@ -356,7 +390,7 @@ The final crawler options.
|
|
|
356
390
|
|
|
357
391
|
> `protected` **\_executePendingActions**(`context`): `Promise`\<`void`\>
|
|
358
392
|
|
|
359
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
393
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:580](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L580)
|
|
360
394
|
|
|
361
395
|
**`Internal`**
|
|
362
396
|
|
|
@@ -409,7 +443,7 @@ If called outside valid page context window (`!this.isPageActive`)
|
|
|
409
443
|
|
|
410
444
|
> `protected` **\_extract**(`schema`, `context`): `Promise`\<`any`\>
|
|
411
445
|
|
|
412
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:246](https://github.com/isdk/web-fetcher.js/blob/
|
|
446
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:246](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L246)
|
|
413
447
|
|
|
414
448
|
#### Parameters
|
|
415
449
|
|
|
@@ -435,7 +469,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:246](https://github.com/isd
|
|
|
435
469
|
|
|
436
470
|
> `protected` **\_extractValue**(`schema`, `context`): `Promise`\<`any`\>
|
|
437
471
|
|
|
438
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:
|
|
472
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:63](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L63)
|
|
439
473
|
|
|
440
474
|
#### Parameters
|
|
441
475
|
|
|
@@ -463,7 +497,7 @@ Defined in: [packages/web-fetcher/src/engine/cheerio.ts:52](https://github.com/i
|
|
|
463
497
|
|
|
464
498
|
> `protected` **\_getSpecificCrawlerOptions**(`ctx`): `CheerioCrawlerOptions`
|
|
465
499
|
|
|
466
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:
|
|
500
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:250](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L250)
|
|
467
501
|
|
|
468
502
|
**`Internal`**
|
|
469
503
|
|
|
@@ -491,7 +525,7 @@ The fetch engine context.
|
|
|
491
525
|
|
|
492
526
|
> `protected` **\_normalizeSchema**(`schema`): `ExtractSchema`
|
|
493
527
|
|
|
494
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
528
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:422](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L422)
|
|
495
529
|
|
|
496
530
|
#### Parameters
|
|
497
531
|
|
|
@@ -513,7 +547,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:415](https://github.com/isd
|
|
|
513
547
|
|
|
514
548
|
> `protected` **\_querySelectorAll**(`context`, `selector`): `Promise`\<`any`[]\>
|
|
515
549
|
|
|
516
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:
|
|
550
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:58](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L58)
|
|
517
551
|
|
|
518
552
|
#### Parameters
|
|
519
553
|
|
|
@@ -545,7 +579,7 @@ Defined in: [packages/web-fetcher/src/engine/cheerio.ts:47](https://github.com/i
|
|
|
545
579
|
|
|
546
580
|
> `protected` **\_sharedFailedRequestHandler**(`context`, `error?`): `Promise`\<`void`\>
|
|
547
581
|
|
|
548
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
582
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:631](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L631)
|
|
549
583
|
|
|
550
584
|
#### Parameters
|
|
551
585
|
|
|
@@ -571,7 +605,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:623](https://github.com/isd
|
|
|
571
605
|
|
|
572
606
|
> `protected` **\_sharedRequestHandler**(`context`): `Promise`\<`void`\>
|
|
573
607
|
|
|
574
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
608
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:603](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L603)
|
|
575
609
|
|
|
576
610
|
#### Parameters
|
|
577
611
|
|
|
@@ -593,7 +627,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:595](https://github.com/isd
|
|
|
593
627
|
|
|
594
628
|
> **blockResources**(`types`, `overwrite?`): `Promise`\<`number`\>
|
|
595
629
|
|
|
596
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
630
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:708](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L708)
|
|
597
631
|
|
|
598
632
|
Blocks specified resource types from loading.
|
|
599
633
|
|
|
@@ -634,11 +668,7 @@ await engine.blockResources(['script'], true); // Replace existing
|
|
|
634
668
|
|
|
635
669
|
> `protected` **buildResponse**(`context`): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
636
670
|
|
|
637
|
-
Defined in: [packages/web-fetcher/src/engine/
|
|
638
|
-
|
|
639
|
-
**`Internal`**
|
|
640
|
-
|
|
641
|
-
Abstract method for building standard [FetchResponse] from Crawlee context.
|
|
671
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:315](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L315)
|
|
642
672
|
|
|
643
673
|
#### Parameters
|
|
644
674
|
|
|
@@ -646,19 +676,11 @@ Abstract method for building standard [FetchResponse] from Crawlee context.
|
|
|
646
676
|
|
|
647
677
|
`CheerioCrawlingContext`
|
|
648
678
|
|
|
649
|
-
Crawlee crawling context
|
|
650
|
-
|
|
651
679
|
#### Returns
|
|
652
680
|
|
|
653
681
|
`Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
654
682
|
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
#### Remarks
|
|
658
|
-
|
|
659
|
-
Converts implementation-specific context (Playwright `page` or Cheerio `$`) to standardized response.
|
|
660
|
-
|
|
661
|
-
#### Overrides
|
|
683
|
+
#### Inherited from
|
|
662
684
|
|
|
663
685
|
[`FetchEngine`](FetchEngine.md).[`buildResponse`](FetchEngine.md#buildresponse)
|
|
664
686
|
|
|
@@ -668,7 +690,7 @@ Converts implementation-specific context (Playwright `page` or Cheerio `$`) to s
|
|
|
668
690
|
|
|
669
691
|
> **cleanup**(): `Promise`\<`void`\>
|
|
670
692
|
|
|
671
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
693
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:544](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L544)
|
|
672
694
|
|
|
673
695
|
#### Returns
|
|
674
696
|
|
|
@@ -684,7 +706,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:536](https://github.com/isd
|
|
|
684
706
|
|
|
685
707
|
> **click**(`selector`): `Promise`\<`void`\>
|
|
686
708
|
|
|
687
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
709
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:372](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L372)
|
|
688
710
|
|
|
689
711
|
Clicks on element matching selector.
|
|
690
712
|
|
|
@@ -718,7 +740,7 @@ When no active page context exists
|
|
|
718
740
|
|
|
719
741
|
> **cookies**(): `Promise`\<[`Cookie`](../interfaces/Cookie.md)[]\>
|
|
720
742
|
|
|
721
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
743
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:819](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L819)
|
|
722
744
|
|
|
723
745
|
Manages cookies for current session with multiple overloads.
|
|
724
746
|
|
|
@@ -747,7 +769,7 @@ await engine.cookies([{ name: 'session', value: '123' }]);
|
|
|
747
769
|
|
|
748
770
|
> **cookies**(`cookies`): `Promise`\<`boolean`\>
|
|
749
771
|
|
|
750
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
772
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:820](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L820)
|
|
751
773
|
|
|
752
774
|
Manages cookies for current session with multiple overloads.
|
|
753
775
|
|
|
@@ -786,7 +808,7 @@ await engine.cookies([{ name: 'session', value: '123' }]);
|
|
|
786
808
|
|
|
787
809
|
> `protected` **dispatchAction**\<`T`\>(`action`): `Promise`\<`T`\>
|
|
788
810
|
|
|
789
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
811
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:647](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L647)
|
|
790
812
|
|
|
791
813
|
#### Type Parameters
|
|
792
814
|
|
|
@@ -814,7 +836,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:639](https://github.com/isd
|
|
|
814
836
|
|
|
815
837
|
> **dispose**(): `Promise`\<`void`\>
|
|
816
838
|
|
|
817
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
839
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:837](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L837)
|
|
818
840
|
|
|
819
841
|
Disposes of engine, cleaning up all resources.
|
|
820
842
|
|
|
@@ -834,7 +856,7 @@ Promise resolving when disposal completes
|
|
|
834
856
|
|
|
835
857
|
> `protected` **executeAction**(`context`, `action`): `Promise`\<`any`\>
|
|
836
858
|
|
|
837
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:
|
|
859
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:91](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L91)
|
|
838
860
|
|
|
839
861
|
**`Internal`**
|
|
840
862
|
|
|
@@ -874,7 +896,7 @@ Handles specific user interactions using underlying technology (Playwright/Cheer
|
|
|
874
896
|
|
|
875
897
|
> **extract**\<`T`\>(`schema`): `Promise`\<`T`\>
|
|
876
898
|
|
|
877
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
899
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:417](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L417)
|
|
878
900
|
|
|
879
901
|
Extracts structured data from the current page content.
|
|
880
902
|
|
|
@@ -908,7 +930,7 @@ A promise that resolves to an object with the extracted data.
|
|
|
908
930
|
|
|
909
931
|
> **fill**(`selector`, `value`): `Promise`\<`void`\>
|
|
910
932
|
|
|
911
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
933
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:384](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L384)
|
|
912
934
|
|
|
913
935
|
Fills input element with specified value.
|
|
914
936
|
|
|
@@ -946,7 +968,7 @@ When no active page context exists
|
|
|
946
968
|
|
|
947
969
|
> **getContent**(): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
948
970
|
|
|
949
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
971
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:722](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L722)
|
|
950
972
|
|
|
951
973
|
Gets content of current page.
|
|
952
974
|
|
|
@@ -970,7 +992,7 @@ When no content has been fetched yet
|
|
|
970
992
|
|
|
971
993
|
> **goto**(`url`, `params?`): `Promise`\<`void` \| [`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
972
994
|
|
|
973
|
-
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:
|
|
995
|
+
Defined in: [packages/web-fetcher/src/engine/cheerio.ts:270](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/cheerio.ts#L270)
|
|
974
996
|
|
|
975
997
|
Navigates to the specified URL.
|
|
976
998
|
|
|
@@ -1012,7 +1034,7 @@ await engine.goto('https://example.com');
|
|
|
1012
1034
|
|
|
1013
1035
|
> **headers**(): `Promise`\<`Record`\<`string`, `string`\>\>
|
|
1014
1036
|
|
|
1015
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1037
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:761](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L761)
|
|
1016
1038
|
|
|
1017
1039
|
Manages HTTP headers for requests with multiple overloads.
|
|
1018
1040
|
|
|
@@ -1043,7 +1065,7 @@ await engine.headers('auth', 'token');
|
|
|
1043
1065
|
|
|
1044
1066
|
> **headers**(`name`): `Promise`\<`string`\>
|
|
1045
1067
|
|
|
1046
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1068
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:762](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L762)
|
|
1047
1069
|
|
|
1048
1070
|
Manages HTTP headers for requests with multiple overloads.
|
|
1049
1071
|
|
|
@@ -1082,7 +1104,7 @@ await engine.headers('auth', 'token');
|
|
|
1082
1104
|
|
|
1083
1105
|
> **headers**(`headers`, `replaced?`): `Promise`\<`boolean`\>
|
|
1084
1106
|
|
|
1085
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1107
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:763](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L763)
|
|
1086
1108
|
|
|
1087
1109
|
Manages HTTP headers for requests with multiple overloads.
|
|
1088
1110
|
|
|
@@ -1127,7 +1149,7 @@ await engine.headers('auth', 'token');
|
|
|
1127
1149
|
|
|
1128
1150
|
> **headers**(`name`, `value`): `Promise`\<`boolean`\>
|
|
1129
1151
|
|
|
1130
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1152
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:764](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L764)
|
|
1131
1153
|
|
|
1132
1154
|
Manages HTTP headers for requests with multiple overloads.
|
|
1133
1155
|
|
|
@@ -1174,7 +1196,7 @@ await engine.headers('auth', 'token');
|
|
|
1174
1196
|
|
|
1175
1197
|
> **initialize**(`context`, `options?`): `Promise`\<`void`\>
|
|
1176
1198
|
|
|
1177
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1199
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:495](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L495)
|
|
1178
1200
|
|
|
1179
1201
|
Initializes the fetch engine with provided context and options.
|
|
1180
1202
|
|
|
@@ -1213,7 +1235,7 @@ Automatically called when creating engine via `FetchEngine.create()`.
|
|
|
1213
1235
|
|
|
1214
1236
|
> **pause**(`message?`): `Promise`\<`void`\>
|
|
1215
1237
|
|
|
1216
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1238
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:407](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L407)
|
|
1217
1239
|
|
|
1218
1240
|
Pauses execution, allowing for manual intervention or inspection.
|
|
1219
1241
|
|
|
@@ -1245,7 +1267,7 @@ When no active page context exists
|
|
|
1245
1267
|
|
|
1246
1268
|
> **submit**(`selector?`, `options?`): `Promise`\<`void`\>
|
|
1247
1269
|
|
|
1248
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1270
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:396](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L396)
|
|
1249
1271
|
|
|
1250
1272
|
Submits a form.
|
|
1251
1273
|
|
|
@@ -1283,7 +1305,7 @@ When no active page context exists
|
|
|
1283
1305
|
|
|
1284
1306
|
> **waitFor**(`params?`): `Promise`\<`void`\>
|
|
1285
1307
|
|
|
1286
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1308
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:361](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L361)
|
|
1287
1309
|
|
|
1288
1310
|
Waits for specified condition before continuing.
|
|
1289
1311
|
|
|
@@ -1318,7 +1340,7 @@ await engine.waitFor({ selector: '#content' }); // Wait for element
|
|
|
1318
1340
|
|
|
1319
1341
|
> `static` **create**(`ctx`, `options?`): `Promise`\<`undefined` \| `AnyFetchEngine`\>
|
|
1320
1342
|
|
|
1321
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:198](https://github.com/isdk/web-fetcher.js/blob/
|
|
1343
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:198](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L198)
|
|
1322
1344
|
|
|
1323
1345
|
Factory method to create and initialize a fetch engine instance.
|
|
1324
1346
|
|
|
@@ -1360,7 +1382,7 @@ Primary entry point for engine creation. Selects appropriate implementation base
|
|
|
1360
1382
|
|
|
1361
1383
|
> `static` **get**(`id`): `undefined` \| `AnyFetchEngineCtor`
|
|
1362
1384
|
|
|
1363
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:171](https://github.com/isdk/web-fetcher.js/blob/
|
|
1385
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:171](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L171)
|
|
1364
1386
|
|
|
1365
1387
|
Retrieves a fetch engine implementation by its unique ID.
|
|
1366
1388
|
|
|
@@ -1388,7 +1410,7 @@ Engine class if found, otherwise `undefined`
|
|
|
1388
1410
|
|
|
1389
1411
|
> `static` **getByMode**(`mode`): `undefined` \| `AnyFetchEngineCtor`
|
|
1390
1412
|
|
|
1391
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:181](https://github.com/isdk/web-fetcher.js/blob/
|
|
1413
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:181](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L181)
|
|
1392
1414
|
|
|
1393
1415
|
Retrieves a fetch engine implementation by execution mode.
|
|
1394
1416
|
|
|
@@ -1416,7 +1438,7 @@ Engine class if found, otherwise `undefined`
|
|
|
1416
1438
|
|
|
1417
1439
|
> `static` **register**(`engineClass`): `void`
|
|
1418
1440
|
|
|
1419
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:158](https://github.com/isdk/web-fetcher.js/blob/
|
|
1441
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:158](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L158)
|
|
1420
1442
|
|
|
1421
1443
|
Registers a fetch engine implementation with the global registry.
|
|
1422
1444
|
|