@isdk/web-fetcher 0.2.1 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.action.cn.md +32 -24
- package/README.action.md +14 -4
- package/README.cn.md +10 -2
- package/README.hackernews.md +52 -0
- package/README.md +10 -2
- package/dist/index.d.mts +5 -3
- package/dist/index.d.ts +5 -3
- package/dist/index.js +1 -1
- package/dist/index.mjs +1 -1
- package/docs/README.md +10 -2
- package/docs/_media/README.action.md +14 -4
- package/docs/_media/README.cn.md +10 -2
- package/docs/classes/CheerioFetchEngine.md +91 -69
- package/docs/classes/ClickAction.md +23 -23
- package/docs/classes/ExtractAction.md +23 -23
- package/docs/classes/FetchAction.md +23 -23
- package/docs/classes/FetchEngine.md +87 -69
- package/docs/classes/FetchSession.md +8 -8
- package/docs/classes/FillAction.md +23 -23
- package/docs/classes/GetContentAction.md +23 -23
- package/docs/classes/GotoAction.md +23 -23
- package/docs/classes/PauseAction.md +23 -23
- package/docs/classes/PlaywrightFetchEngine.md +91 -69
- package/docs/classes/SubmitAction.md +23 -23
- package/docs/classes/WaitForAction.md +23 -23
- package/docs/classes/WebFetcher.md +5 -5
- package/docs/enumerations/FetchActionResultStatus.md +4 -4
- package/docs/functions/fetchWeb.md +2 -2
- package/docs/interfaces/BaseFetchActionProperties.md +9 -9
- package/docs/interfaces/BaseFetchCollectorActionProperties.md +13 -13
- package/docs/interfaces/BaseFetcherProperties.md +29 -21
- package/docs/interfaces/DispatchedEngineAction.md +4 -4
- package/docs/interfaces/ExtractActionProperties.md +9 -9
- package/docs/interfaces/FetchActionInContext.md +13 -13
- package/docs/interfaces/FetchActionProperties.md +10 -10
- package/docs/interfaces/FetchActionResult.md +6 -6
- package/docs/interfaces/FetchContext.md +43 -31
- package/docs/interfaces/FetchEngineContext.md +38 -26
- package/docs/interfaces/FetchMetadata.md +5 -5
- package/docs/interfaces/FetchResponse.md +13 -13
- package/docs/interfaces/FetchReturnTypeRegistry.md +7 -7
- package/docs/interfaces/FetchSite.md +36 -24
- package/docs/interfaces/FetcherOptions.md +35 -23
- package/docs/interfaces/GotoActionOptions.md +6 -6
- package/docs/interfaces/PendingEngineRequest.md +3 -3
- package/docs/interfaces/SubmitActionOptions.md +2 -2
- package/docs/interfaces/WaitForActionOptions.md +4 -4
- package/docs/type-aliases/BaseFetchActionOptions.md +1 -1
- package/docs/type-aliases/BaseFetchCollectorOptions.md +1 -1
- package/docs/type-aliases/BrowserEngine.md +1 -1
- package/docs/type-aliases/FetchActionCapabilities.md +1 -1
- package/docs/type-aliases/FetchActionCapabilityMode.md +1 -1
- package/docs/type-aliases/FetchActionOptions.md +1 -1
- package/docs/type-aliases/FetchEngineAction.md +1 -1
- package/docs/type-aliases/FetchEngineType.md +1 -1
- package/docs/type-aliases/FetchReturnType.md +1 -1
- package/docs/type-aliases/FetchReturnTypeFor.md +1 -1
- package/docs/type-aliases/OnFetchPauseCallback.md +1 -1
- package/docs/type-aliases/ResourceType.md +1 -1
- package/docs/variables/DefaultFetcherProperties.md +1 -1
- package/package.json +2 -1
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Class: PlaywrightFetchEngine
|
|
8
8
|
|
|
9
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:16](https://github.com/isdk/web-fetcher.js/blob/
|
|
9
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:16](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L16)
|
|
10
10
|
|
|
11
11
|
## Extends
|
|
12
12
|
|
|
@@ -32,7 +32,7 @@ Defined in: [packages/web-fetcher/src/engine/playwright.ts:16](https://github.co
|
|
|
32
32
|
|
|
33
33
|
> `protected` **actionEmitter**: `EventEmitter`
|
|
34
34
|
|
|
35
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:235](https://github.com/isdk/web-fetcher.js/blob/
|
|
35
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:235](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L235)
|
|
36
36
|
|
|
37
37
|
#### Inherited from
|
|
38
38
|
|
|
@@ -44,7 +44,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:235](https://github.com/isd
|
|
|
44
44
|
|
|
45
45
|
> `protected` **blockedTypes**: `Set`\<`string`\>
|
|
46
46
|
|
|
47
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:239](https://github.com/isdk/web-fetcher.js/blob/
|
|
47
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:239](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L239)
|
|
48
48
|
|
|
49
49
|
#### Inherited from
|
|
50
50
|
|
|
@@ -56,7 +56,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:239](https://github.com/isd
|
|
|
56
56
|
|
|
57
57
|
> `protected` `optional` **crawler**: `PlaywrightCrawler`
|
|
58
58
|
|
|
59
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:227](https://github.com/isdk/web-fetcher.js/blob/
|
|
59
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:227](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L227)
|
|
60
60
|
|
|
61
61
|
#### Inherited from
|
|
62
62
|
|
|
@@ -68,7 +68,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:227](https://github.com/isd
|
|
|
68
68
|
|
|
69
69
|
> `protected` `optional` **ctx**: [`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
70
70
|
|
|
71
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:225](https://github.com/isdk/web-fetcher.js/blob/
|
|
71
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:225](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L225)
|
|
72
72
|
|
|
73
73
|
#### Inherited from
|
|
74
74
|
|
|
@@ -80,7 +80,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:225](https://github.com/isd
|
|
|
80
80
|
|
|
81
81
|
> `protected` **hdrs**: `Record`\<`string`, `string`\> = `{}`
|
|
82
82
|
|
|
83
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:231](https://github.com/isdk/web-fetcher.js/blob/
|
|
83
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:231](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L231)
|
|
84
84
|
|
|
85
85
|
#### Inherited from
|
|
86
86
|
|
|
@@ -92,7 +92,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:231](https://github.com/isd
|
|
|
92
92
|
|
|
93
93
|
> `protected` `optional` **isCrawlerReady**: `boolean`
|
|
94
94
|
|
|
95
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:228](https://github.com/isdk/web-fetcher.js/blob/
|
|
95
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:228](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L228)
|
|
96
96
|
|
|
97
97
|
#### Inherited from
|
|
98
98
|
|
|
@@ -104,7 +104,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:228](https://github.com/isd
|
|
|
104
104
|
|
|
105
105
|
> `protected` **isPageActive**: `boolean` = `false`
|
|
106
106
|
|
|
107
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:236](https://github.com/isdk/web-fetcher.js/blob/
|
|
107
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:236](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L236)
|
|
108
108
|
|
|
109
109
|
#### Inherited from
|
|
110
110
|
|
|
@@ -116,7 +116,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:236](https://github.com/isd
|
|
|
116
116
|
|
|
117
117
|
> `protected` **jar**: [`Cookie`](../interfaces/Cookie.md)[] = `[]`
|
|
118
118
|
|
|
119
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:232](https://github.com/isdk/web-fetcher.js/blob/
|
|
119
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:232](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L232)
|
|
120
120
|
|
|
121
121
|
#### Inherited from
|
|
122
122
|
|
|
@@ -128,7 +128,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:232](https://github.com/isd
|
|
|
128
128
|
|
|
129
129
|
> `protected` `optional` **lastResponse**: [`FetchResponse`](../interfaces/FetchResponse.md)
|
|
130
130
|
|
|
131
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:238](https://github.com/isdk/web-fetcher.js/blob/
|
|
131
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:238](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L238)
|
|
132
132
|
|
|
133
133
|
#### Inherited from
|
|
134
134
|
|
|
@@ -140,7 +140,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:238](https://github.com/isd
|
|
|
140
140
|
|
|
141
141
|
> `protected` **navigationLock**: `PromiseLock`
|
|
142
142
|
|
|
143
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:237](https://github.com/isdk/web-fetcher.js/blob/
|
|
143
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:237](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L237)
|
|
144
144
|
|
|
145
145
|
#### Inherited from
|
|
146
146
|
|
|
@@ -152,7 +152,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:237](https://github.com/isd
|
|
|
152
152
|
|
|
153
153
|
> `protected` `optional` **opts**: [`BaseFetcherProperties`](../interfaces/BaseFetcherProperties.md)
|
|
154
154
|
|
|
155
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:226](https://github.com/isdk/web-fetcher.js/blob/
|
|
155
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:226](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L226)
|
|
156
156
|
|
|
157
157
|
#### Inherited from
|
|
158
158
|
|
|
@@ -164,7 +164,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:226](https://github.com/isd
|
|
|
164
164
|
|
|
165
165
|
> `protected` **pendingRequests**: `Map`\<`string`, [`PendingEngineRequest`](../interfaces/PendingEngineRequest.md)\>
|
|
166
166
|
|
|
167
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:233](https://github.com/isdk/web-fetcher.js/blob/
|
|
167
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:233](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L233)
|
|
168
168
|
|
|
169
169
|
#### Inherited from
|
|
170
170
|
|
|
@@ -176,7 +176,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:233](https://github.com/isd
|
|
|
176
176
|
|
|
177
177
|
> `protected` **requestCounter**: `number` = `0`
|
|
178
178
|
|
|
179
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:234](https://github.com/isdk/web-fetcher.js/blob/
|
|
179
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:234](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L234)
|
|
180
180
|
|
|
181
181
|
#### Inherited from
|
|
182
182
|
|
|
@@ -188,7 +188,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:234](https://github.com/isd
|
|
|
188
188
|
|
|
189
189
|
> `protected` `optional` **requestQueue**: `RequestQueue`
|
|
190
190
|
|
|
191
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:229](https://github.com/isdk/web-fetcher.js/blob/
|
|
191
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:229](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L229)
|
|
192
192
|
|
|
193
193
|
#### Inherited from
|
|
194
194
|
|
|
@@ -200,7 +200,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:229](https://github.com/isd
|
|
|
200
200
|
|
|
201
201
|
> `readonly` `static` **id**: `"playwright"` = `'playwright'`
|
|
202
202
|
|
|
203
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:21](https://github.com/isdk/web-fetcher.js/blob/
|
|
203
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:21](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L21)
|
|
204
204
|
|
|
205
205
|
Unique identifier for the engine implementation.
|
|
206
206
|
|
|
@@ -218,7 +218,7 @@ Must be defined by concrete implementations. Used for registration and lookup in
|
|
|
218
218
|
|
|
219
219
|
> `readonly` `static` **mode**: `"browser"` = `'browser'`
|
|
220
220
|
|
|
221
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:22](https://github.com/isdk/web-fetcher.js/blob/
|
|
221
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:22](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L22)
|
|
222
222
|
|
|
223
223
|
Execution mode of the engine (`'http'` or `'browser'`).
|
|
224
224
|
|
|
@@ -238,7 +238,7 @@ Must be defined by concrete implementations. Indicates whether engine operates a
|
|
|
238
238
|
|
|
239
239
|
> **get** **context**(): `undefined` \| [`FetchEngineContext`](../interfaces/FetchEngineContext.md)
|
|
240
240
|
|
|
241
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
241
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:480](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L480)
|
|
242
242
|
|
|
243
243
|
Gets the fetch engine context associated with this instance.
|
|
244
244
|
|
|
@@ -258,7 +258,7 @@ Gets the fetch engine context associated with this instance.
|
|
|
258
258
|
|
|
259
259
|
> **get** **id**(): `string`
|
|
260
260
|
|
|
261
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
261
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:466](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L466)
|
|
262
262
|
|
|
263
263
|
Gets the unique identifier of this engine implementation.
|
|
264
264
|
|
|
@@ -278,7 +278,7 @@ Gets the unique identifier of this engine implementation.
|
|
|
278
278
|
|
|
279
279
|
> **get** **mode**(): [`FetchEngineType`](../type-aliases/FetchEngineType.md)
|
|
280
280
|
|
|
281
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
281
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:473](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L473)
|
|
282
282
|
|
|
283
283
|
Gets the execution mode of this engine (`'http'` or `'browser'`).
|
|
284
284
|
|
|
@@ -292,11 +292,45 @@ Gets the execution mode of this engine (`'http'` or `'browser'`).
|
|
|
292
292
|
|
|
293
293
|
## Methods
|
|
294
294
|
|
|
295
|
+
### \_buildResponse()
|
|
296
|
+
|
|
297
|
+
> `protected` **\_buildResponse**(`context`): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
298
|
+
|
|
299
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:24](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L24)
|
|
300
|
+
|
|
301
|
+
**`Internal`**
|
|
302
|
+
|
|
303
|
+
Abstract method for building standard [FetchResponse] from Crawlee context.
|
|
304
|
+
|
|
305
|
+
#### Parameters
|
|
306
|
+
|
|
307
|
+
##### context
|
|
308
|
+
|
|
309
|
+
`PlaywrightCrawlingContext`
|
|
310
|
+
|
|
311
|
+
Crawlee crawling context
|
|
312
|
+
|
|
313
|
+
#### Returns
|
|
314
|
+
|
|
315
|
+
`Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
316
|
+
|
|
317
|
+
Promise resolving to [FetchResponse] object
|
|
318
|
+
|
|
319
|
+
#### Remarks
|
|
320
|
+
|
|
321
|
+
Converts implementation-specific context (Playwright `page` or Cheerio `$`) to standardized response.
|
|
322
|
+
|
|
323
|
+
#### Overrides
|
|
324
|
+
|
|
325
|
+
[`FetchEngine`](FetchEngine.md).[`_buildResponse`](FetchEngine.md#_buildresponse)
|
|
326
|
+
|
|
327
|
+
***
|
|
328
|
+
|
|
295
329
|
### \_cleanup()?
|
|
296
330
|
|
|
297
331
|
> `protected` `optional` **\_cleanup**(): `Promise`\<`void`\>
|
|
298
332
|
|
|
299
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:241](https://github.com/isdk/web-fetcher.js/blob/
|
|
333
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:241](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L241)
|
|
300
334
|
|
|
301
335
|
#### Returns
|
|
302
336
|
|
|
@@ -312,7 +346,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:241](https://github.com/isd
|
|
|
312
346
|
|
|
313
347
|
> `protected` **\_commonCleanup**(): `Promise`\<`void`\>
|
|
314
348
|
|
|
315
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
349
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:664](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L664)
|
|
316
350
|
|
|
317
351
|
#### Returns
|
|
318
352
|
|
|
@@ -328,7 +362,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:656](https://github.com/isd
|
|
|
328
362
|
|
|
329
363
|
> `protected` **\_createCrawler**(`options`): `PlaywrightCrawler`
|
|
330
364
|
|
|
331
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:202](https://github.com/isdk/web-fetcher.js/blob/
|
|
365
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:202](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L202)
|
|
332
366
|
|
|
333
367
|
**`Internal`**
|
|
334
368
|
|
|
@@ -356,7 +390,7 @@ The final crawler options.
|
|
|
356
390
|
|
|
357
391
|
> `protected` **\_executePendingActions**(`context`): `Promise`\<`void`\>
|
|
358
392
|
|
|
359
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
393
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:580](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L580)
|
|
360
394
|
|
|
361
395
|
**`Internal`**
|
|
362
396
|
|
|
@@ -409,7 +443,7 @@ If called outside valid page context window (`!this.isPageActive`)
|
|
|
409
443
|
|
|
410
444
|
> `protected` **\_extract**(`schema`, `context`): `Promise`\<`any`\>
|
|
411
445
|
|
|
412
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:246](https://github.com/isdk/web-fetcher.js/blob/
|
|
446
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:246](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L246)
|
|
413
447
|
|
|
414
448
|
#### Parameters
|
|
415
449
|
|
|
@@ -435,7 +469,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:246](https://github.com/isd
|
|
|
435
469
|
|
|
436
470
|
> `protected` **\_extractValue**(`schema`, `context`): `Promise`\<`any`\>
|
|
437
471
|
|
|
438
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:57](https://github.com/isdk/web-fetcher.js/blob/
|
|
472
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:57](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L57)
|
|
439
473
|
|
|
440
474
|
#### Parameters
|
|
441
475
|
|
|
@@ -461,7 +495,7 @@ Defined in: [packages/web-fetcher/src/engine/playwright.ts:57](https://github.co
|
|
|
461
495
|
|
|
462
496
|
> `protected` **\_getSpecificCrawlerOptions**(`ctx`): `Promise`\<`Partial`\<`PlaywrightCrawlerOptions`\>\>
|
|
463
497
|
|
|
464
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:206](https://github.com/isdk/web-fetcher.js/blob/
|
|
498
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:206](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L206)
|
|
465
499
|
|
|
466
500
|
**`Internal`**
|
|
467
501
|
|
|
@@ -489,7 +523,7 @@ The fetch engine context.
|
|
|
489
523
|
|
|
490
524
|
> `protected` **\_normalizeSchema**(`schema`): `ExtractSchema`
|
|
491
525
|
|
|
492
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
526
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:422](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L422)
|
|
493
527
|
|
|
494
528
|
#### Parameters
|
|
495
529
|
|
|
@@ -511,7 +545,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:415](https://github.com/isd
|
|
|
511
545
|
|
|
512
546
|
> `protected` **\_querySelectorAll**(`context`, `selector`): `Promise`\<`any`[]\>
|
|
513
547
|
|
|
514
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:53](https://github.com/isdk/web-fetcher.js/blob/
|
|
548
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:53](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L53)
|
|
515
549
|
|
|
516
550
|
#### Parameters
|
|
517
551
|
|
|
@@ -537,7 +571,7 @@ Defined in: [packages/web-fetcher/src/engine/playwright.ts:53](https://github.co
|
|
|
537
571
|
|
|
538
572
|
> `protected` **\_sharedFailedRequestHandler**(`context`, `error?`): `Promise`\<`void`\>
|
|
539
573
|
|
|
540
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
574
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:631](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L631)
|
|
541
575
|
|
|
542
576
|
#### Parameters
|
|
543
577
|
|
|
@@ -563,7 +597,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:623](https://github.com/isd
|
|
|
563
597
|
|
|
564
598
|
> `protected` **\_sharedRequestHandler**(`context`): `Promise`\<`void`\>
|
|
565
599
|
|
|
566
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
600
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:603](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L603)
|
|
567
601
|
|
|
568
602
|
#### Parameters
|
|
569
603
|
|
|
@@ -585,7 +619,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:595](https://github.com/isd
|
|
|
585
619
|
|
|
586
620
|
> **blockResources**(`types`, `overwrite?`): `Promise`\<`number`\>
|
|
587
621
|
|
|
588
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
622
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:708](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L708)
|
|
589
623
|
|
|
590
624
|
Blocks specified resource types from loading.
|
|
591
625
|
|
|
@@ -626,11 +660,7 @@ await engine.blockResources(['script'], true); // Replace existing
|
|
|
626
660
|
|
|
627
661
|
> `protected` **buildResponse**(`context`): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
628
662
|
|
|
629
|
-
Defined in: [packages/web-fetcher/src/engine/
|
|
630
|
-
|
|
631
|
-
**`Internal`**
|
|
632
|
-
|
|
633
|
-
Abstract method for building standard [FetchResponse] from Crawlee context.
|
|
663
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:315](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L315)
|
|
634
664
|
|
|
635
665
|
#### Parameters
|
|
636
666
|
|
|
@@ -638,19 +668,11 @@ Abstract method for building standard [FetchResponse] from Crawlee context.
|
|
|
638
668
|
|
|
639
669
|
`PlaywrightCrawlingContext`
|
|
640
670
|
|
|
641
|
-
Crawlee crawling context
|
|
642
|
-
|
|
643
671
|
#### Returns
|
|
644
672
|
|
|
645
673
|
`Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
646
674
|
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
#### Remarks
|
|
650
|
-
|
|
651
|
-
Converts implementation-specific context (Playwright `page` or Cheerio `$`) to standardized response.
|
|
652
|
-
|
|
653
|
-
#### Overrides
|
|
675
|
+
#### Inherited from
|
|
654
676
|
|
|
655
677
|
[`FetchEngine`](FetchEngine.md).[`buildResponse`](FetchEngine.md#buildresponse)
|
|
656
678
|
|
|
@@ -660,7 +682,7 @@ Converts implementation-specific context (Playwright `page` or Cheerio `$`) to s
|
|
|
660
682
|
|
|
661
683
|
> **cleanup**(): `Promise`\<`void`\>
|
|
662
684
|
|
|
663
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
685
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:544](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L544)
|
|
664
686
|
|
|
665
687
|
#### Returns
|
|
666
688
|
|
|
@@ -676,7 +698,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:536](https://github.com/isd
|
|
|
676
698
|
|
|
677
699
|
> **click**(`selector`): `Promise`\<`void`\>
|
|
678
700
|
|
|
679
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
701
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:372](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L372)
|
|
680
702
|
|
|
681
703
|
Clicks on element matching selector.
|
|
682
704
|
|
|
@@ -710,7 +732,7 @@ When no active page context exists
|
|
|
710
732
|
|
|
711
733
|
> **cookies**(): `Promise`\<[`Cookie`](../interfaces/Cookie.md)[]\>
|
|
712
734
|
|
|
713
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
735
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:819](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L819)
|
|
714
736
|
|
|
715
737
|
Manages cookies for current session with multiple overloads.
|
|
716
738
|
|
|
@@ -739,7 +761,7 @@ await engine.cookies([{ name: 'session', value: '123' }]);
|
|
|
739
761
|
|
|
740
762
|
> **cookies**(`cookies`): `Promise`\<`boolean`\>
|
|
741
763
|
|
|
742
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
764
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:820](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L820)
|
|
743
765
|
|
|
744
766
|
Manages cookies for current session with multiple overloads.
|
|
745
767
|
|
|
@@ -778,7 +800,7 @@ await engine.cookies([{ name: 'session', value: '123' }]);
|
|
|
778
800
|
|
|
779
801
|
> `protected` **dispatchAction**\<`T`\>(`action`): `Promise`\<`T`\>
|
|
780
802
|
|
|
781
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
803
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:647](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L647)
|
|
782
804
|
|
|
783
805
|
#### Type Parameters
|
|
784
806
|
|
|
@@ -806,7 +828,7 @@ Defined in: [packages/web-fetcher/src/engine/base.ts:639](https://github.com/isd
|
|
|
806
828
|
|
|
807
829
|
> **dispose**(): `Promise`\<`void`\>
|
|
808
830
|
|
|
809
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
831
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:837](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L837)
|
|
810
832
|
|
|
811
833
|
Disposes of engine, cleaning up all resources.
|
|
812
834
|
|
|
@@ -826,7 +848,7 @@ Promise resolving when disposal completes
|
|
|
826
848
|
|
|
827
849
|
> `protected` **executeAction**(`context`, `action`): `Promise`\<`any`\>
|
|
828
850
|
|
|
829
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:85](https://github.com/isdk/web-fetcher.js/blob/
|
|
851
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:85](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L85)
|
|
830
852
|
|
|
831
853
|
**`Internal`**
|
|
832
854
|
|
|
@@ -866,7 +888,7 @@ Handles specific user interactions using underlying technology (Playwright/Cheer
|
|
|
866
888
|
|
|
867
889
|
> **extract**\<`T`\>(`schema`): `Promise`\<`T`\>
|
|
868
890
|
|
|
869
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
891
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:417](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L417)
|
|
870
892
|
|
|
871
893
|
Extracts structured data from the current page content.
|
|
872
894
|
|
|
@@ -900,7 +922,7 @@ A promise that resolves to an object with the extracted data.
|
|
|
900
922
|
|
|
901
923
|
> **fill**(`selector`, `value`): `Promise`\<`void`\>
|
|
902
924
|
|
|
903
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
925
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:384](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L384)
|
|
904
926
|
|
|
905
927
|
Fills input element with specified value.
|
|
906
928
|
|
|
@@ -938,7 +960,7 @@ When no active page context exists
|
|
|
938
960
|
|
|
939
961
|
> **getContent**(): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
940
962
|
|
|
941
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
963
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:722](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L722)
|
|
942
964
|
|
|
943
965
|
Gets content of current page.
|
|
944
966
|
|
|
@@ -962,7 +984,7 @@ When no content has been fetched yet
|
|
|
962
984
|
|
|
963
985
|
> **goto**(`url`, `opts?`): `Promise`\<[`FetchResponse`](../interfaces/FetchResponse.md)\>
|
|
964
986
|
|
|
965
|
-
Defined in: [packages/web-fetcher/src/engine/playwright.ts:
|
|
987
|
+
Defined in: [packages/web-fetcher/src/engine/playwright.ts:265](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/playwright.ts#L265)
|
|
966
988
|
|
|
967
989
|
Navigates to the specified URL.
|
|
968
990
|
|
|
@@ -1002,7 +1024,7 @@ await engine.goto('https://example.com');
|
|
|
1002
1024
|
|
|
1003
1025
|
> **headers**(): `Promise`\<`Record`\<`string`, `string`\>\>
|
|
1004
1026
|
|
|
1005
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1027
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:761](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L761)
|
|
1006
1028
|
|
|
1007
1029
|
Manages HTTP headers for requests with multiple overloads.
|
|
1008
1030
|
|
|
@@ -1033,7 +1055,7 @@ await engine.headers('auth', 'token');
|
|
|
1033
1055
|
|
|
1034
1056
|
> **headers**(`name`): `Promise`\<`string`\>
|
|
1035
1057
|
|
|
1036
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1058
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:762](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L762)
|
|
1037
1059
|
|
|
1038
1060
|
Manages HTTP headers for requests with multiple overloads.
|
|
1039
1061
|
|
|
@@ -1072,7 +1094,7 @@ await engine.headers('auth', 'token');
|
|
|
1072
1094
|
|
|
1073
1095
|
> **headers**(`headers`, `replaced?`): `Promise`\<`boolean`\>
|
|
1074
1096
|
|
|
1075
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1097
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:763](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L763)
|
|
1076
1098
|
|
|
1077
1099
|
Manages HTTP headers for requests with multiple overloads.
|
|
1078
1100
|
|
|
@@ -1117,7 +1139,7 @@ await engine.headers('auth', 'token');
|
|
|
1117
1139
|
|
|
1118
1140
|
> **headers**(`name`, `value`): `Promise`\<`boolean`\>
|
|
1119
1141
|
|
|
1120
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1142
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:764](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L764)
|
|
1121
1143
|
|
|
1122
1144
|
Manages HTTP headers for requests with multiple overloads.
|
|
1123
1145
|
|
|
@@ -1164,7 +1186,7 @@ await engine.headers('auth', 'token');
|
|
|
1164
1186
|
|
|
1165
1187
|
> **initialize**(`context`, `options?`): `Promise`\<`void`\>
|
|
1166
1188
|
|
|
1167
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1189
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:495](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L495)
|
|
1168
1190
|
|
|
1169
1191
|
Initializes the fetch engine with provided context and options.
|
|
1170
1192
|
|
|
@@ -1203,7 +1225,7 @@ Automatically called when creating engine via `FetchEngine.create()`.
|
|
|
1203
1225
|
|
|
1204
1226
|
> **pause**(`message?`): `Promise`\<`void`\>
|
|
1205
1227
|
|
|
1206
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1228
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:407](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L407)
|
|
1207
1229
|
|
|
1208
1230
|
Pauses execution, allowing for manual intervention or inspection.
|
|
1209
1231
|
|
|
@@ -1235,7 +1257,7 @@ When no active page context exists
|
|
|
1235
1257
|
|
|
1236
1258
|
> **submit**(`selector?`, `options?`): `Promise`\<`void`\>
|
|
1237
1259
|
|
|
1238
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1260
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:396](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L396)
|
|
1239
1261
|
|
|
1240
1262
|
Submits a form.
|
|
1241
1263
|
|
|
@@ -1273,7 +1295,7 @@ When no active page context exists
|
|
|
1273
1295
|
|
|
1274
1296
|
> **waitFor**(`params?`): `Promise`\<`void`\>
|
|
1275
1297
|
|
|
1276
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:
|
|
1298
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:361](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L361)
|
|
1277
1299
|
|
|
1278
1300
|
Waits for specified condition before continuing.
|
|
1279
1301
|
|
|
@@ -1308,7 +1330,7 @@ await engine.waitFor({ selector: '#content' }); // Wait for element
|
|
|
1308
1330
|
|
|
1309
1331
|
> `static` **create**(`ctx`, `options?`): `Promise`\<`undefined` \| `AnyFetchEngine`\>
|
|
1310
1332
|
|
|
1311
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:198](https://github.com/isdk/web-fetcher.js/blob/
|
|
1333
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:198](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L198)
|
|
1312
1334
|
|
|
1313
1335
|
Factory method to create and initialize a fetch engine instance.
|
|
1314
1336
|
|
|
@@ -1350,7 +1372,7 @@ Primary entry point for engine creation. Selects appropriate implementation base
|
|
|
1350
1372
|
|
|
1351
1373
|
> `static` **get**(`id`): `undefined` \| `AnyFetchEngineCtor`
|
|
1352
1374
|
|
|
1353
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:171](https://github.com/isdk/web-fetcher.js/blob/
|
|
1375
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:171](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L171)
|
|
1354
1376
|
|
|
1355
1377
|
Retrieves a fetch engine implementation by its unique ID.
|
|
1356
1378
|
|
|
@@ -1378,7 +1400,7 @@ Engine class if found, otherwise `undefined`
|
|
|
1378
1400
|
|
|
1379
1401
|
> `static` **getByMode**(`mode`): `undefined` \| `AnyFetchEngineCtor`
|
|
1380
1402
|
|
|
1381
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:181](https://github.com/isdk/web-fetcher.js/blob/
|
|
1403
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:181](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L181)
|
|
1382
1404
|
|
|
1383
1405
|
Retrieves a fetch engine implementation by execution mode.
|
|
1384
1406
|
|
|
@@ -1406,7 +1428,7 @@ Engine class if found, otherwise `undefined`
|
|
|
1406
1428
|
|
|
1407
1429
|
> `static` **register**(`engineClass`): `void`
|
|
1408
1430
|
|
|
1409
|
-
Defined in: [packages/web-fetcher/src/engine/base.ts:158](https://github.com/isdk/web-fetcher.js/blob/
|
|
1431
|
+
Defined in: [packages/web-fetcher/src/engine/base.ts:158](https://github.com/isdk/web-fetcher.js/blob/9d976e330f39f712a4e409b1bd1bd44a5fb476bf/src/engine/base.ts#L158)
|
|
1410
1432
|
|
|
1411
1433
|
Registers a fetch engine implementation with the global registry.
|
|
1412
1434
|
|