@isdk/web-searcher 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.cn.md +196 -7
- package/README.md +196 -7
- package/dist/index.d.mts +234 -11
- package/dist/index.d.ts +234 -11
- package/dist/index.js +1 -1
- package/dist/index.mjs +1 -1
- package/docs/README.md +196 -7
- package/docs/classes/GoogleSearcher.md +289 -60
- package/docs/classes/WebSearcher.md +264 -61
- package/docs/functions/extractDate.md +42 -0
- package/docs/functions/extractMetadataFrom.md +40 -0
- package/docs/functions/fetchHeaders.md +34 -0
- package/docs/functions/fetchPartial.md +41 -0
- package/docs/functions/normalizeDate.md +29 -0
- package/docs/functions/parseHeaders.md +28 -0
- package/docs/functions/parseHtml.md +31 -0
- package/docs/functions/testUrlsByLatency.md +42 -0
- package/docs/globals.md +18 -0
- package/docs/interfaces/CustomTimeRange.md +3 -3
- package/docs/interfaces/ExtractOptions.md +54 -0
- package/docs/interfaces/FetchExtractorOptions.md +35 -0
- package/docs/interfaces/FetcherOptions.md +436 -0
- package/docs/interfaces/HtmlData.md +53 -0
- package/docs/interfaces/MetadataResult.md +27 -0
- package/docs/interfaces/PaginationConfig.md +9 -9
- package/docs/interfaces/SearchContext.md +30 -4
- package/docs/interfaces/SearchOptions.md +77 -11
- package/docs/interfaces/StandardSearchResult.md +10 -10
- package/docs/interfaces/VerifiedUrl.md +25 -0
- package/docs/type-aliases/MetadataType.md +13 -0
- package/docs/type-aliases/SafeSearchLevel.md +1 -1
- package/docs/type-aliases/SearchCategory.md +2 -2
- package/docs/type-aliases/SearchTimeRange.md +1 -1
- package/docs/type-aliases/SearchTimeRangePreset.md +1 -1
- package/docs/type-aliases/SearcherConstructor.md +2 -2
- package/package.json +3 -2
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Class: GoogleSearcher
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/engines/google.ts:24](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/engines/google.ts:24](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/engines/google.ts#L24)
|
|
10
10
|
|
|
11
11
|
A sample implementation of a Google Search scraper.
|
|
12
12
|
|
|
@@ -37,7 +37,7 @@ Use this class to understand:
|
|
|
37
37
|
|
|
38
38
|
> **new GoogleSearcher**(`options?`): `GoogleSearcher`
|
|
39
39
|
|
|
40
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
40
|
+
Defined in: web-fetcher/dist/index.d.ts:1172
|
|
41
41
|
|
|
42
42
|
Creates a new FetchSession.
|
|
43
43
|
|
|
@@ -45,7 +45,7 @@ Creates a new FetchSession.
|
|
|
45
45
|
|
|
46
46
|
##### options?
|
|
47
47
|
|
|
48
|
-
`FetcherOptions`
|
|
48
|
+
[`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
49
49
|
|
|
50
50
|
Configuration options for the fetcher.
|
|
51
51
|
|
|
@@ -63,7 +63,7 @@ Configuration options for the fetcher.
|
|
|
63
63
|
|
|
64
64
|
> `protected` **closed**: `boolean`
|
|
65
65
|
|
|
66
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
66
|
+
Defined in: web-fetcher/dist/index.d.ts:1166
|
|
67
67
|
|
|
68
68
|
#### Inherited from
|
|
69
69
|
|
|
@@ -75,7 +75,7 @@ Defined in: web-fetcher/dist/index.d.ts:2281
|
|
|
75
75
|
|
|
76
76
|
> `readonly` **context**: `FetchContext`
|
|
77
77
|
|
|
78
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
78
|
+
Defined in: web-fetcher/dist/index.d.ts:1165
|
|
79
79
|
|
|
80
80
|
The execution context for this session, containing configurations, event bus, and shared state.
|
|
81
81
|
|
|
@@ -89,7 +89,7 @@ The execution context for this session, containing configurations, event bus, an
|
|
|
89
89
|
|
|
90
90
|
> `readonly` **id**: `string`
|
|
91
91
|
|
|
92
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
92
|
+
Defined in: web-fetcher/dist/index.d.ts:1161
|
|
93
93
|
|
|
94
94
|
Unique identifier for the session.
|
|
95
95
|
|
|
@@ -101,9 +101,9 @@ Unique identifier for the session.
|
|
|
101
101
|
|
|
102
102
|
### options
|
|
103
103
|
|
|
104
|
-
> `protected` **options**: `FetcherOptions`
|
|
104
|
+
> `protected` **options**: [`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
105
105
|
|
|
106
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
106
|
+
Defined in: web-fetcher/dist/index.d.ts:1157
|
|
107
107
|
|
|
108
108
|
#### Inherited from
|
|
109
109
|
|
|
@@ -111,11 +111,25 @@ Defined in: web-fetcher/dist/index.d.ts:2272
|
|
|
111
111
|
|
|
112
112
|
***
|
|
113
113
|
|
|
114
|
+
### \_defaultOptions?
|
|
115
|
+
|
|
116
|
+
> `static` `optional` **\_defaultOptions**: [`SearchOptions`](../interfaces/SearchOptions.md)
|
|
117
|
+
|
|
118
|
+
Defined in: [web-searcher/src/searcher.ts:55](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L55)
|
|
119
|
+
|
|
120
|
+
**`Internal`**
|
|
121
|
+
|
|
122
|
+
#### Inherited from
|
|
123
|
+
|
|
124
|
+
[`WebSearcher`](WebSearcher.md).[`_defaultOptions`](WebSearcher.md#_defaultoptions)
|
|
125
|
+
|
|
126
|
+
***
|
|
127
|
+
|
|
114
128
|
### \_isFactory
|
|
115
129
|
|
|
116
130
|
> `static` **\_isFactory**: `boolean` = `false`
|
|
117
131
|
|
|
118
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
132
|
+
Defined in: [web-searcher/src/searcher.ts:34](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L34)
|
|
119
133
|
|
|
120
134
|
#### Inherited from
|
|
121
135
|
|
|
@@ -127,7 +141,7 @@ Defined in: [web-searcher/src/searcher.ts:33](https://github.com/isdk/web-search
|
|
|
127
141
|
|
|
128
142
|
> `static` **alias**: `string`[]
|
|
129
143
|
|
|
130
|
-
Defined in: [web-searcher/src/engines/google.ts:25](https://github.com/isdk/web-searcher.js/blob/
|
|
144
|
+
Defined in: [web-searcher/src/engines/google.ts:25](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/engines/google.ts#L25)
|
|
131
145
|
|
|
132
146
|
Engine alias(es). Can be a single string or an array of strings.
|
|
133
147
|
Useful for registering shorthand names (e.g., 'g' for 'Google').
|
|
@@ -142,7 +156,7 @@ Useful for registering shorthand names (e.g., 'g' for 'Google').
|
|
|
142
156
|
|
|
143
157
|
> `static` **createObject**: (`name`, ...`args`) => [`WebSearcher`](WebSearcher.md)
|
|
144
158
|
|
|
145
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
159
|
+
Defined in: [web-searcher/src/searcher.ts:122](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L122)
|
|
146
160
|
|
|
147
161
|
Creates an instance of the registered search engine.
|
|
148
162
|
|
|
@@ -172,11 +186,39 @@ An instance of the search engine.
|
|
|
172
186
|
|
|
173
187
|
***
|
|
174
188
|
|
|
189
|
+
### currentInstanceIndex?
|
|
190
|
+
|
|
191
|
+
> `static` `optional` **currentInstanceIndex**: `number`
|
|
192
|
+
|
|
193
|
+
Defined in: [web-searcher/src/searcher.ts:52](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L52)
|
|
194
|
+
|
|
195
|
+
Globally shared index for tracking the currently active instance (node) across sessions.
|
|
196
|
+
|
|
197
|
+
#### Inherited from
|
|
198
|
+
|
|
199
|
+
[`WebSearcher`](WebSearcher.md).[`currentInstanceIndex`](WebSearcher.md#currentinstanceindex)
|
|
200
|
+
|
|
201
|
+
***
|
|
202
|
+
|
|
203
|
+
### defaultBaseUrls?
|
|
204
|
+
|
|
205
|
+
> `static` `optional` **defaultBaseUrls**: `string`[]
|
|
206
|
+
|
|
207
|
+
Defined in: [web-searcher/src/searcher.ts:49](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L49)
|
|
208
|
+
|
|
209
|
+
Default base URLs for engines that support multiple instances.
|
|
210
|
+
|
|
211
|
+
#### Inherited from
|
|
212
|
+
|
|
213
|
+
[`WebSearcher`](WebSearcher.md).[`defaultBaseUrls`](WebSearcher.md#defaultbaseurls)
|
|
214
|
+
|
|
215
|
+
***
|
|
216
|
+
|
|
175
217
|
### forEach()
|
|
176
218
|
|
|
177
219
|
> `static` **forEach**: (`cb`) => `void`
|
|
178
220
|
|
|
179
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
221
|
+
Defined in: [web-searcher/src/searcher.ts:129](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L129)
|
|
180
222
|
|
|
181
223
|
Iterates over all registered engines.
|
|
182
224
|
|
|
@@ -202,7 +244,7 @@ Callback function to invoke for each registered engine.
|
|
|
202
244
|
|
|
203
245
|
> `static` **get**: (`name`) => *typeof* [`WebSearcher`](WebSearcher.md)
|
|
204
246
|
|
|
205
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
247
|
+
Defined in: [web-searcher/src/searcher.ts:113](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L113)
|
|
206
248
|
|
|
207
249
|
Retrieves a registered search engine class by name.
|
|
208
250
|
|
|
@@ -230,7 +272,7 @@ The search engine class constructor.
|
|
|
230
272
|
|
|
231
273
|
> `static` `optional` **name**: `string`
|
|
232
274
|
|
|
233
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
275
|
+
Defined in: [web-searcher/src/searcher.ts:41](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L41)
|
|
234
276
|
|
|
235
277
|
Custom engine name. If not provided, it is derived from the class name.
|
|
236
278
|
For example, `GoogleSearcher` becomes `Google`.
|
|
@@ -245,7 +287,7 @@ For example, `GoogleSearcher` becomes `Google`.
|
|
|
245
287
|
|
|
246
288
|
> `static` **register**: (`ctor`, `options?`) => `boolean`
|
|
247
289
|
|
|
248
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
290
|
+
Defined in: [web-searcher/src/searcher.ts:98](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L98)
|
|
249
291
|
|
|
250
292
|
Registers a search engine class.
|
|
251
293
|
|
|
@@ -279,7 +321,7 @@ Registration options. If a string is provided, it is used as the registered name
|
|
|
279
321
|
|
|
280
322
|
> `static` **setAliases**: (`ctor`, ...`aliases`) => `void`
|
|
281
323
|
|
|
282
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
324
|
+
Defined in: [web-searcher/src/searcher.ts:137](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L137)
|
|
283
325
|
|
|
284
326
|
Sets aliases for a registered engine.
|
|
285
327
|
|
|
@@ -311,7 +353,7 @@ Aliases to add.
|
|
|
311
353
|
|
|
312
354
|
> `static` **unregister**: (`name?`) => `void`
|
|
313
355
|
|
|
314
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
356
|
+
Defined in: [web-searcher/src/searcher.ts:105](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L105)
|
|
315
357
|
|
|
316
358
|
Unregisters a search engine.
|
|
317
359
|
|
|
@@ -339,7 +381,7 @@ The name or class to unregister.
|
|
|
339
381
|
|
|
340
382
|
> **get** **pagination**(): [`PaginationConfig`](../interfaces/PaginationConfig.md)
|
|
341
383
|
|
|
342
|
-
Defined in: [web-searcher/src/engines/google.ts:61](https://github.com/isdk/web-searcher.js/blob/
|
|
384
|
+
Defined in: [web-searcher/src/engines/google.ts:61](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/engines/google.ts#L61)
|
|
343
385
|
|
|
344
386
|
Configures pagination for Google Search results.
|
|
345
387
|
Uses the 'start' URL parameter, incrementing by 10 for each page.
|
|
@@ -358,15 +400,15 @@ Uses the 'start' URL parameter, incrementing by 10 for each page.
|
|
|
358
400
|
|
|
359
401
|
#### Get Signature
|
|
360
402
|
|
|
361
|
-
> **get** **template**(): `FetcherOptions`
|
|
403
|
+
> **get** **template**(): [`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
362
404
|
|
|
363
|
-
Defined in: [web-searcher/src/engines/google.ts:32](https://github.com/isdk/web-searcher.js/blob/
|
|
405
|
+
Defined in: [web-searcher/src/engines/google.ts:32](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/engines/google.ts#L32)
|
|
364
406
|
|
|
365
407
|
Defines the fetch template for Google Search.
|
|
366
408
|
|
|
367
409
|
##### Returns
|
|
368
410
|
|
|
369
|
-
`FetcherOptions`
|
|
411
|
+
[`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
370
412
|
|
|
371
413
|
The fetcher configuration including the URL pattern and extraction rules.
|
|
372
414
|
|
|
@@ -374,19 +416,130 @@ The fetcher configuration including the URL pattern and extraction rules.
|
|
|
374
416
|
|
|
375
417
|
[`WebSearcher`](WebSearcher.md).[`template`](WebSearcher.md#template)
|
|
376
418
|
|
|
419
|
+
***
|
|
420
|
+
|
|
421
|
+
### defaultOptions
|
|
422
|
+
|
|
423
|
+
#### Get Signature
|
|
424
|
+
|
|
425
|
+
> **get** `static` **defaultOptions**(): [`SearchOptions`](../interfaces/SearchOptions.md)
|
|
426
|
+
|
|
427
|
+
Defined in: [web-searcher/src/searcher.ts:61](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L61)
|
|
428
|
+
|
|
429
|
+
Gets or sets the default search parameters for this specific engine class.
|
|
430
|
+
This does not include settings from parent classes.
|
|
431
|
+
|
|
432
|
+
##### Returns
|
|
433
|
+
|
|
434
|
+
[`SearchOptions`](../interfaces/SearchOptions.md)
|
|
435
|
+
|
|
436
|
+
#### Set Signature
|
|
437
|
+
|
|
438
|
+
> **set** `static` **defaultOptions**(`options`): `void`
|
|
439
|
+
|
|
440
|
+
Defined in: [web-searcher/src/searcher.ts:68](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L68)
|
|
441
|
+
|
|
442
|
+
##### Parameters
|
|
443
|
+
|
|
444
|
+
###### options
|
|
445
|
+
|
|
446
|
+
[`SearchOptions`](../interfaces/SearchOptions.md)
|
|
447
|
+
|
|
448
|
+
##### Returns
|
|
449
|
+
|
|
450
|
+
`void`
|
|
451
|
+
|
|
452
|
+
#### Inherited from
|
|
453
|
+
|
|
454
|
+
[`WebSearcher`](WebSearcher.md).[`defaultOptions`](WebSearcher.md#defaultoptions)
|
|
455
|
+
|
|
377
456
|
## Methods
|
|
378
457
|
|
|
458
|
+
### \_execute()
|
|
459
|
+
|
|
460
|
+
> `protected` **\_execute**\<`R`\>(`actionOptions`, `context?`): `Promise`\<`FetchActionResult`\<`R`\>\>
|
|
461
|
+
|
|
462
|
+
Defined in: web-fetcher/dist/index.d.ts:1187
|
|
463
|
+
|
|
464
|
+
Executes a single action within the session.
|
|
465
|
+
|
|
466
|
+
#### Type Parameters
|
|
467
|
+
|
|
468
|
+
##### R
|
|
469
|
+
|
|
470
|
+
`R` *extends* `FetchReturnType` = `"response"`
|
|
471
|
+
|
|
472
|
+
The expected return type of the action.
|
|
473
|
+
|
|
474
|
+
#### Parameters
|
|
475
|
+
|
|
476
|
+
##### actionOptions
|
|
477
|
+
|
|
478
|
+
`_RequireAtLeastOne`
|
|
479
|
+
|
|
480
|
+
Configuration for the action to be executed.
|
|
481
|
+
|
|
482
|
+
##### context?
|
|
483
|
+
|
|
484
|
+
`FetchContext`
|
|
485
|
+
|
|
486
|
+
Optional context override for this specific execution. Defaults to the session context.
|
|
487
|
+
|
|
488
|
+
#### Returns
|
|
489
|
+
|
|
490
|
+
`Promise`\<`FetchActionResult`\<`R`\>\>
|
|
491
|
+
|
|
492
|
+
A promise that resolves to the result of the action.
|
|
493
|
+
|
|
494
|
+
#### Example
|
|
495
|
+
|
|
496
|
+
```ts
|
|
497
|
+
await session.execute({ name: 'goto', params: { url: 'https://example.com' } });
|
|
498
|
+
```
|
|
499
|
+
|
|
500
|
+
#### Inherited from
|
|
501
|
+
|
|
502
|
+
[`WebSearcher`](WebSearcher.md).[`_execute`](WebSearcher.md#_execute)
|
|
503
|
+
|
|
504
|
+
***
|
|
505
|
+
|
|
506
|
+
### \_logDebug()
|
|
507
|
+
|
|
508
|
+
> `protected` **\_logDebug**(`category`, ...`args`): `void`
|
|
509
|
+
|
|
510
|
+
Defined in: web-fetcher/dist/index.d.ts:1173
|
|
511
|
+
|
|
512
|
+
#### Parameters
|
|
513
|
+
|
|
514
|
+
##### category
|
|
515
|
+
|
|
516
|
+
`string`
|
|
517
|
+
|
|
518
|
+
##### args
|
|
519
|
+
|
|
520
|
+
...`any`[]
|
|
521
|
+
|
|
522
|
+
#### Returns
|
|
523
|
+
|
|
524
|
+
`void`
|
|
525
|
+
|
|
526
|
+
#### Inherited from
|
|
527
|
+
|
|
528
|
+
[`WebSearcher`](WebSearcher.md).[`_logDebug`](WebSearcher.md#_logdebug)
|
|
529
|
+
|
|
530
|
+
***
|
|
531
|
+
|
|
379
532
|
### createContext()
|
|
380
533
|
|
|
381
534
|
> `protected` **createContext**(`options`): `FetchContext`
|
|
382
535
|
|
|
383
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
536
|
+
Defined in: [web-searcher/src/searcher.ts:254](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L254)
|
|
384
537
|
|
|
385
538
|
#### Parameters
|
|
386
539
|
|
|
387
540
|
##### options
|
|
388
541
|
|
|
389
|
-
`FetcherOptions` = `...`
|
|
542
|
+
[`FetcherOptions`](../interfaces/FetcherOptions.md) = `...`
|
|
390
543
|
|
|
391
544
|
#### Returns
|
|
392
545
|
|
|
@@ -402,7 +555,7 @@ Defined in: [web-searcher/src/searcher.ts:155](https://github.com/isdk/web-searc
|
|
|
402
555
|
|
|
403
556
|
> **dispose**(): `Promise`\<`void`\>
|
|
404
557
|
|
|
405
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
558
|
+
Defined in: web-fetcher/dist/index.d.ts:1233
|
|
406
559
|
|
|
407
560
|
Disposes of the session and its associated engine.
|
|
408
561
|
|
|
@@ -425,9 +578,7 @@ This method should be called when the session is no longer needed to free up res
|
|
|
425
578
|
|
|
426
579
|
> **execute**\<`R`\>(`actionOptions`, `context?`): `Promise`\<`FetchActionResult`\<`R`\>\>
|
|
427
580
|
|
|
428
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
429
|
-
|
|
430
|
-
Executes a single action within the session.
|
|
581
|
+
Defined in: web-fetcher/dist/index.d.ts:1188
|
|
431
582
|
|
|
432
583
|
#### Type Parameters
|
|
433
584
|
|
|
@@ -435,34 +586,20 @@ Executes a single action within the session.
|
|
|
435
586
|
|
|
436
587
|
`R` *extends* `FetchReturnType` = `"response"`
|
|
437
588
|
|
|
438
|
-
The expected return type of the action.
|
|
439
|
-
|
|
440
589
|
#### Parameters
|
|
441
590
|
|
|
442
591
|
##### actionOptions
|
|
443
592
|
|
|
444
593
|
`_RequireAtLeastOne`
|
|
445
594
|
|
|
446
|
-
Configuration for the action to be executed.
|
|
447
|
-
|
|
448
595
|
##### context?
|
|
449
596
|
|
|
450
597
|
`FetchContext`
|
|
451
598
|
|
|
452
|
-
Optional context override for this specific execution. Defaults to the session context.
|
|
453
|
-
|
|
454
599
|
#### Returns
|
|
455
600
|
|
|
456
601
|
`Promise`\<`FetchActionResult`\<`R`\>\>
|
|
457
602
|
|
|
458
|
-
A promise that resolves to the result of the action.
|
|
459
|
-
|
|
460
|
-
#### Example
|
|
461
|
-
|
|
462
|
-
```ts
|
|
463
|
-
await session.execute({ name: 'goto', params: { url: 'https://example.com' } });
|
|
464
|
-
```
|
|
465
|
-
|
|
466
603
|
#### Inherited from
|
|
467
604
|
|
|
468
605
|
[`WebSearcher`](WebSearcher.md).[`execute`](WebSearcher.md#execute)
|
|
@@ -473,7 +610,7 @@ await session.execute({ name: 'goto', params: { url: 'https://example.com' } });
|
|
|
473
610
|
|
|
474
611
|
> **executeAll**(`actions`, `options?`): `Promise`\<\{ `outputs`: `Record`\<`string`, `any`\>; `result`: `FetchResponse` \| `undefined`; \}\>
|
|
475
612
|
|
|
476
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
613
|
+
Defined in: web-fetcher/dist/index.d.ts:1205
|
|
477
614
|
|
|
478
615
|
Executes a sequence of actions.
|
|
479
616
|
|
|
@@ -481,13 +618,13 @@ Executes a sequence of actions.
|
|
|
481
618
|
|
|
482
619
|
##### actions
|
|
483
620
|
|
|
484
|
-
`_RequireAtLeastOne`\<`FetchActionProperties`, `"
|
|
621
|
+
`_RequireAtLeastOne`\<`FetchActionProperties`, `"name"` \| `"id"` \| `"action"`\>[]
|
|
485
622
|
|
|
486
623
|
An array of action options to be executed in order.
|
|
487
624
|
|
|
488
625
|
##### options?
|
|
489
626
|
|
|
490
|
-
`Partial
|
|
627
|
+
`Partial`\<[`FetcherOptions`](../interfaces/FetcherOptions.md)\> & `object`
|
|
491
628
|
|
|
492
629
|
Optional temporary configuration overrides (e.g., timeoutMs, headers) for this batch of actions.
|
|
493
630
|
These overrides do not affect the main session context.
|
|
@@ -517,7 +654,7 @@ const { result, outputs } = await session.executeAll([
|
|
|
517
654
|
|
|
518
655
|
> `protected` **formatOptions**(`options`): `Record`\<`string`, `any`\>
|
|
519
656
|
|
|
520
|
-
Defined in: [web-searcher/src/engines/google.ts:82](https://github.com/isdk/web-searcher.js/blob/
|
|
657
|
+
Defined in: [web-searcher/src/engines/google.ts:82](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/engines/google.ts#L82)
|
|
521
658
|
|
|
522
659
|
Maps standard `SearchOptions` to Google's specific URL parameters.
|
|
523
660
|
|
|
@@ -551,7 +688,7 @@ A map of variables to inject into the URL template.
|
|
|
551
688
|
|
|
552
689
|
> **getOutputs**(): `Record`\<`string`, `any`\>
|
|
553
690
|
|
|
554
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
691
|
+
Defined in: web-fetcher/dist/index.d.ts:1216
|
|
555
692
|
|
|
556
693
|
Retrieves all outputs accumulated during the session.
|
|
557
694
|
|
|
@@ -571,7 +708,7 @@ A record of stored output data.
|
|
|
571
708
|
|
|
572
709
|
> **getState**(): `Promise`\<\{ `cookies`: `Cookie`[]; `sessionState?`: `any`; \} \| `undefined`\>
|
|
573
710
|
|
|
574
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
711
|
+
Defined in: web-fetcher/dist/index.d.ts:1222
|
|
575
712
|
|
|
576
713
|
Gets the current state of the session, including cookies and engine-specific state.
|
|
577
714
|
|
|
@@ -587,16 +724,53 @@ A promise resolving to the session state, or undefined if no engine is initializ
|
|
|
587
724
|
|
|
588
725
|
***
|
|
589
726
|
|
|
727
|
+
### getTemplate()
|
|
728
|
+
|
|
729
|
+
> `protected` **getTemplate**(`variables`, `options`): [`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
730
|
+
|
|
731
|
+
Defined in: [web-searcher/src/searcher.ts:250](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L250)
|
|
732
|
+
|
|
733
|
+
Dynamically retrieves the fetch template based on current variables and search options.
|
|
734
|
+
|
|
735
|
+
Subclasses can override this method to return different extraction rules (actions)
|
|
736
|
+
or URL patterns based on the search category, region, or other parameters.
|
|
737
|
+
|
|
738
|
+
#### Parameters
|
|
739
|
+
|
|
740
|
+
##### variables
|
|
741
|
+
|
|
742
|
+
`Record`\<`string`, `any`\>
|
|
743
|
+
|
|
744
|
+
The calculated variables (from formatOptions, pagination, etc.).
|
|
745
|
+
|
|
746
|
+
##### options
|
|
747
|
+
|
|
748
|
+
[`SearchOptions`](../interfaces/SearchOptions.md)
|
|
749
|
+
|
|
750
|
+
The original search options provided by the user.
|
|
751
|
+
|
|
752
|
+
#### Returns
|
|
753
|
+
|
|
754
|
+
[`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
755
|
+
|
|
756
|
+
The fetcher configuration to be used for the current request.
|
|
757
|
+
|
|
758
|
+
#### Inherited from
|
|
759
|
+
|
|
760
|
+
[`WebSearcher`](WebSearcher.md).[`getTemplate`](WebSearcher.md#gettemplate)
|
|
761
|
+
|
|
762
|
+
***
|
|
763
|
+
|
|
590
764
|
### search()
|
|
591
765
|
|
|
592
766
|
> **search**(`query`, `options`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
593
767
|
|
|
594
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
768
|
+
Defined in: [web-searcher/src/searcher.ts:284](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L284)
|
|
595
769
|
|
|
596
770
|
Executes a search query.
|
|
597
771
|
|
|
598
|
-
This method handles the pagination loop, variable injection,
|
|
599
|
-
and result transformation.
|
|
772
|
+
This method handles the pagination loop, multi-instance failover, variable injection,
|
|
773
|
+
fetching, and result transformation.
|
|
600
774
|
|
|
601
775
|
#### Parameters
|
|
602
776
|
|
|
@@ -628,7 +802,7 @@ A promise resolving to an array of standardized search results.
|
|
|
628
802
|
|
|
629
803
|
> `protected` **transform**(`outputs`): `Promise`\<`any`[]\>
|
|
630
804
|
|
|
631
|
-
Defined in: [web-searcher/src/engines/google.ts:145](https://github.com/isdk/web-searcher.js/blob/
|
|
805
|
+
Defined in: [web-searcher/src/engines/google.ts:145](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/engines/google.ts#L145)
|
|
632
806
|
|
|
633
807
|
Cleans and normalizes the extracted results.
|
|
634
808
|
Specifically, it unwraps Google's redirect URLs (starting with `/url?q=`).
|
|
@@ -653,24 +827,79 @@ An array of cleaned search results.
|
|
|
653
827
|
|
|
654
828
|
***
|
|
655
829
|
|
|
830
|
+
### validateFetchResult()
|
|
831
|
+
|
|
832
|
+
> `protected` **validateFetchResult**(`results`, `context`): `Promise`\<`boolean`\>
|
|
833
|
+
|
|
834
|
+
Defined in: [web-searcher/src/searcher.ts:462](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L462)
|
|
835
|
+
|
|
836
|
+
Hook for subclasses to validate fetched results before they are accepted.
|
|
837
|
+
If this returns false, the instance manager will consider the fetch a failure
|
|
838
|
+
and automatically switch to the next available baseUrl (if any).
|
|
839
|
+
|
|
840
|
+
#### Parameters
|
|
841
|
+
|
|
842
|
+
##### results
|
|
843
|
+
|
|
844
|
+
[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]
|
|
845
|
+
|
|
846
|
+
The extracted results.
|
|
847
|
+
|
|
848
|
+
##### context
|
|
849
|
+
|
|
850
|
+
[`SearchContext`](../interfaces/SearchContext.md)
|
|
851
|
+
|
|
852
|
+
Context including the current baseUrl and page.
|
|
853
|
+
|
|
854
|
+
#### Returns
|
|
855
|
+
|
|
856
|
+
`Promise`\<`boolean`\>
|
|
857
|
+
|
|
858
|
+
A promise resolving to true if valid, false otherwise.
|
|
859
|
+
|
|
860
|
+
#### Inherited from
|
|
861
|
+
|
|
862
|
+
[`WebSearcher`](WebSearcher.md).[`validateFetchResult`](WebSearcher.md#validatefetchresult)
|
|
863
|
+
|
|
864
|
+
***
|
|
865
|
+
|
|
866
|
+
### getDefaultOptions()
|
|
867
|
+
|
|
868
|
+
> `static` **getDefaultOptions**(): [`SearchOptions`](../interfaces/SearchOptions.md)
|
|
869
|
+
|
|
870
|
+
Defined in: [web-searcher/src/searcher.ts:76](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L76)
|
|
871
|
+
|
|
872
|
+
Retrieves the combined default search options by traversing the prototype chain.
|
|
873
|
+
Priority: Current class > Parent class > WebSearcher base class.
|
|
874
|
+
|
|
875
|
+
#### Returns
|
|
876
|
+
|
|
877
|
+
[`SearchOptions`](../interfaces/SearchOptions.md)
|
|
878
|
+
|
|
879
|
+
#### Inherited from
|
|
880
|
+
|
|
881
|
+
[`WebSearcher`](WebSearcher.md).[`getDefaultOptions`](WebSearcher.md#getdefaultoptions)
|
|
882
|
+
|
|
883
|
+
***
|
|
884
|
+
|
|
656
885
|
### search()
|
|
657
886
|
|
|
658
|
-
> `static` **search**(`
|
|
887
|
+
> `static` **search**(`engineNames`, `query`, `options`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
659
888
|
|
|
660
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
889
|
+
Defined in: [web-searcher/src/searcher.ts:150](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L150)
|
|
661
890
|
|
|
662
|
-
Static helper to execute a one-off search.
|
|
891
|
+
Static helper to execute a one-off search or a fallback chain.
|
|
663
892
|
|
|
664
|
-
It creates an instance of the specified engine, executes the search, and
|
|
665
|
-
|
|
893
|
+
It creates an instance of the specified engine(s), executes the search, and automatically
|
|
894
|
+
falls back to the next engine in the list if the current one fails or is exhausted.
|
|
666
895
|
|
|
667
896
|
#### Parameters
|
|
668
897
|
|
|
669
|
-
#####
|
|
898
|
+
##### engineNames
|
|
670
899
|
|
|
671
|
-
|
|
900
|
+
The name(s) of the engine(s) to use (e.g., 'Google' or ['SearXNG', 'Google']).
|
|
672
901
|
|
|
673
|
-
|
|
902
|
+
`string` | `string`[]
|
|
674
903
|
|
|
675
904
|
##### query
|
|
676
905
|
|
|
@@ -680,7 +909,7 @@ The search query string.
|
|
|
680
909
|
|
|
681
910
|
##### options
|
|
682
911
|
|
|
683
|
-
[`SearchOptions`](../interfaces/SearchOptions.md) & `FetcherOptions` = `{}`
|
|
912
|
+
[`SearchOptions`](../interfaces/SearchOptions.md) & [`FetcherOptions`](../interfaces/FetcherOptions.md) = `{}`
|
|
684
913
|
|
|
685
914
|
Combined search options and fetcher options.
|
|
686
915
|
|