@isdk/web-searcher 0.1.5 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.cn.md +40 -0
- package/README.md +40 -0
- package/dist/index.d.mts +14 -0
- package/dist/index.d.ts +14 -0
- package/dist/index.js +1 -1
- package/dist/index.mjs +1 -1
- package/docs/README.md +40 -0
- package/docs/classes/GoogleSearcher.md +150 -48
- package/docs/classes/WebSearcher.md +138 -48
- package/docs/functions/extractDate.md +1 -1
- package/docs/functions/extractMetadataFrom.md +1 -1
- package/docs/functions/fetchHeaders.md +1 -1
- package/docs/functions/fetchPartial.md +1 -1
- package/docs/functions/normalizeDate.md +1 -1
- package/docs/functions/parseHeaders.md +1 -1
- package/docs/functions/parseHtml.md +1 -1
- package/docs/functions/testUrlsByLatency.md +5 -1
- package/docs/interfaces/CustomTimeRange.md +3 -3
- package/docs/interfaces/ExtractOptions.md +4 -4
- package/docs/interfaces/FetchExtractorOptions.md +3 -3
- package/docs/interfaces/FetcherOptions.md +41 -29
- package/docs/interfaces/HtmlData.md +4 -4
- package/docs/interfaces/MetadataResult.md +2 -2
- package/docs/interfaces/PaginationConfig.md +7 -7
- package/docs/interfaces/SearchContext.md +6 -6
- package/docs/interfaces/SearchOptions.md +13 -13
- package/docs/interfaces/StandardSearchResult.md +10 -10
- package/docs/interfaces/VerifiedUrl.md +3 -3
- package/docs/type-aliases/MetadataType.md +1 -1
- package/docs/type-aliases/SafeSearchLevel.md +1 -1
- package/docs/type-aliases/SearchCategory.md +1 -1
- package/docs/type-aliases/SearchTimeRange.md +1 -1
- package/docs/type-aliases/SearchTimeRangePreset.md +1 -1
- package/docs/type-aliases/SearcherConstructor.md +1 -1
- package/package.json +2 -2
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Abstract Class: WebSearcher
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/searcher.ts:32](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/searcher.ts:32](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L32)
|
|
10
10
|
|
|
11
11
|
The abstract base class for all search engines.
|
|
12
12
|
|
|
@@ -41,7 +41,7 @@ WebSearcher.register(MySearcher);
|
|
|
41
41
|
|
|
42
42
|
> **new WebSearcher**(`options?`): `WebSearcher`
|
|
43
43
|
|
|
44
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
44
|
+
Defined in: web-fetcher/dist/index.d.ts:1172
|
|
45
45
|
|
|
46
46
|
Creates a new FetchSession.
|
|
47
47
|
|
|
@@ -67,7 +67,7 @@ Configuration options for the fetcher.
|
|
|
67
67
|
|
|
68
68
|
> `protected` **closed**: `boolean`
|
|
69
69
|
|
|
70
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
70
|
+
Defined in: web-fetcher/dist/index.d.ts:1166
|
|
71
71
|
|
|
72
72
|
#### Inherited from
|
|
73
73
|
|
|
@@ -79,7 +79,7 @@ Defined in: web-fetcher/dist/index.d.ts:1165
|
|
|
79
79
|
|
|
80
80
|
> `readonly` **context**: `FetchContext`
|
|
81
81
|
|
|
82
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
82
|
+
Defined in: web-fetcher/dist/index.d.ts:1165
|
|
83
83
|
|
|
84
84
|
The execution context for this session, containing configurations, event bus, and shared state.
|
|
85
85
|
|
|
@@ -93,7 +93,7 @@ The execution context for this session, containing configurations, event bus, an
|
|
|
93
93
|
|
|
94
94
|
> `readonly` **id**: `string`
|
|
95
95
|
|
|
96
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
96
|
+
Defined in: web-fetcher/dist/index.d.ts:1161
|
|
97
97
|
|
|
98
98
|
Unique identifier for the session.
|
|
99
99
|
|
|
@@ -107,7 +107,7 @@ Unique identifier for the session.
|
|
|
107
107
|
|
|
108
108
|
> `protected` **options**: [`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
109
109
|
|
|
110
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
110
|
+
Defined in: web-fetcher/dist/index.d.ts:1157
|
|
111
111
|
|
|
112
112
|
#### Inherited from
|
|
113
113
|
|
|
@@ -115,11 +115,21 @@ Defined in: web-fetcher/dist/index.d.ts:1156
|
|
|
115
115
|
|
|
116
116
|
***
|
|
117
117
|
|
|
118
|
+
### \_defaultOptions?
|
|
119
|
+
|
|
120
|
+
> `static` `optional` **\_defaultOptions**: [`SearchOptions`](../interfaces/SearchOptions.md)
|
|
121
|
+
|
|
122
|
+
Defined in: [web-searcher/src/searcher.ts:55](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L55)
|
|
123
|
+
|
|
124
|
+
**`Internal`**
|
|
125
|
+
|
|
126
|
+
***
|
|
127
|
+
|
|
118
128
|
### \_isFactory
|
|
119
129
|
|
|
120
130
|
> `static` **\_isFactory**: `boolean` = `false`
|
|
121
131
|
|
|
122
|
-
Defined in: [web-searcher/src/searcher.ts:34](https://github.com/isdk/web-searcher.js/blob/
|
|
132
|
+
Defined in: [web-searcher/src/searcher.ts:34](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L34)
|
|
123
133
|
|
|
124
134
|
***
|
|
125
135
|
|
|
@@ -127,7 +137,7 @@ Defined in: [web-searcher/src/searcher.ts:34](https://github.com/isdk/web-search
|
|
|
127
137
|
|
|
128
138
|
> `static` `optional` **alias**: `string` \| `string`[]
|
|
129
139
|
|
|
130
|
-
Defined in: [web-searcher/src/searcher.ts:46](https://github.com/isdk/web-searcher.js/blob/
|
|
140
|
+
Defined in: [web-searcher/src/searcher.ts:46](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L46)
|
|
131
141
|
|
|
132
142
|
Engine alias(es). Can be a single string or an array of strings.
|
|
133
143
|
Useful for registering shorthand names (e.g., 'g' for 'Google').
|
|
@@ -138,7 +148,7 @@ Useful for registering shorthand names (e.g., 'g' for 'Google').
|
|
|
138
148
|
|
|
139
149
|
> `static` **createObject**: (`name`, ...`args`) => `WebSearcher`
|
|
140
150
|
|
|
141
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
151
|
+
Defined in: [web-searcher/src/searcher.ts:122](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L122)
|
|
142
152
|
|
|
143
153
|
Creates an instance of the registered search engine.
|
|
144
154
|
|
|
@@ -168,7 +178,7 @@ An instance of the search engine.
|
|
|
168
178
|
|
|
169
179
|
> `static` `optional` **currentInstanceIndex**: `number`
|
|
170
180
|
|
|
171
|
-
Defined in: [web-searcher/src/searcher.ts:52](https://github.com/isdk/web-searcher.js/blob/
|
|
181
|
+
Defined in: [web-searcher/src/searcher.ts:52](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L52)
|
|
172
182
|
|
|
173
183
|
Globally shared index for tracking the currently active instance (node) across sessions.
|
|
174
184
|
|
|
@@ -178,7 +188,7 @@ Globally shared index for tracking the currently active instance (node) across s
|
|
|
178
188
|
|
|
179
189
|
> `static` `optional` **defaultBaseUrls**: `string`[]
|
|
180
190
|
|
|
181
|
-
Defined in: [web-searcher/src/searcher.ts:49](https://github.com/isdk/web-searcher.js/blob/
|
|
191
|
+
Defined in: [web-searcher/src/searcher.ts:49](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L49)
|
|
182
192
|
|
|
183
193
|
Default base URLs for engines that support multiple instances.
|
|
184
194
|
|
|
@@ -188,7 +198,7 @@ Default base URLs for engines that support multiple instances.
|
|
|
188
198
|
|
|
189
199
|
> `static` **forEach**: (`cb`) => `void`
|
|
190
200
|
|
|
191
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
201
|
+
Defined in: [web-searcher/src/searcher.ts:129](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L129)
|
|
192
202
|
|
|
193
203
|
Iterates over all registered engines.
|
|
194
204
|
|
|
@@ -210,7 +220,7 @@ Callback function to invoke for each registered engine.
|
|
|
210
220
|
|
|
211
221
|
> `static` **get**: (`name`) => *typeof* `WebSearcher`
|
|
212
222
|
|
|
213
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
223
|
+
Defined in: [web-searcher/src/searcher.ts:113](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L113)
|
|
214
224
|
|
|
215
225
|
Retrieves a registered search engine class by name.
|
|
216
226
|
|
|
@@ -234,7 +244,7 @@ The search engine class constructor.
|
|
|
234
244
|
|
|
235
245
|
> `static` `optional` **name**: `string`
|
|
236
246
|
|
|
237
|
-
Defined in: [web-searcher/src/searcher.ts:41](https://github.com/isdk/web-searcher.js/blob/
|
|
247
|
+
Defined in: [web-searcher/src/searcher.ts:41](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L41)
|
|
238
248
|
|
|
239
249
|
Custom engine name. If not provided, it is derived from the class name.
|
|
240
250
|
For example, `GoogleSearcher` becomes `Google`.
|
|
@@ -245,7 +255,7 @@ For example, `GoogleSearcher` becomes `Google`.
|
|
|
245
255
|
|
|
246
256
|
> `static` **register**: (`ctor`, `options?`) => `boolean`
|
|
247
257
|
|
|
248
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
258
|
+
Defined in: [web-searcher/src/searcher.ts:98](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L98)
|
|
249
259
|
|
|
250
260
|
Registers a search engine class.
|
|
251
261
|
|
|
@@ -275,7 +285,7 @@ Registration options. If a string is provided, it is used as the registered name
|
|
|
275
285
|
|
|
276
286
|
> `static` **setAliases**: (`ctor`, ...`aliases`) => `void`
|
|
277
287
|
|
|
278
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
288
|
+
Defined in: [web-searcher/src/searcher.ts:137](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L137)
|
|
279
289
|
|
|
280
290
|
Sets aliases for a registered engine.
|
|
281
291
|
|
|
@@ -303,7 +313,7 @@ Aliases to add.
|
|
|
303
313
|
|
|
304
314
|
> `static` **unregister**: (`name?`) => `void`
|
|
305
315
|
|
|
306
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
316
|
+
Defined in: [web-searcher/src/searcher.ts:105](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L105)
|
|
307
317
|
|
|
308
318
|
Unregisters a search engine.
|
|
309
319
|
|
|
@@ -327,7 +337,7 @@ The name or class to unregister.
|
|
|
327
337
|
|
|
328
338
|
> **get** **pagination**(): [`PaginationConfig`](../interfaces/PaginationConfig.md) \| `undefined`
|
|
329
339
|
|
|
330
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
340
|
+
Defined in: [web-searcher/src/searcher.ts:236](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L236)
|
|
331
341
|
|
|
332
342
|
Optional pagination configuration.
|
|
333
343
|
Defines how the searcher navigates to subsequent pages.
|
|
@@ -346,7 +356,7 @@ If undefined, the searcher will only fetch the first page.
|
|
|
346
356
|
|
|
347
357
|
> **get** **template**(): [`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
348
358
|
|
|
349
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
359
|
+
Defined in: [web-searcher/src/searcher.ts:226](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L226)
|
|
350
360
|
|
|
351
361
|
The declarative template for the fetch options.
|
|
352
362
|
|
|
@@ -372,13 +382,94 @@ get template() {
|
|
|
372
382
|
|
|
373
383
|
[`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
374
384
|
|
|
385
|
+
***
|
|
386
|
+
|
|
387
|
+
### defaultOptions
|
|
388
|
+
|
|
389
|
+
#### Get Signature
|
|
390
|
+
|
|
391
|
+
> **get** `static` **defaultOptions**(): [`SearchOptions`](../interfaces/SearchOptions.md)
|
|
392
|
+
|
|
393
|
+
Defined in: [web-searcher/src/searcher.ts:61](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L61)
|
|
394
|
+
|
|
395
|
+
Gets or sets the default search parameters for this specific engine class.
|
|
396
|
+
This does not include settings from parent classes.
|
|
397
|
+
|
|
398
|
+
##### Returns
|
|
399
|
+
|
|
400
|
+
[`SearchOptions`](../interfaces/SearchOptions.md)
|
|
401
|
+
|
|
402
|
+
#### Set Signature
|
|
403
|
+
|
|
404
|
+
> **set** `static` **defaultOptions**(`options`): `void`
|
|
405
|
+
|
|
406
|
+
Defined in: [web-searcher/src/searcher.ts:68](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L68)
|
|
407
|
+
|
|
408
|
+
##### Parameters
|
|
409
|
+
|
|
410
|
+
###### options
|
|
411
|
+
|
|
412
|
+
[`SearchOptions`](../interfaces/SearchOptions.md)
|
|
413
|
+
|
|
414
|
+
##### Returns
|
|
415
|
+
|
|
416
|
+
`void`
|
|
417
|
+
|
|
375
418
|
## Methods
|
|
376
419
|
|
|
420
|
+
### \_execute()
|
|
421
|
+
|
|
422
|
+
> `protected` **\_execute**\<`R`\>(`actionOptions`, `context?`): `Promise`\<`FetchActionResult`\<`R`\>\>
|
|
423
|
+
|
|
424
|
+
Defined in: web-fetcher/dist/index.d.ts:1187
|
|
425
|
+
|
|
426
|
+
Executes a single action within the session.
|
|
427
|
+
|
|
428
|
+
#### Type Parameters
|
|
429
|
+
|
|
430
|
+
##### R
|
|
431
|
+
|
|
432
|
+
`R` *extends* `FetchReturnType` = `"response"`
|
|
433
|
+
|
|
434
|
+
The expected return type of the action.
|
|
435
|
+
|
|
436
|
+
#### Parameters
|
|
437
|
+
|
|
438
|
+
##### actionOptions
|
|
439
|
+
|
|
440
|
+
`_RequireAtLeastOne`
|
|
441
|
+
|
|
442
|
+
Configuration for the action to be executed.
|
|
443
|
+
|
|
444
|
+
##### context?
|
|
445
|
+
|
|
446
|
+
`FetchContext`
|
|
447
|
+
|
|
448
|
+
Optional context override for this specific execution. Defaults to the session context.
|
|
449
|
+
|
|
450
|
+
#### Returns
|
|
451
|
+
|
|
452
|
+
`Promise`\<`FetchActionResult`\<`R`\>\>
|
|
453
|
+
|
|
454
|
+
A promise that resolves to the result of the action.
|
|
455
|
+
|
|
456
|
+
#### Example
|
|
457
|
+
|
|
458
|
+
```ts
|
|
459
|
+
await session.execute({ name: 'goto', params: { url: 'https://example.com' } });
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
#### Inherited from
|
|
463
|
+
|
|
464
|
+
`FetchSession._execute`
|
|
465
|
+
|
|
466
|
+
***
|
|
467
|
+
|
|
377
468
|
### \_logDebug()
|
|
378
469
|
|
|
379
470
|
> `protected` **\_logDebug**(`category`, ...`args`): `void`
|
|
380
471
|
|
|
381
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
472
|
+
Defined in: web-fetcher/dist/index.d.ts:1173
|
|
382
473
|
|
|
383
474
|
#### Parameters
|
|
384
475
|
|
|
@@ -404,7 +495,7 @@ Defined in: web-fetcher/dist/index.d.ts:1172
|
|
|
404
495
|
|
|
405
496
|
> `protected` **createContext**(`options`): `FetchContext`
|
|
406
497
|
|
|
407
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
498
|
+
Defined in: [web-searcher/src/searcher.ts:254](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L254)
|
|
408
499
|
|
|
409
500
|
#### Parameters
|
|
410
501
|
|
|
@@ -426,7 +517,7 @@ Defined in: [web-searcher/src/searcher.ts:216](https://github.com/isdk/web-searc
|
|
|
426
517
|
|
|
427
518
|
> **dispose**(): `Promise`\<`void`\>
|
|
428
519
|
|
|
429
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
520
|
+
Defined in: web-fetcher/dist/index.d.ts:1233
|
|
430
521
|
|
|
431
522
|
Disposes of the session and its associated engine.
|
|
432
523
|
|
|
@@ -449,9 +540,7 @@ This method should be called when the session is no longer needed to free up res
|
|
|
449
540
|
|
|
450
541
|
> **execute**\<`R`\>(`actionOptions`, `context?`): `Promise`\<`FetchActionResult`\<`R`\>\>
|
|
451
542
|
|
|
452
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
453
|
-
|
|
454
|
-
Executes a single action within the session.
|
|
543
|
+
Defined in: web-fetcher/dist/index.d.ts:1188
|
|
455
544
|
|
|
456
545
|
#### Type Parameters
|
|
457
546
|
|
|
@@ -459,34 +548,20 @@ Executes a single action within the session.
|
|
|
459
548
|
|
|
460
549
|
`R` *extends* `FetchReturnType` = `"response"`
|
|
461
550
|
|
|
462
|
-
The expected return type of the action.
|
|
463
|
-
|
|
464
551
|
#### Parameters
|
|
465
552
|
|
|
466
553
|
##### actionOptions
|
|
467
554
|
|
|
468
555
|
`_RequireAtLeastOne`
|
|
469
556
|
|
|
470
|
-
Configuration for the action to be executed.
|
|
471
|
-
|
|
472
557
|
##### context?
|
|
473
558
|
|
|
474
559
|
`FetchContext`
|
|
475
560
|
|
|
476
|
-
Optional context override for this specific execution. Defaults to the session context.
|
|
477
|
-
|
|
478
561
|
#### Returns
|
|
479
562
|
|
|
480
563
|
`Promise`\<`FetchActionResult`\<`R`\>\>
|
|
481
564
|
|
|
482
|
-
A promise that resolves to the result of the action.
|
|
483
|
-
|
|
484
|
-
#### Example
|
|
485
|
-
|
|
486
|
-
```ts
|
|
487
|
-
await session.execute({ name: 'goto', params: { url: 'https://example.com' } });
|
|
488
|
-
```
|
|
489
|
-
|
|
490
565
|
#### Inherited from
|
|
491
566
|
|
|
492
567
|
`FetchSession.execute`
|
|
@@ -497,7 +572,7 @@ await session.execute({ name: 'goto', params: { url: 'https://example.com' } });
|
|
|
497
572
|
|
|
498
573
|
> **executeAll**(`actions`, `options?`): `Promise`\<\{ `outputs`: `Record`\<`string`, `any`\>; `result`: `FetchResponse` \| `undefined`; \}\>
|
|
499
574
|
|
|
500
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
575
|
+
Defined in: web-fetcher/dist/index.d.ts:1205
|
|
501
576
|
|
|
502
577
|
Executes a sequence of actions.
|
|
503
578
|
|
|
@@ -541,7 +616,7 @@ const { result, outputs } = await session.executeAll([
|
|
|
541
616
|
|
|
542
617
|
> `protected` **formatOptions**(`options`): `Record`\<`string`, `any`\>
|
|
543
618
|
|
|
544
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
619
|
+
Defined in: [web-searcher/src/searcher.ts:498](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L498)
|
|
545
620
|
|
|
546
621
|
Transforms standard options into engine-specific template variables.
|
|
547
622
|
|
|
@@ -569,7 +644,7 @@ A dictionary of variables to be injected into the template.
|
|
|
569
644
|
|
|
570
645
|
> **getOutputs**(): `Record`\<`string`, `any`\>
|
|
571
646
|
|
|
572
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
647
|
+
Defined in: web-fetcher/dist/index.d.ts:1216
|
|
573
648
|
|
|
574
649
|
Retrieves all outputs accumulated during the session.
|
|
575
650
|
|
|
@@ -589,7 +664,7 @@ A record of stored output data.
|
|
|
589
664
|
|
|
590
665
|
> **getState**(): `Promise`\<\{ `cookies`: `Cookie`[]; `sessionState?`: `any`; \} \| `undefined`\>
|
|
591
666
|
|
|
592
|
-
Defined in: web-fetcher/dist/index.d.ts:
|
|
667
|
+
Defined in: web-fetcher/dist/index.d.ts:1222
|
|
593
668
|
|
|
594
669
|
Gets the current state of the session, including cookies and engine-specific state.
|
|
595
670
|
|
|
@@ -609,7 +684,7 @@ A promise resolving to the session state, or undefined if no engine is initializ
|
|
|
609
684
|
|
|
610
685
|
> `protected` **getTemplate**(`variables`, `options`): [`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
611
686
|
|
|
612
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
687
|
+
Defined in: [web-searcher/src/searcher.ts:250](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L250)
|
|
613
688
|
|
|
614
689
|
Dynamically retrieves the fetch template based on current variables and search options.
|
|
615
690
|
|
|
@@ -642,7 +717,7 @@ The fetcher configuration to be used for the current request.
|
|
|
642
717
|
|
|
643
718
|
> **search**(`query`, `options`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
644
719
|
|
|
645
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
720
|
+
Defined in: [web-searcher/src/searcher.ts:284](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L284)
|
|
646
721
|
|
|
647
722
|
Executes a search query.
|
|
648
723
|
|
|
@@ -675,7 +750,7 @@ A promise resolving to an array of standardized search results.
|
|
|
675
750
|
|
|
676
751
|
> `protected` **transform**(`outputs`, `context`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
677
752
|
|
|
678
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
753
|
+
Defined in: [web-searcher/src/searcher.ts:480](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L480)
|
|
679
754
|
|
|
680
755
|
Transform and clean the raw extracted results.
|
|
681
756
|
|
|
@@ -708,7 +783,7 @@ A promise resolving to an array of standardized search results.
|
|
|
708
783
|
|
|
709
784
|
> `protected` **validateFetchResult**(`results`, `context`): `Promise`\<`boolean`\>
|
|
710
785
|
|
|
711
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
786
|
+
Defined in: [web-searcher/src/searcher.ts:462](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L462)
|
|
712
787
|
|
|
713
788
|
Hook for subclasses to validate fetched results before they are accepted.
|
|
714
789
|
If this returns false, the instance manager will consider the fetch a failure
|
|
@@ -736,11 +811,26 @@ A promise resolving to true if valid, false otherwise.
|
|
|
736
811
|
|
|
737
812
|
***
|
|
738
813
|
|
|
814
|
+
### getDefaultOptions()
|
|
815
|
+
|
|
816
|
+
> `static` **getDefaultOptions**(): [`SearchOptions`](../interfaces/SearchOptions.md)
|
|
817
|
+
|
|
818
|
+
Defined in: [web-searcher/src/searcher.ts:76](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L76)
|
|
819
|
+
|
|
820
|
+
Retrieves the combined default search options by traversing the prototype chain.
|
|
821
|
+
Priority: Current class > Parent class > WebSearcher base class.
|
|
822
|
+
|
|
823
|
+
#### Returns
|
|
824
|
+
|
|
825
|
+
[`SearchOptions`](../interfaces/SearchOptions.md)
|
|
826
|
+
|
|
827
|
+
***
|
|
828
|
+
|
|
739
829
|
### search()
|
|
740
830
|
|
|
741
831
|
> `static` **search**(`engineNames`, `query`, `options`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
742
832
|
|
|
743
|
-
Defined in: [web-searcher/src/searcher.ts:
|
|
833
|
+
Defined in: [web-searcher/src/searcher.ts:150](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L150)
|
|
744
834
|
|
|
745
835
|
Static helper to execute a one-off search or a fallback chain.
|
|
746
836
|
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **extractDate**(`url`, `options`): `Promise`\<`string` \| `null`\>
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:30](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:30](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/date-extractor.ts#L30)
|
|
12
12
|
|
|
13
13
|
High-level convenience function to extract the publication or modification date from a URL.
|
|
14
14
|
It performs a partial fetch of the content and applies multiple extraction rules
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **extractMetadataFrom**(`result`, `type`): `string` \| `null`
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/utils/extractor/extractor.ts:27](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/utils/extractor/extractor.ts:27](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/extractor.ts#L27)
|
|
12
12
|
|
|
13
13
|
Extracts specific metadata from parsed HTML and headers based on a requested type.
|
|
14
14
|
Currently supports 'date' extraction with a prioritized fallback mechanism.
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **fetchHeaders**(`url`, `options`): `Promise`\<`Headers` \| `null`\>
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:19](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:19](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L19)
|
|
12
12
|
|
|
13
13
|
Fetches only the HTTP headers for a given URL using a HEAD request.
|
|
14
14
|
Useful for checking 'last-modified' without downloading the body.
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **fetchPartial**(`url`, `maxBytes`, `options`): `Promise`\<\{ `content`: `string`; `headers`: `Headers`; \} \| `null`\>
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:55](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:55](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L55)
|
|
12
12
|
|
|
13
13
|
Fetches a partial amount of content from a URL.
|
|
14
14
|
Automatically handles character set detection from the Content-Type header.
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **normalizeDate**(`dateStr`): `string` \| `null`
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/utils/extractor/date-normalizer.ts:9](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/utils/extractor/date-normalizer.ts:9](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/date-normalizer.ts#L9)
|
|
12
12
|
|
|
13
13
|
Normalizes a date string into a standard ISO 8601 format (UTC).
|
|
14
14
|
It handles various formats (YYYY-MM-DD, RFC2822, etc.) and performs
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **parseHeaders**(`headers`): `Record`\<`string`, `string`\>
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/utils/extractor/parser.ts:25](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/utils/extractor/parser.ts:25](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/parser.ts#L25)
|
|
12
12
|
|
|
13
13
|
Converts a Web API Headers object into a plain JavaScript record.
|
|
14
14
|
All header names are converted to lowercase for consistent access.
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **parseHtml**(`html`): [`HtmlData`](../interfaces/HtmlData.md)
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/utils/extractor/parser.ts:49](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/utils/extractor/parser.ts:49](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/parser.ts#L49)
|
|
12
12
|
|
|
13
13
|
Parses an HTML string to extract generic metadata structures (Meta tags, JSON-LD, Time tags).
|
|
14
14
|
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **testUrlsByLatency**(`urls`, `options`): `Promise`\<[`VerifiedUrl`](../interfaces/VerifiedUrl.md)[]\>
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/utils/latency.ts:12](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/utils/latency.ts:12](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/latency.ts#L12)
|
|
12
12
|
|
|
13
13
|
A general utility to test a list of URLs for availability and latency.
|
|
14
14
|
Returns a list of verified URLs sorted by response time.
|
|
@@ -25,6 +25,10 @@ Returns a list of verified URLs sorted by response time.
|
|
|
25
25
|
|
|
26
26
|
`number`
|
|
27
27
|
|
|
28
|
+
#### proxy?
|
|
29
|
+
|
|
30
|
+
`string`
|
|
31
|
+
|
|
28
32
|
#### testPath?
|
|
29
33
|
|
|
30
34
|
`string`
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: CustomTimeRange
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:113](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:113](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L113)
|
|
10
10
|
|
|
11
11
|
## Properties
|
|
12
12
|
|
|
@@ -14,7 +14,7 @@ Defined in: [web-searcher/src/types.ts:113](https://github.com/isdk/web-searcher
|
|
|
14
14
|
|
|
15
15
|
> **from**: `string` \| `Date`
|
|
16
16
|
|
|
17
|
-
Defined in: [web-searcher/src/types.ts:115](https://github.com/isdk/web-searcher.js/blob/
|
|
17
|
+
Defined in: [web-searcher/src/types.ts:115](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L115)
|
|
18
18
|
|
|
19
19
|
Start date (Date object or string like 'YYYY-MM-DD').
|
|
20
20
|
|
|
@@ -24,6 +24,6 @@ Start date (Date object or string like 'YYYY-MM-DD').
|
|
|
24
24
|
|
|
25
25
|
> `optional` **to**: `string` \| `Date`
|
|
26
26
|
|
|
27
|
-
Defined in: [web-searcher/src/types.ts:117](https://github.com/isdk/web-searcher.js/blob/
|
|
27
|
+
Defined in: [web-searcher/src/types.ts:117](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L117)
|
|
28
28
|
|
|
29
29
|
End date (Date object or string like 'YYYY-MM-DD'). Defaults to current date if omitted.
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: ExtractOptions
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:7](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:7](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/date-extractor.ts#L7)
|
|
10
10
|
|
|
11
11
|
Options for the extractDate function.
|
|
12
12
|
|
|
@@ -20,7 +20,7 @@ Options for the extractDate function.
|
|
|
20
20
|
|
|
21
21
|
> `optional` **headers**: `Record`\<`string`, `string`\>
|
|
22
22
|
|
|
23
|
-
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:8](https://github.com/isdk/web-searcher.js/blob/
|
|
23
|
+
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:8](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L8)
|
|
24
24
|
|
|
25
25
|
Custom HTTP headers to include in the request.
|
|
26
26
|
|
|
@@ -34,7 +34,7 @@ Custom HTTP headers to include in the request.
|
|
|
34
34
|
|
|
35
35
|
> `optional` **maxBytes**: `number`
|
|
36
36
|
|
|
37
|
-
Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:12](https://github.com/isdk/web-searcher.js/blob/
|
|
37
|
+
Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:12](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/date-extractor.ts#L12)
|
|
38
38
|
|
|
39
39
|
Maximum number of bytes to download from the URL.
|
|
40
40
|
Defaults to 32768 (32KB), which is usually enough for the HTML <head>.
|
|
@@ -45,7 +45,7 @@ Defaults to 32768 (32KB), which is usually enough for the HTML <head>.
|
|
|
45
45
|
|
|
46
46
|
> `optional` **timeout**: `number`
|
|
47
47
|
|
|
48
|
-
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:6](https://github.com/isdk/web-searcher.js/blob/
|
|
48
|
+
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:6](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L6)
|
|
49
49
|
|
|
50
50
|
Timeout in milliseconds. Defaults vary by function (5s to 10s).
|
|
51
51
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: FetchExtractorOptions
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:4](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:4](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L4)
|
|
10
10
|
|
|
11
11
|
Options for network requests.
|
|
12
12
|
|
|
@@ -20,7 +20,7 @@ Options for network requests.
|
|
|
20
20
|
|
|
21
21
|
> `optional` **headers**: `Record`\<`string`, `string`\>
|
|
22
22
|
|
|
23
|
-
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:8](https://github.com/isdk/web-searcher.js/blob/
|
|
23
|
+
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:8](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L8)
|
|
24
24
|
|
|
25
25
|
Custom HTTP headers to include in the request.
|
|
26
26
|
|
|
@@ -30,6 +30,6 @@ Custom HTTP headers to include in the request.
|
|
|
30
30
|
|
|
31
31
|
> `optional` **timeout**: `number`
|
|
32
32
|
|
|
33
|
-
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:6](https://github.com/isdk/web-searcher.js/blob/
|
|
33
|
+
Defined in: [web-searcher/src/utils/extractor/fetcher.ts:6](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L6)
|
|
34
34
|
|
|
35
35
|
Timeout in milliseconds. Defaults vary by function (5s to 10s).
|