@isdk/web-searcher 0.1.3 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.cn.md +168 -8
- package/README.md +168 -8
- package/dist/index.d.mts +221 -12
- package/dist/index.d.ts +221 -12
- package/dist/index.js +1 -1
- package/dist/index.mjs +1 -1
- package/docs/README.md +168 -8
- package/docs/classes/GoogleSearcher.md +171 -44
- package/docs/classes/WebSearcher.md +158 -45
- package/docs/functions/extractDate.md +42 -0
- package/docs/functions/extractMetadataFrom.md +40 -0
- package/docs/functions/fetchHeaders.md +34 -0
- package/docs/functions/fetchPartial.md +41 -0
- package/docs/functions/normalizeDate.md +29 -0
- package/docs/functions/parseHeaders.md +28 -0
- package/docs/functions/parseHtml.md +31 -0
- package/docs/functions/testUrlsByLatency.md +38 -0
- package/docs/globals.md +18 -0
- package/docs/interfaces/CustomTimeRange.md +3 -3
- package/docs/interfaces/ExtractOptions.md +54 -0
- package/docs/interfaces/FetchExtractorOptions.md +35 -0
- package/docs/interfaces/FetcherOptions.md +424 -0
- package/docs/interfaces/HtmlData.md +53 -0
- package/docs/interfaces/MetadataResult.md +27 -0
- package/docs/interfaces/PaginationConfig.md +9 -9
- package/docs/interfaces/SearchContext.md +30 -4
- package/docs/interfaces/SearchOptions.md +77 -11
- package/docs/interfaces/StandardSearchResult.md +10 -10
- package/docs/interfaces/VerifiedUrl.md +25 -0
- package/docs/type-aliases/MetadataType.md +13 -0
- package/docs/type-aliases/SafeSearchLevel.md +1 -1
- package/docs/type-aliases/SearchCategory.md +2 -2
- package/docs/type-aliases/SearchTimeRange.md +1 -1
- package/docs/type-aliases/SearchTimeRangePreset.md +2 -2
- package/docs/type-aliases/SearcherConstructor.md +2 -2
- package/package.json +3 -2
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
[**@isdk/web-searcher**](../README.md)
|
|
2
|
+
|
|
3
|
+
***
|
|
4
|
+
|
|
5
|
+
[@isdk/web-searcher](../globals.md) / MetadataResult
|
|
6
|
+
|
|
7
|
+
# Interface: MetadataResult
|
|
8
|
+
|
|
9
|
+
Defined in: [web-searcher/src/utils/extractor/extractor.ts:7](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/extractor.ts#L7)
|
|
10
|
+
|
|
11
|
+
Result object for generic metadata extraction.
|
|
12
|
+
|
|
13
|
+
## Indexable
|
|
14
|
+
|
|
15
|
+
\[`key`: `string`\]: `any`
|
|
16
|
+
|
|
17
|
+
Placeholders for future metadata fields.
|
|
18
|
+
|
|
19
|
+
## Properties
|
|
20
|
+
|
|
21
|
+
### date?
|
|
22
|
+
|
|
23
|
+
> `optional` **date**: `string` \| `null`
|
|
24
|
+
|
|
25
|
+
Defined in: [web-searcher/src/utils/extractor/extractor.ts:9](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/extractor.ts#L9)
|
|
26
|
+
|
|
27
|
+
The extracted and normalized date, if any.
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: PaginationConfig
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:41](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:41](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L41)
|
|
10
10
|
|
|
11
11
|
Configuration for pagination strategies.
|
|
12
12
|
Defines how the searcher should navigate to the next page of results.
|
|
@@ -17,7 +17,7 @@ Defines how the searcher should navigate to the next page of results.
|
|
|
17
17
|
|
|
18
18
|
> `optional` **increment**: `number`
|
|
19
19
|
|
|
20
|
-
Defined in: [web-searcher/src/types.ts:68](https://github.com/isdk/web-searcher.js/blob/
|
|
20
|
+
Defined in: [web-searcher/src/types.ts:68](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L68)
|
|
21
21
|
|
|
22
22
|
The increment step for each page.
|
|
23
23
|
- If the parameter represents an item offset (like Google's 'start'), this might be 10.
|
|
@@ -35,12 +35,12 @@ The increment step for each page.
|
|
|
35
35
|
|
|
36
36
|
> `optional` **maxPages**: `number`
|
|
37
37
|
|
|
38
|
-
Defined in: [web-searcher/src/types.ts:85](https://github.com/isdk/web-searcher.js/blob/
|
|
38
|
+
Defined in: [web-searcher/src/types.ts:85](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L85)
|
|
39
39
|
|
|
40
|
-
The safety threshold for the maximum number of pages to fetch automatically
|
|
40
|
+
The safety threshold for the maximum number of pages to fetch automatically
|
|
41
41
|
in a single search call.
|
|
42
42
|
|
|
43
|
-
Even if the requested `limit` of results hasn't been reached, the searcher
|
|
43
|
+
Even if the requested `limit` of results hasn't been reached, the searcher
|
|
44
44
|
will stop after this many pages to prevent infinite loops or excessive API usage.
|
|
45
45
|
|
|
46
46
|
#### Default
|
|
@@ -55,7 +55,7 @@ will stop after this many pages to prevent infinite loops or excessive API usage
|
|
|
55
55
|
|
|
56
56
|
> `optional` **nextButtonSelector**: `string`
|
|
57
57
|
|
|
58
|
-
Defined in: [web-searcher/src/types.ts:74](https://github.com/isdk/web-searcher.js/blob/
|
|
58
|
+
Defined in: [web-searcher/src/types.ts:74](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L74)
|
|
59
59
|
|
|
60
60
|
The CSS selector for the "Next" page button.
|
|
61
61
|
Required if type is 'click-next'.
|
|
@@ -66,7 +66,7 @@ Required if type is 'click-next'.
|
|
|
66
66
|
|
|
67
67
|
> `optional` **paramName**: `string`
|
|
68
68
|
|
|
69
|
-
Defined in: [web-searcher/src/types.ts:54](https://github.com/isdk/web-searcher.js/blob/
|
|
69
|
+
Defined in: [web-searcher/src/types.ts:54](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L54)
|
|
70
70
|
|
|
71
71
|
The name of the URL parameter used for pagination.
|
|
72
72
|
Required if type is 'url-param'.
|
|
@@ -83,7 +83,7 @@ Required if type is 'url-param'.
|
|
|
83
83
|
|
|
84
84
|
> `optional` **startValue**: `number`
|
|
85
85
|
|
|
86
|
-
Defined in: [web-searcher/src/types.ts:60](https://github.com/isdk/web-searcher.js/blob/
|
|
86
|
+
Defined in: [web-searcher/src/types.ts:60](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L60)
|
|
87
87
|
|
|
88
88
|
The starting value for the pagination parameter.
|
|
89
89
|
|
|
@@ -99,7 +99,7 @@ The starting value for the pagination parameter.
|
|
|
99
99
|
|
|
100
100
|
> **type**: `"url-param"` \| `"click-next"`
|
|
101
101
|
|
|
102
|
-
Defined in: [web-searcher/src/types.ts:47](https://github.com/isdk/web-searcher.js/blob/
|
|
102
|
+
Defined in: [web-searcher/src/types.ts:47](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L47)
|
|
103
103
|
|
|
104
104
|
The type of pagination mechanism:
|
|
105
105
|
- 'url-param': Pagination is handled by modifying URL parameters (e.g., `?page=2` or `?start=10`).
|
|
@@ -6,17 +6,43 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: SearchContext
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:91](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:91](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L91)
|
|
10
10
|
|
|
11
11
|
Context object passed to the transform function.
|
|
12
12
|
|
|
13
|
+
## Indexable
|
|
14
|
+
|
|
15
|
+
\[`key`: `string`\]: `any`
|
|
16
|
+
|
|
17
|
+
Allows for custom variables passed via search options.
|
|
18
|
+
|
|
13
19
|
## Properties
|
|
14
20
|
|
|
21
|
+
### baseUrl?
|
|
22
|
+
|
|
23
|
+
> `optional` **baseUrl**: `string`
|
|
24
|
+
|
|
25
|
+
Defined in: [web-searcher/src/types.ts:105](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L105)
|
|
26
|
+
|
|
27
|
+
The baseUrl used for this specific fetch (if multi-instance is enabled)
|
|
28
|
+
|
|
29
|
+
***
|
|
30
|
+
|
|
31
|
+
### engine?
|
|
32
|
+
|
|
33
|
+
> `optional` **engine**: `string`
|
|
34
|
+
|
|
35
|
+
Defined in: [web-searcher/src/types.ts:108](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L108)
|
|
36
|
+
|
|
37
|
+
The name of the engine executing the search
|
|
38
|
+
|
|
39
|
+
***
|
|
40
|
+
|
|
15
41
|
### limit?
|
|
16
42
|
|
|
17
43
|
> `optional` **limit**: `number`
|
|
18
44
|
|
|
19
|
-
Defined in: [web-searcher/src/types.ts:99](https://github.com/isdk/web-searcher.js/blob/
|
|
45
|
+
Defined in: [web-searcher/src/types.ts:99](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L99)
|
|
20
46
|
|
|
21
47
|
The requested limit of results.
|
|
22
48
|
|
|
@@ -26,7 +52,7 @@ The requested limit of results.
|
|
|
26
52
|
|
|
27
53
|
> **page**: `number`
|
|
28
54
|
|
|
29
|
-
Defined in: [web-searcher/src/types.ts:96](https://github.com/isdk/web-searcher.js/blob/
|
|
55
|
+
Defined in: [web-searcher/src/types.ts:96](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L96)
|
|
30
56
|
|
|
31
57
|
The current page index (0-based).
|
|
32
58
|
|
|
@@ -36,6 +62,6 @@ The current page index (0-based).
|
|
|
36
62
|
|
|
37
63
|
> **query**: `string`
|
|
38
64
|
|
|
39
|
-
Defined in: [web-searcher/src/types.ts:93](https://github.com/isdk/web-searcher.js/blob/
|
|
65
|
+
Defined in: [web-searcher/src/types.ts:93](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L93)
|
|
40
66
|
|
|
41
67
|
The original search query.
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: SearchOptions
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:129](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L129)
|
|
10
10
|
|
|
11
11
|
Options provided when executing a search.
|
|
12
12
|
|
|
@@ -18,22 +18,46 @@ Any other custom variables to be injected into the template.
|
|
|
18
18
|
|
|
19
19
|
## Properties
|
|
20
20
|
|
|
21
|
+
### baseUrls?
|
|
22
|
+
|
|
23
|
+
> `optional` **baseUrls**: `string`[] \| `Record`\<`string`, `string`[]\>
|
|
24
|
+
|
|
25
|
+
Defined in: [web-searcher/src/types.ts:187](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L187)
|
|
26
|
+
|
|
27
|
+
Allows the user to dynamically specify or override the base URLs for the engines.
|
|
28
|
+
Can be an array of URLs for a single engine, or a map of engine names to URL arrays.
|
|
29
|
+
|
|
30
|
+
***
|
|
31
|
+
|
|
21
32
|
### category?
|
|
22
33
|
|
|
23
|
-
> `optional` **category**:
|
|
34
|
+
> `optional` **category**: `string`
|
|
24
35
|
|
|
25
|
-
Defined in: [web-searcher/src/types.ts:
|
|
36
|
+
Defined in: [web-searcher/src/types.ts:153](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L153)
|
|
26
37
|
|
|
27
38
|
The category of results to return.
|
|
28
39
|
Default: 'all' (web search)
|
|
29
40
|
|
|
30
41
|
***
|
|
31
42
|
|
|
43
|
+
### fillLimit?
|
|
44
|
+
|
|
45
|
+
> `optional` **fillLimit**: `boolean`
|
|
46
|
+
|
|
47
|
+
Defined in: [web-searcher/src/types.ts:204](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L204)
|
|
48
|
+
|
|
49
|
+
If true (default), the searcher will attempt to fulfill the requested `limit`
|
|
50
|
+
by falling back to subsequent engines in the chain if previous ones are exhausted.
|
|
51
|
+
If false, it will stop after the first successful engine regardless of whether
|
|
52
|
+
the limit was reached.
|
|
53
|
+
|
|
54
|
+
***
|
|
55
|
+
|
|
32
56
|
### language?
|
|
33
57
|
|
|
34
58
|
> `optional` **language**: `string`
|
|
35
59
|
|
|
36
|
-
Defined in: [web-searcher/src/types.ts:
|
|
60
|
+
Defined in: [web-searcher/src/types.ts:163](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L163)
|
|
37
61
|
|
|
38
62
|
Language code (ISO 639-1) for the interface or results (e.g., 'en', 'zh-CN').
|
|
39
63
|
|
|
@@ -43,7 +67,7 @@ Language code (ISO 639-1) for the interface or results (e.g., 'en', 'zh-CN').
|
|
|
43
67
|
|
|
44
68
|
> `optional` **limit**: `number`
|
|
45
69
|
|
|
46
|
-
Defined in: [web-searcher/src/types.ts:
|
|
70
|
+
Defined in: [web-searcher/src/types.ts:131](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L131)
|
|
47
71
|
|
|
48
72
|
The maximum number of results to retrieve.
|
|
49
73
|
|
|
@@ -53,11 +77,11 @@ The maximum number of results to retrieve.
|
|
|
53
77
|
|
|
54
78
|
> `optional` **maxPages**: `number`
|
|
55
79
|
|
|
56
|
-
Defined in: [web-searcher/src/types.ts:
|
|
80
|
+
Defined in: [web-searcher/src/types.ts:141](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L141)
|
|
57
81
|
|
|
58
82
|
The maximum number of pages (fetch cycles) allowed to reach the requested `limit`.
|
|
59
83
|
|
|
60
|
-
This is a safety guard. If the `limit` is high but each page has few results,
|
|
84
|
+
This is a safety guard. If the `limit` is high but each page has few results,
|
|
61
85
|
the searcher will stop once this page count is reached.
|
|
62
86
|
|
|
63
87
|
If not provided, it defaults to the value in `PaginationConfig` or 10.
|
|
@@ -68,7 +92,7 @@ If not provided, it defaults to the value in `PaginationConfig` or 10.
|
|
|
68
92
|
|
|
69
93
|
> `optional` **region**: `string`
|
|
70
94
|
|
|
71
|
-
Defined in: [web-searcher/src/types.ts:
|
|
95
|
+
Defined in: [web-searcher/src/types.ts:158](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L158)
|
|
72
96
|
|
|
73
97
|
Region code (ISO 3166-1 alpha-2) to bias results (e.g., 'US', 'CN', 'JP').
|
|
74
98
|
|
|
@@ -78,18 +102,35 @@ Region code (ISO 3166-1 alpha-2) to bias results (e.g., 'US', 'CN', 'JP').
|
|
|
78
102
|
|
|
79
103
|
> `optional` **safeSearch**: [`SafeSearchLevel`](../type-aliases/SafeSearchLevel.md)
|
|
80
104
|
|
|
81
|
-
Defined in: [web-searcher/src/types.ts:
|
|
105
|
+
Defined in: [web-searcher/src/types.ts:169](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L169)
|
|
82
106
|
|
|
83
107
|
Safe search filtering level.
|
|
84
108
|
Default: engine dependent (usually 'moderate' or 'strict' by default).
|
|
85
109
|
|
|
86
110
|
***
|
|
87
111
|
|
|
112
|
+
### startPage?
|
|
113
|
+
|
|
114
|
+
> `optional` **startPage**: `number`
|
|
115
|
+
|
|
116
|
+
Defined in: [web-searcher/src/types.ts:211](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L211)
|
|
117
|
+
|
|
118
|
+
Specifies which page index to start the search from.
|
|
119
|
+
Useful when delegating pagination across different sessions.
|
|
120
|
+
|
|
121
|
+
#### Default
|
|
122
|
+
|
|
123
|
+
```ts
|
|
124
|
+
0
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
***
|
|
128
|
+
|
|
88
129
|
### timeRange?
|
|
89
130
|
|
|
90
131
|
> `optional` **timeRange**: [`SearchTimeRange`](../type-aliases/SearchTimeRange.md)
|
|
91
132
|
|
|
92
|
-
Defined in: [web-searcher/src/types.ts:
|
|
133
|
+
Defined in: [web-searcher/src/types.ts:147](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L147)
|
|
93
134
|
|
|
94
135
|
Date range for the search results.
|
|
95
136
|
Default: 'all'
|
|
@@ -100,7 +141,7 @@ Default: 'all'
|
|
|
100
141
|
|
|
101
142
|
> `optional` **transform**: (`results`, `context`) => [`StandardSearchResult`](StandardSearchResult.md)[] \| `Promise`\<[`StandardSearchResult`](StandardSearchResult.md)[]\>
|
|
102
143
|
|
|
103
|
-
Defined in: [web-searcher/src/types.ts:
|
|
144
|
+
Defined in: [web-searcher/src/types.ts:175](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L175)
|
|
104
145
|
|
|
105
146
|
A custom transform function to filter or modify results at runtime.
|
|
106
147
|
This runs AFTER the engine-level transform.
|
|
@@ -118,3 +159,28 @@ This runs AFTER the engine-level transform.
|
|
|
118
159
|
#### Returns
|
|
119
160
|
|
|
120
161
|
[`StandardSearchResult`](StandardSearchResult.md)[] \| `Promise`\<[`StandardSearchResult`](StandardSearchResult.md)[]\>
|
|
162
|
+
|
|
163
|
+
***
|
|
164
|
+
|
|
165
|
+
### validator()?
|
|
166
|
+
|
|
167
|
+
> `optional` **validator**: (`results`, `context`) => `boolean` \| `Promise`\<`boolean`\>
|
|
168
|
+
|
|
169
|
+
Defined in: [web-searcher/src/types.ts:193](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L193)
|
|
170
|
+
|
|
171
|
+
User-defined callback to validate the fetched results for a page.
|
|
172
|
+
If it returns false, the fetch is considered a failure, triggering the retry/failover mechanism.
|
|
173
|
+
|
|
174
|
+
#### Parameters
|
|
175
|
+
|
|
176
|
+
##### results
|
|
177
|
+
|
|
178
|
+
[`StandardSearchResult`](StandardSearchResult.md)[]
|
|
179
|
+
|
|
180
|
+
##### context
|
|
181
|
+
|
|
182
|
+
[`SearchContext`](SearchContext.md)
|
|
183
|
+
|
|
184
|
+
#### Returns
|
|
185
|
+
|
|
186
|
+
`boolean` \| `Promise`\<`boolean`\>
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: StandardSearchResult
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:5](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:5](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L5)
|
|
10
10
|
|
|
11
11
|
Interface representing a standardized search result item.
|
|
12
12
|
This ensures consistency across different search engines.
|
|
@@ -23,7 +23,7 @@ Allows for engine-specific extra fields (e.g., siteIcon, category).
|
|
|
23
23
|
|
|
24
24
|
> `optional` **author**: `string`
|
|
25
25
|
|
|
26
|
-
Defined in: [web-searcher/src/types.ts:22](https://github.com/isdk/web-searcher.js/blob/
|
|
26
|
+
Defined in: [web-searcher/src/types.ts:22](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L22)
|
|
27
27
|
|
|
28
28
|
The author or source name of the result.
|
|
29
29
|
|
|
@@ -33,7 +33,7 @@ The author or source name of the result.
|
|
|
33
33
|
|
|
34
34
|
> `optional` **date**: `string` \| `Date`
|
|
35
35
|
|
|
36
|
-
Defined in: [web-searcher/src/types.ts:19](https://github.com/isdk/web-searcher.js/blob/
|
|
36
|
+
Defined in: [web-searcher/src/types.ts:19](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L19)
|
|
37
37
|
|
|
38
38
|
The date the result was published or last updated.
|
|
39
39
|
|
|
@@ -43,7 +43,7 @@ The date the result was published or last updated.
|
|
|
43
43
|
|
|
44
44
|
> `optional` **favicon**: `string`
|
|
45
45
|
|
|
46
|
-
Defined in: [web-searcher/src/types.ts:25](https://github.com/isdk/web-searcher.js/blob/
|
|
46
|
+
Defined in: [web-searcher/src/types.ts:25](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L25)
|
|
47
47
|
|
|
48
48
|
The favicon URL of the source website.
|
|
49
49
|
|
|
@@ -53,7 +53,7 @@ The favicon URL of the source website.
|
|
|
53
53
|
|
|
54
54
|
> `optional` **image**: `string`
|
|
55
55
|
|
|
56
|
-
Defined in: [web-searcher/src/types.ts:16](https://github.com/isdk/web-searcher.js/blob/
|
|
56
|
+
Defined in: [web-searcher/src/types.ts:16](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L16)
|
|
57
57
|
|
|
58
58
|
An optional image URL associated with the result.
|
|
59
59
|
|
|
@@ -63,7 +63,7 @@ An optional image URL associated with the result.
|
|
|
63
63
|
|
|
64
64
|
> `optional` **rank**: `number`
|
|
65
65
|
|
|
66
|
-
Defined in: [web-searcher/src/types.ts:28](https://github.com/isdk/web-searcher.js/blob/
|
|
66
|
+
Defined in: [web-searcher/src/types.ts:28](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L28)
|
|
67
67
|
|
|
68
68
|
The rank or position of the result (usually 1-indexed).
|
|
69
69
|
|
|
@@ -73,7 +73,7 @@ The rank or position of the result (usually 1-indexed).
|
|
|
73
73
|
|
|
74
74
|
> `optional` **snippet**: `string`
|
|
75
75
|
|
|
76
|
-
Defined in: [web-searcher/src/types.ts:13](https://github.com/isdk/web-searcher.js/blob/
|
|
76
|
+
Defined in: [web-searcher/src/types.ts:13](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L13)
|
|
77
77
|
|
|
78
78
|
A brief snippet or description of the result.
|
|
79
79
|
|
|
@@ -83,7 +83,7 @@ A brief snippet or description of the result.
|
|
|
83
83
|
|
|
84
84
|
> `optional` **source**: `string`
|
|
85
85
|
|
|
86
|
-
Defined in: [web-searcher/src/types.ts:31](https://github.com/isdk/web-searcher.js/blob/
|
|
86
|
+
Defined in: [web-searcher/src/types.ts:31](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L31)
|
|
87
87
|
|
|
88
88
|
The source website name (e.g., 'GitHub', 'StackOverflow').
|
|
89
89
|
|
|
@@ -93,7 +93,7 @@ The source website name (e.g., 'GitHub', 'StackOverflow').
|
|
|
93
93
|
|
|
94
94
|
> **title**: `string`
|
|
95
95
|
|
|
96
|
-
Defined in: [web-searcher/src/types.ts:7](https://github.com/isdk/web-searcher.js/blob/
|
|
96
|
+
Defined in: [web-searcher/src/types.ts:7](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L7)
|
|
97
97
|
|
|
98
98
|
The title of the search result.
|
|
99
99
|
|
|
@@ -103,6 +103,6 @@ The title of the search result.
|
|
|
103
103
|
|
|
104
104
|
> **url**: `string`
|
|
105
105
|
|
|
106
|
-
Defined in: [web-searcher/src/types.ts:10](https://github.com/isdk/web-searcher.js/blob/
|
|
106
|
+
Defined in: [web-searcher/src/types.ts:10](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L10)
|
|
107
107
|
|
|
108
108
|
The URL of the search result.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
[**@isdk/web-searcher**](../README.md)
|
|
2
|
+
|
|
3
|
+
***
|
|
4
|
+
|
|
5
|
+
[@isdk/web-searcher](../globals.md) / VerifiedUrl
|
|
6
|
+
|
|
7
|
+
# Interface: VerifiedUrl
|
|
8
|
+
|
|
9
|
+
Defined in: [web-searcher/src/utils/latency.ts:3](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/latency.ts#L3)
|
|
10
|
+
|
|
11
|
+
## Properties
|
|
12
|
+
|
|
13
|
+
### latency
|
|
14
|
+
|
|
15
|
+
> **latency**: `number`
|
|
16
|
+
|
|
17
|
+
Defined in: [web-searcher/src/utils/latency.ts:5](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/latency.ts#L5)
|
|
18
|
+
|
|
19
|
+
***
|
|
20
|
+
|
|
21
|
+
### url
|
|
22
|
+
|
|
23
|
+
> **url**: `string`
|
|
24
|
+
|
|
25
|
+
Defined in: [web-searcher/src/utils/latency.ts:4](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/latency.ts#L4)
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
[**@isdk/web-searcher**](../README.md)
|
|
2
|
+
|
|
3
|
+
***
|
|
4
|
+
|
|
5
|
+
[@isdk/web-searcher](../globals.md) / MetadataType
|
|
6
|
+
|
|
7
|
+
# Type Alias: MetadataType
|
|
8
|
+
|
|
9
|
+
> **MetadataType** = `"date"` \| `string`
|
|
10
|
+
|
|
11
|
+
Defined in: [web-searcher/src/utils/extractor/extractor.ts:17](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/extractor.ts#L17)
|
|
12
|
+
|
|
13
|
+
Supported metadata types for extraction.
|
|
@@ -8,4 +8,4 @@
|
|
|
8
8
|
|
|
9
9
|
> **SafeSearchLevel** = `"off"` \| `"moderate"` \| `"strict"`
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/types.ts:
|
|
11
|
+
Defined in: [web-searcher/src/types.ts:124](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L124)
|
|
@@ -6,6 +6,6 @@
|
|
|
6
6
|
|
|
7
7
|
# Type Alias: SearchCategory
|
|
8
8
|
|
|
9
|
-
> **SearchCategory** = `"all"` \| `"images"` \| `"videos"` \| `"news"`
|
|
9
|
+
> **SearchCategory** = `"all"` \| `"images"` \| `"videos"` \| `"news"` \| `string`
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/types.ts:
|
|
11
|
+
Defined in: [web-searcher/src/types.ts:122](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L122)
|
|
@@ -8,4 +8,4 @@
|
|
|
8
8
|
|
|
9
9
|
> **SearchTimeRange** = [`SearchTimeRangePreset`](SearchTimeRangePreset.md) \| [`CustomTimeRange`](../interfaces/CustomTimeRange.md)
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/types.ts:
|
|
11
|
+
Defined in: [web-searcher/src/types.ts:120](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L120)
|
|
@@ -6,6 +6,6 @@
|
|
|
6
6
|
|
|
7
7
|
# Type Alias: SearchTimeRangePreset
|
|
8
8
|
|
|
9
|
-
> **SearchTimeRangePreset** = `"all"` \| `"day"` \| `"week"` \| `"month"` \| `"year"`
|
|
9
|
+
> **SearchTimeRangePreset** = `"all"` \| `"hour"` \| `"day"` \| `"week"` \| `"month"` \| `"year"`
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/types.ts:
|
|
11
|
+
Defined in: [web-searcher/src/types.ts:111](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L111)
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **SearcherConstructor** = (`options?`) => [`WebSearcher`](../classes/WebSearcher.md)
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/searcher.ts:10](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/searcher.ts:10](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/searcher.ts#L10)
|
|
12
12
|
|
|
13
13
|
Constructor definition for Searcher subclasses.
|
|
14
14
|
|
|
@@ -16,7 +16,7 @@ Constructor definition for Searcher subclasses.
|
|
|
16
16
|
|
|
17
17
|
### options?
|
|
18
18
|
|
|
19
|
-
`FetcherOptions`
|
|
19
|
+
[`FetcherOptions`](../interfaces/FetcherOptions.md)
|
|
20
20
|
|
|
21
21
|
## Returns
|
|
22
22
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@isdk/web-searcher",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.5",
|
|
4
4
|
"description": "A high-level framework for building search engine scrapers, supporting multi-page navigation, session persistence, and result standardization.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Riceball LEE <snowyu.lee@gmail.com>",
|
|
@@ -21,7 +21,7 @@
|
|
|
21
21
|
"dependencies": {
|
|
22
22
|
"custom-factory": "^3.0.0",
|
|
23
23
|
"lodash-es": "^4.17.22",
|
|
24
|
-
"@isdk/web-fetcher": "^0.
|
|
24
|
+
"@isdk/web-fetcher": "^0.4.0"
|
|
25
25
|
},
|
|
26
26
|
"devDependencies": {
|
|
27
27
|
"@antfu/eslint-config": "^7.0.1",
|
|
@@ -31,6 +31,7 @@
|
|
|
31
31
|
"@types/jsdom": "^27.0.0",
|
|
32
32
|
"@types/lodash-es": "^4.17.12",
|
|
33
33
|
"@typescript-eslint/eslint-plugin": "~8.53.0",
|
|
34
|
+
"cheerio": "^1.2.0",
|
|
34
35
|
"eslint": "^9.39.2",
|
|
35
36
|
"eslint-config-prettier": "^10.1.8",
|
|
36
37
|
"eslint-plugin-tsdoc": "^0.5.0",
|