@isdk/web-searcher 0.1.4 → 0.1.6

package/docs/interfaces/SearchOptions.md

@@ -6,7 +6,7 @@
 
 # Interface: SearchOptions
 
-Defined in: [web-searcher/src/types.ts:120](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L120)
+Defined in: [web-searcher/src/types.ts:129](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L129)
 
 Options provided when executing a search.
 
@@ -18,22 +18,46 @@ Any other custom variables to be injected into the template.
 
 ## Properties
 
+### baseUrls?
+
+> `optional` **baseUrls**: `string`[] \| `Record`\<`string`, `string`[]\>
+
+Defined in: [web-searcher/src/types.ts:187](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L187)
+
+Allows the user to dynamically specify or override the base URLs for the engines.
+Can be an array of URLs for a single engine, or a map of engine names to URL arrays.
+
+***
+
 ### category?
 
-> `optional` **category**: [`SearchCategory`](../type-aliases/SearchCategory.md)
+> `optional` **category**: `string`
 
-Defined in: [web-searcher/src/types.ts:144](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L144)
+Defined in: [web-searcher/src/types.ts:153](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L153)
 
 The category of results to return.
 Default: 'all' (web search)
 
 ***
 
+### fillLimit?
+
+> `optional` **fillLimit**: `boolean`
+
+Defined in: [web-searcher/src/types.ts:204](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L204)
+
+If true (default), the searcher will attempt to fulfill the requested `limit`
+by falling back to subsequent engines in the chain if previous ones are exhausted.
+If false, it will stop after the first successful engine regardless of whether 
+the limit was reached.
+
+***
+
 ### language?
 
 > `optional` **language**: `string`
 
-Defined in: [web-searcher/src/types.ts:154](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L154)
+Defined in: [web-searcher/src/types.ts:163](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L163)
 
 Language code (ISO 639-1) for the interface or results (e.g., 'en', 'zh-CN').
 
@@ -43,7 +67,7 @@ Language code (ISO 639-1) for the interface or results (e.g., 'en', 'zh-CN').
 
 > `optional` **limit**: `number`
 
-Defined in: [web-searcher/src/types.ts:122](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L122)
+Defined in: [web-searcher/src/types.ts:131](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L131)
 
 The maximum number of results to retrieve.
 
@@ -53,11 +77,11 @@ The maximum number of results to retrieve.
 
 > `optional` **maxPages**: `number`
 
-Defined in: [web-searcher/src/types.ts:132](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L132)
+Defined in: [web-searcher/src/types.ts:141](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L141)
 
 The maximum number of pages (fetch cycles) allowed to reach the requested `limit`.
 
-This is a safety guard. If the `limit` is high but each page has few results, 
+This is a safety guard. If the `limit` is high but each page has few results,
 the searcher will stop once this page count is reached.
 
 If not provided, it defaults to the value in `PaginationConfig` or 10.
@@ -68,7 +92,7 @@ If not provided, it defaults to the value in `PaginationConfig` or 10.
 
 > `optional` **region**: `string`
 
-Defined in: [web-searcher/src/types.ts:149](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L149)
+Defined in: [web-searcher/src/types.ts:158](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L158)
 
 Region code (ISO 3166-1 alpha-2) to bias results (e.g., 'US', 'CN', 'JP').
 
@@ -78,18 +102,35 @@ Region code (ISO 3166-1 alpha-2) to bias results (e.g., 'US', 'CN', 'JP').
 
 > `optional` **safeSearch**: [`SafeSearchLevel`](../type-aliases/SafeSearchLevel.md)
 
-Defined in: [web-searcher/src/types.ts:160](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L160)
+Defined in: [web-searcher/src/types.ts:169](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L169)
 
 Safe search filtering level.
 Default: engine dependent (usually 'moderate' or 'strict' by default).
 
 ***
 
+### startPage?
+
+> `optional` **startPage**: `number`
+
+Defined in: [web-searcher/src/types.ts:211](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L211)
+
+Specifies which page index to start the search from.
+Useful when delegating pagination across different sessions.
+
+#### Default
+
+```ts
+0
+```
+
+***
+
 ### timeRange?
 
 > `optional` **timeRange**: [`SearchTimeRange`](../type-aliases/SearchTimeRange.md)
 
-Defined in: [web-searcher/src/types.ts:138](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L138)
+Defined in: [web-searcher/src/types.ts:147](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L147)
 
 Date range for the search results.
 Default: 'all'
@@ -100,7 +141,7 @@ Default: 'all'
 
 > `optional` **transform**: (`results`, `context`) => [`StandardSearchResult`](StandardSearchResult.md)[] \| `Promise`\<[`StandardSearchResult`](StandardSearchResult.md)[]\>
 
-Defined in: [web-searcher/src/types.ts:166](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L166)
+Defined in: [web-searcher/src/types.ts:175](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L175)
 
 A custom transform function to filter or modify results at runtime.
 This runs AFTER the engine-level transform.
@@ -118,3 +159,28 @@ This runs AFTER the engine-level transform.
 #### Returns
 
 [`StandardSearchResult`](StandardSearchResult.md)[] \| `Promise`\<[`StandardSearchResult`](StandardSearchResult.md)[]\>
+
+***
+
+### validator()?
+
+> `optional` **validator**: (`results`, `context`) => `boolean` \| `Promise`\<`boolean`\>
+
+Defined in: [web-searcher/src/types.ts:193](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L193)
+
+User-defined callback to validate the fetched results for a page.
+If it returns false, the fetch is considered a failure, triggering the retry/failover mechanism.
+
+#### Parameters
+
+##### results
+
+[`StandardSearchResult`](StandardSearchResult.md)[]
+
+##### context
+
+[`SearchContext`](SearchContext.md)
+
+#### Returns
+
+`boolean` \| `Promise`\<`boolean`\>

package/docs/interfaces/StandardSearchResult.md

@@ -6,7 +6,7 @@
 
 # Interface: StandardSearchResult
 
-Defined in: [web-searcher/src/types.ts:5](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L5)
+Defined in: [web-searcher/src/types.ts:5](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L5)
 
 Interface representing a standardized search result item.
 This ensures consistency across different search engines.
@@ -23,7 +23,7 @@ Allows for engine-specific extra fields (e.g., siteIcon, category).
 
 > `optional` **author**: `string`
 
-Defined in: [web-searcher/src/types.ts:22](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L22)
+Defined in: [web-searcher/src/types.ts:22](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L22)
 
 The author or source name of the result.
 
@@ -33,7 +33,7 @@ The author or source name of the result.
 
 > `optional` **date**: `string` \| `Date`
 
-Defined in: [web-searcher/src/types.ts:19](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L19)
+Defined in: [web-searcher/src/types.ts:19](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L19)
 
 The date the result was published or last updated.
 
@@ -43,7 +43,7 @@ The date the result was published or last updated.
 
 > `optional` **favicon**: `string`
 
-Defined in: [web-searcher/src/types.ts:25](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L25)
+Defined in: [web-searcher/src/types.ts:25](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L25)
 
 The favicon URL of the source website.
 
@@ -53,7 +53,7 @@ The favicon URL of the source website.
 
 > `optional` **image**: `string`
 
-Defined in: [web-searcher/src/types.ts:16](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L16)
+Defined in: [web-searcher/src/types.ts:16](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L16)
 
 An optional image URL associated with the result.
 
@@ -63,7 +63,7 @@ An optional image URL associated with the result.
 
 > `optional` **rank**: `number`
 
-Defined in: [web-searcher/src/types.ts:28](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L28)
+Defined in: [web-searcher/src/types.ts:28](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L28)
 
 The rank or position of the result (usually 1-indexed).
 
@@ -73,7 +73,7 @@ The rank or position of the result (usually 1-indexed).
 
 > `optional` **snippet**: `string`
 
-Defined in: [web-searcher/src/types.ts:13](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L13)
+Defined in: [web-searcher/src/types.ts:13](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L13)
 
 A brief snippet or description of the result.
 
@@ -83,7 +83,7 @@ A brief snippet or description of the result.
 
 > `optional` **source**: `string`
 
-Defined in: [web-searcher/src/types.ts:31](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L31)
+Defined in: [web-searcher/src/types.ts:31](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L31)
 
 The source website name (e.g., 'GitHub', 'StackOverflow').
 
@@ -93,7 +93,7 @@ The source website name (e.g., 'GitHub', 'StackOverflow').
 
 > **title**: `string`
 
-Defined in: [web-searcher/src/types.ts:7](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L7)
+Defined in: [web-searcher/src/types.ts:7](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L7)
 
 The title of the search result.
 
@@ -103,6 +103,6 @@ The title of the search result.
 
 > **url**: `string`
 
-Defined in: [web-searcher/src/types.ts:10](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L10)
+Defined in: [web-searcher/src/types.ts:10](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L10)
 
 The URL of the search result.

package/docs/interfaces/VerifiedUrl.md

@@ -0,0 +1,25 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / VerifiedUrl
+
+# Interface: VerifiedUrl
+
+Defined in: [web-searcher/src/utils/latency.ts:3](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/latency.ts#L3)
+
+## Properties
+
+### latency
+
+> **latency**: `number`
+
+Defined in: [web-searcher/src/utils/latency.ts:5](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/latency.ts#L5)
+
+***
+
+### url
+
+> **url**: `string`
+
+Defined in: [web-searcher/src/utils/latency.ts:4](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/latency.ts#L4)

package/docs/type-aliases/MetadataType.md

@@ -0,0 +1,13 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / MetadataType
+
+# Type Alias: MetadataType
+
+> **MetadataType** = `"date"` \| `string`
+
+Defined in: [web-searcher/src/utils/extractor/extractor.ts:17](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/extractor.ts#L17)
+
+Supported metadata types for extraction.

package/docs/type-aliases/SafeSearchLevel.md

@@ -8,4 +8,4 @@
 
 > **SafeSearchLevel** = `"off"` \| `"moderate"` \| `"strict"`
 
-Defined in: [web-searcher/src/types.ts:115](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L115)
+Defined in: [web-searcher/src/types.ts:124](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L124)

package/docs/type-aliases/SearchCategory.md

@@ -6,6 +6,6 @@
 
 # Type Alias: SearchCategory
 
-> **SearchCategory** = `"all"` \| `"images"` \| `"videos"` \| `"news"`
+> **SearchCategory** = `"all"` \| `"images"` \| `"videos"` \| `"news"` \| `string`
 
-Defined in: [web-searcher/src/types.ts:113](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L113)
+Defined in: [web-searcher/src/types.ts:122](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L122)

package/docs/type-aliases/SearchTimeRange.md

@@ -8,4 +8,4 @@
 
 > **SearchTimeRange** = [`SearchTimeRangePreset`](SearchTimeRangePreset.md) \| [`CustomTimeRange`](../interfaces/CustomTimeRange.md)
 
-Defined in: [web-searcher/src/types.ts:111](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L111)
+Defined in: [web-searcher/src/types.ts:120](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L120)

package/docs/type-aliases/SearchTimeRangePreset.md

@@ -8,4 +8,4 @@
 
 > **SearchTimeRangePreset** = `"all"` \| `"hour"` \| `"day"` \| `"week"` \| `"month"` \| `"year"`
 
-Defined in: [web-searcher/src/types.ts:102](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L102)
+Defined in: [web-searcher/src/types.ts:111](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L111)

package/docs/type-aliases/SearcherConstructor.md

@@ -8,7 +8,7 @@
 
 > **SearcherConstructor** = (`options?`) => [`WebSearcher`](../classes/WebSearcher.md)
 
-Defined in: [web-searcher/src/searcher.ts:10](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/searcher.ts#L10)
+Defined in: [web-searcher/src/searcher.ts:10](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/searcher.ts#L10)
 
 Constructor definition for Searcher subclasses.
 
@@ -16,7 +16,7 @@ Constructor definition for Searcher subclasses.
 
 ### options?
 
-`FetcherOptions`
+[`FetcherOptions`](../interfaces/FetcherOptions.md)
 
 ## Returns
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@isdk/web-searcher",
-  "version": "0.1.4",
+  "version": "0.1.6",
   "description": "A high-level framework for building search engine scrapers, supporting multi-page navigation, session persistence, and result standardization.",
   "license": "MIT",
   "author": "Riceball LEE <snowyu.lee@gmail.com>",
@@ -21,7 +21,7 @@
   "dependencies": {
     "custom-factory": "^3.0.0",
     "lodash-es": "^4.17.22",
-    "@isdk/web-fetcher": "^0.2.12"
+    "@isdk/web-fetcher": "^0.4.1"
   },
   "devDependencies": {
     "@antfu/eslint-config": "^7.0.1",
@@ -31,6 +31,7 @@
     "@types/jsdom": "^27.0.0",
     "@types/lodash-es": "^4.17.12",
     "@typescript-eslint/eslint-plugin": "~8.53.0",
+    "cheerio": "^1.2.0",
     "eslint": "^9.39.2",
     "eslint-config-prettier": "^10.1.8",
     "eslint-plugin-tsdoc": "^0.5.0",