@isdk/web-searcher 0.1.4 → 0.1.5

package/docs/functions/parseHtml.md

@@ -0,0 +1,31 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / parseHtml
+
+# Function: parseHtml()
+
+> **parseHtml**(`html`): [`HtmlData`](../interfaces/HtmlData.md)
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:49](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/parser.ts#L49)
+
+Parses an HTML string to extract generic metadata structures (Meta tags, JSON-LD, Time tags).
+
+This function does not perform field-specific logic (like finding a date); it simply
+
+collects available structured data.
+
+## Parameters
+
+### html
+
+`string`
+
+The raw HTML content to parse.
+
+## Returns
+
+[`HtmlData`](../interfaces/HtmlData.md)
+
+An object containing grouped metadata from the HTML.

package/docs/functions/testUrlsByLatency.md

@@ -0,0 +1,38 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / testUrlsByLatency
+
+# Function: testUrlsByLatency()
+
+> **testUrlsByLatency**(`urls`, `options`): `Promise`\<[`VerifiedUrl`](../interfaces/VerifiedUrl.md)[]\>
+
+Defined in: [web-searcher/src/utils/latency.ts:12](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/latency.ts#L12)
+
+A general utility to test a list of URLs for availability and latency.
+Returns a list of verified URLs sorted by response time.
+
+## Parameters
+
+### urls
+
+`string`[]
+
+### options
+
+#### limit?
+
+`number`
+
+#### testPath?
+
+`string`
+
+#### timeout?
+
+`number`
+
+## Returns
+
+`Promise`\<[`VerifiedUrl`](../interfaces/VerifiedUrl.md)[]\>

package/docs/globals.md

@@ -12,15 +12,33 @@
 ## Interfaces
 
 - [CustomTimeRange](interfaces/CustomTimeRange.md)
+- [ExtractOptions](interfaces/ExtractOptions.md)
+- [FetcherOptions](interfaces/FetcherOptions.md)
+- [FetchExtractorOptions](interfaces/FetchExtractorOptions.md)
+- [HtmlData](interfaces/HtmlData.md)
+- [MetadataResult](interfaces/MetadataResult.md)
 - [PaginationConfig](interfaces/PaginationConfig.md)
 - [SearchContext](interfaces/SearchContext.md)
 - [SearchOptions](interfaces/SearchOptions.md)
 - [StandardSearchResult](interfaces/StandardSearchResult.md)
+- [VerifiedUrl](interfaces/VerifiedUrl.md)
 
 ## Type Aliases
 
+- [MetadataType](type-aliases/MetadataType.md)
 - [SafeSearchLevel](type-aliases/SafeSearchLevel.md)
 - [SearchCategory](type-aliases/SearchCategory.md)
 - [SearcherConstructor](type-aliases/SearcherConstructor.md)
 - [SearchTimeRange](type-aliases/SearchTimeRange.md)
 - [SearchTimeRangePreset](type-aliases/SearchTimeRangePreset.md)
+
+## Functions
+
+- [extractDate](functions/extractDate.md)
+- [extractMetadataFrom](functions/extractMetadataFrom.md)
+- [fetchHeaders](functions/fetchHeaders.md)
+- [fetchPartial](functions/fetchPartial.md)
+- [normalizeDate](functions/normalizeDate.md)
+- [parseHeaders](functions/parseHeaders.md)
+- [parseHtml](functions/parseHtml.md)
+- [testUrlsByLatency](functions/testUrlsByLatency.md)

package/docs/interfaces/CustomTimeRange.md

@@ -6,7 +6,7 @@
 
 # Interface: CustomTimeRange
 
-Defined in: [web-searcher/src/types.ts:104](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L104)
+Defined in: [web-searcher/src/types.ts:113](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L113)
 
 ## Properties
 
@@ -14,7 +14,7 @@ Defined in: [web-searcher/src/types.ts:104](https://github.com/isdk/web-searcher
 
 > **from**: `string` \| `Date`
 
-Defined in: [web-searcher/src/types.ts:106](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L106)
+Defined in: [web-searcher/src/types.ts:115](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L115)
 
 Start date (Date object or string like 'YYYY-MM-DD').
 
@@ -24,6 +24,6 @@ Start date (Date object or string like 'YYYY-MM-DD').
 
 > `optional` **to**: `string` \| `Date`
 
-Defined in: [web-searcher/src/types.ts:108](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L108)
+Defined in: [web-searcher/src/types.ts:117](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/types.ts#L117)
 
 End date (Date object or string like 'YYYY-MM-DD'). Defaults to current date if omitted.

package/docs/interfaces/ExtractOptions.md

@@ -0,0 +1,54 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / ExtractOptions
+
+# Interface: ExtractOptions
+
+Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:7](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/date-extractor.ts#L7)
+
+Options for the extractDate function.
+
+## Extends
+
+- [`FetchExtractorOptions`](FetchExtractorOptions.md)
+
+## Properties
+
+### headers?
+
+> `optional` **headers**: `Record`\<`string`, `string`\>
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:8](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/fetcher.ts#L8)
+
+Custom HTTP headers to include in the request.
+
+#### Inherited from
+
+[`FetchExtractorOptions`](FetchExtractorOptions.md).[`headers`](FetchExtractorOptions.md#headers)
+
+***
+
+### maxBytes?
+
+> `optional` **maxBytes**: `number`
+
+Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:12](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/date-extractor.ts#L12)
+
+Maximum number of bytes to download from the URL.
+Defaults to 32768 (32KB), which is usually enough for the HTML <head>.
+
+***
+
+### timeout?
+
+> `optional` **timeout**: `number`
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:6](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/fetcher.ts#L6)
+
+Timeout in milliseconds. Defaults vary by function (5s to 10s).
+
+#### Inherited from
+
+[`FetchExtractorOptions`](FetchExtractorOptions.md).[`timeout`](FetchExtractorOptions.md#timeout)

package/docs/interfaces/FetchExtractorOptions.md

@@ -0,0 +1,35 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / FetchExtractorOptions
+
+# Interface: FetchExtractorOptions
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:4](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/fetcher.ts#L4)
+
+Options for network requests.
+
+## Extended by
+
+- [`ExtractOptions`](ExtractOptions.md)
+
+## Properties
+
+### headers?
+
+> `optional` **headers**: `Record`\<`string`, `string`\>
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:8](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/fetcher.ts#L8)
+
+Custom HTTP headers to include in the request.
+
+***
+
+### timeout?
+
+> `optional` **timeout**: `number`
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:6](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/fetcher.ts#L6)
+
+Timeout in milliseconds. Defaults vary by function (5s to 10s).

package/docs/interfaces/FetcherOptions.md

@@ -0,0 +1,424 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / FetcherOptions
+
+# Interface: FetcherOptions
+
+Defined in: web-fetcher/dist/index.d.ts:1108
+
+## Extends
+
+- `BaseFetcherProperties`
+
+## Properties
+
+### actions?
+
+> `optional` **actions**: `_RequireAtLeastOne`\<`FetchActionProperties`, `"name"` \| `"id"` \| `"action"`\>[]
+
+Defined in: web-fetcher/dist/index.d.ts:1109
+
+***
+
+### antibot?
+
+> `optional` **antibot**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1048
+
+#### Inherited from
+
+`BaseFetcherProperties.antibot`
+
+***
+
+### blockResources?
+
+> `optional` **blockResources**: `string`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1061
+
+#### Inherited from
+
+`BaseFetcherProperties.blockResources`
+
+***
+
+### browser?
+
+> `optional` **browser**: `object`
+
+Defined in: web-fetcher/dist/index.d.ts:1071
+
+#### engine?
+
+> `optional` **engine**: `BrowserEngine`
+
+浏览器引擎，默认为 playwright
+
+- `playwright`: 使用 Playwright 引擎
+- `puppeteer`: 使用 Puppeteer 引擎
+
+#### headless?
+
+> `optional` **headless**: `boolean`
+
+#### launchOptions?
+
+> `optional` **launchOptions**: `Record`\<`string`, `any`\>
+
+#### waitUntil?
+
+> `optional` **waitUntil**: `"load"` \| `"domcontentloaded"` \| `"networkidle"` \| `"commit"`
+
+#### Inherited from
+
+`BaseFetcherProperties.browser`
+
+***
+
+### cache?
+
+> `optional` **cache**: `FetchCacheOptions`
+
+Defined in: web-fetcher/dist/index.d.ts:1069
+
+Cache configuration for persistent HTTP caching.
+
+#### Inherited from
+
+`BaseFetcherProperties.cache`
+
+***
+
+### cookies?
+
+> `optional` **cookies**: `Cookie`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1051
+
+#### Inherited from
+
+`BaseFetcherProperties.cookies`
+
+***
+
+### debug?
+
+> `optional` **debug**: `string` \| `boolean` \| `string`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1049
+
+#### Inherited from
+
+`BaseFetcherProperties.debug`
+
+***
+
+### delayBetweenRequestsMs?
+
+> `optional` **delayBetweenRequestsMs**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1091
+
+#### Inherited from
+
+`BaseFetcherProperties.delayBetweenRequestsMs`
+
+***
+
+### enableSmart?
+
+> `optional` **enableSmart**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1044
+
+#### Inherited from
+
+`BaseFetcherProperties.enableSmart`
+
+***
+
+### engine?
+
+> `optional` **engine**: `string`
+
+Defined in: web-fetcher/dist/index.d.ts:1043
+
+抓取模式
+
+- `http`: 使用 HTTP 进行抓取
+- `browser`: 使用浏览器进行抓取
+- `auto`: auto 会走“智能探测”选择 http 或 browser, 但是如果没有启用 smart，并且在站点注册表中没有，那么则等价为 http.
+
+#### Inherited from
+
+`BaseFetcherProperties.engine`
+
+***
+
+### headers?
+
+> `optional` **headers**: `Record`\<`string`, `string`\>
+
+Defined in: web-fetcher/dist/index.d.ts:1050
+
+#### Inherited from
+
+`BaseFetcherProperties.headers`
+
+***
+
+### http?
+
+> `optional` **http**: `object`
+
+Defined in: web-fetcher/dist/index.d.ts:1083
+
+#### body?
+
+> `optional` **body**: `any`
+
+#### method?
+
+> `optional` **method**: `"GET"` \| `"POST"` \| `"PUT"` \| `"PATCH"` \| `"DELETE"`
+
+#### Inherited from
+
+`BaseFetcherProperties.http`
+
+***
+
+### ignoreSslErrors?
+
+> `optional` **ignoreSslErrors**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1070
+
+#### Inherited from
+
+`BaseFetcherProperties.ignoreSslErrors`
+
+***
+
+### maxConcurrency?
+
+> `optional` **maxConcurrency**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1089
+
+#### Inherited from
+
+`BaseFetcherProperties.maxConcurrency`
+
+***
+
+### maxRequestsPerMinute?
+
+> `optional` **maxRequestsPerMinute**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1090
+
+#### Inherited from
+
+`BaseFetcherProperties.maxRequestsPerMinute`
+
+***
+
+### onPause?
+
+> `optional` **onPause**: `OnFetchPauseCallback`
+
+Defined in: web-fetcher/dist/index.d.ts:1110
+
+***
+
+### output?
+
+> `optional` **output**: `object`
+
+Defined in: web-fetcher/dist/index.d.ts:1056
+
+#### cookies?
+
+> `optional` **cookies**: `boolean`
+
+#### sessionState?
+
+> `optional` **sessionState**: `boolean`
+
+#### Inherited from
+
+`BaseFetcherProperties.output`
+
+***
+
+### overrideSessionState?
+
+> `optional` **overrideSessionState**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1054
+
+#### Inherited from
+
+`BaseFetcherProperties.overrideSessionState`
+
+***
+
+### proxy?
+
+> `optional` **proxy**: `string` \| `string`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1060
+
+#### Inherited from
+
+`BaseFetcherProperties.proxy`
+
+***
+
+### requestHandlerTimeoutSecs?
+
+> `optional` **requestHandlerTimeoutSecs**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1088
+
+#### Inherited from
+
+`BaseFetcherProperties.requestHandlerTimeoutSecs`
+
+***
+
+### retries?
+
+> `optional` **retries**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1092
+
+#### Inherited from
+
+`BaseFetcherProperties.retries`
+
+***
+
+### sessionPoolOptions?
+
+> `optional` **sessionPoolOptions**: `SessionPoolOptions`
+
+Defined in: web-fetcher/dist/index.d.ts:1053
+
+#### Inherited from
+
+`BaseFetcherProperties.sessionPoolOptions`
+
+***
+
+### sessionState?
+
+> `optional` **sessionState**: `any`
+
+Defined in: web-fetcher/dist/index.d.ts:1052
+
+#### Inherited from
+
+`BaseFetcherProperties.sessionState`
+
+***
+
+### sites?
+
+> `optional` **sites**: `FetchSite`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1093
+
+#### Inherited from
+
+`BaseFetcherProperties.sites`
+
+***
+
+### storage?
+
+> `optional` **storage**: `StorageOptions`
+
+Defined in: web-fetcher/dist/index.d.ts:1065
+
+Storage configuration for session isolation and persistence.
+
+#### Inherited from
+
+`BaseFetcherProperties.storage`
+
+***
+
+### syncStateOnUpgrade?
+
+> `optional` **syncStateOnUpgrade**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1045
+
+#### Inherited from
+
+`BaseFetcherProperties.syncStateOnUpgrade`
+
+***
+
+### throwHttpErrors?
+
+> `optional` **throwHttpErrors**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1055
+
+#### Inherited from
+
+`BaseFetcherProperties.throwHttpErrors`
+
+***
+
+### timeoutMs?
+
+> `optional` **timeoutMs**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1087
+
+#### Inherited from
+
+`BaseFetcherProperties.timeoutMs`
+
+***
+
+### upgradeThresholdMs?
+
+> `optional` **upgradeThresholdMs**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1046
+
+#### Inherited from
+
+`BaseFetcherProperties.upgradeThresholdMs`
+
+***
+
+### url?
+
+> `optional` **url**: `string`
+
+Defined in: web-fetcher/dist/index.d.ts:1094
+
+#### Inherited from
+
+`BaseFetcherProperties.url`
+
+***
+
+### useSiteRegistry?
+
+> `optional` **useSiteRegistry**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1047
+
+#### Inherited from
+
+`BaseFetcherProperties.useSiteRegistry`

package/docs/interfaces/HtmlData.md

@@ -0,0 +1,53 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / HtmlData
+
+# Interface: HtmlData
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:4](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/parser.ts#L4)
+
+Represents structured data extracted from an HTML document.
+
+## Properties
+
+### jsonLd
+
+> **jsonLd**: `any`[]
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:8](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/parser.ts#L8)
+
+Array of parsed JSON-LD objects found in the document.
+
+***
+
+### meta
+
+> **meta**: `Record`\<`string`, `string`\>
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:6](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/parser.ts#L6)
+
+Map of meta tag names/properties to their content. Keys are lowercase.
+
+***
+
+### time
+
+> **time**: `object`[]
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:10](https://github.com/isdk/web-searcher.js/blob/955bc509edda39926bd12c6c2b8c28da7eb13ff5/src/utils/extractor/parser.ts#L10)
+
+Array of data from HTML <time> tags.
+
+#### datetime
+
+> **datetime**: `string` \| `null`
+
+The value of the 'datetime' attribute, if present.
+
+#### text
+
+> **text**: `string`
+
+The text content within the <time> tag, with HTML stripped.