@isdk/web-searcher 0.1.4 → 0.1.6

package/docs/functions/fetchHeaders.md

@@ -0,0 +1,34 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / fetchHeaders
+
+# Function: fetchHeaders()
+
+> **fetchHeaders**(`url`, `options`): `Promise`\<`Headers` \| `null`\>
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:19](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L19)
+
+Fetches only the HTTP headers for a given URL using a HEAD request.
+Useful for checking 'last-modified' without downloading the body.
+
+## Parameters
+
+### url
+
+`string`
+
+The URL to check.
+
+### options
+
+[`FetchExtractorOptions`](../interfaces/FetchExtractorOptions.md) = `{}`
+
+Request options.
+
+## Returns
+
+`Promise`\<`Headers` \| `null`\>
+
+The Headers object, or null on failure.

package/docs/functions/fetchPartial.md

@@ -0,0 +1,41 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / fetchPartial
+
+# Function: fetchPartial()
+
+> **fetchPartial**(`url`, `maxBytes`, `options`): `Promise`\<\{ `content`: `string`; `headers`: `Headers`; \} \| `null`\>
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:55](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L55)
+
+Fetches a partial amount of content from a URL.
+Automatically handles character set detection from the Content-Type header.
+Aborts the request once the specified maxBytes is reached.
+
+## Parameters
+
+### url
+
+`string`
+
+The URL to fetch.
+
+### maxBytes
+
+`number` = `32768`
+
+The maximum number of bytes to read. Defaults to 32KB.
+
+### options
+
+[`FetchExtractorOptions`](../interfaces/FetchExtractorOptions.md) = `{}`
+
+Request options.
+
+## Returns
+
+`Promise`\<\{ `content`: `string`; `headers`: `Headers`; \} \| `null`\>
+
+An object containing the decoded content string and the response headers.

package/docs/functions/normalizeDate.md

@@ -0,0 +1,29 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / normalizeDate
+
+# Function: normalizeDate()
+
+> **normalizeDate**(`dateStr`): `string` \| `null`
+
+Defined in: [web-searcher/src/utils/extractor/date-normalizer.ts:9](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/date-normalizer.ts#L9)
+
+Normalizes a date string into a standard ISO 8601 format (UTC).
+It handles various formats (YYYY-MM-DD, RFC2822, etc.) and performs
+aggressive cleaning and sanity checks.
+
+## Parameters
+
+### dateStr
+
+The raw date string to normalize.
+
+`string` | `null`
+
+## Returns
+
+`string` \| `null`
+
+An ISO 8601 string (e.g., "2024-01-20T00:00:00.000Z") or null if invalid.

package/docs/functions/parseHeaders.md

@@ -0,0 +1,28 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / parseHeaders
+
+# Function: parseHeaders()
+
+> **parseHeaders**(`headers`): `Record`\<`string`, `string`\>
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:25](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/parser.ts#L25)
+
+Converts a Web API Headers object into a plain JavaScript record.
+All header names are converted to lowercase for consistent access.
+
+## Parameters
+
+### headers
+
+`Headers`
+
+The Headers object to parse.
+
+## Returns
+
+`Record`\<`string`, `string`\>
+
+A record where keys are lowercase header names.

package/docs/functions/parseHtml.md

@@ -0,0 +1,31 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / parseHtml
+
+# Function: parseHtml()
+
+> **parseHtml**(`html`): [`HtmlData`](../interfaces/HtmlData.md)
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:49](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/parser.ts#L49)
+
+Parses an HTML string to extract generic metadata structures (Meta tags, JSON-LD, Time tags).
+
+This function does not perform field-specific logic (like finding a date); it simply
+
+collects available structured data.
+
+## Parameters
+
+### html
+
+`string`
+
+The raw HTML content to parse.
+
+## Returns
+
+[`HtmlData`](../interfaces/HtmlData.md)
+
+An object containing grouped metadata from the HTML.

package/docs/functions/testUrlsByLatency.md

@@ -0,0 +1,42 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / testUrlsByLatency
+
+# Function: testUrlsByLatency()
+
+> **testUrlsByLatency**(`urls`, `options`): `Promise`\<[`VerifiedUrl`](../interfaces/VerifiedUrl.md)[]\>
+
+Defined in: [web-searcher/src/utils/latency.ts:12](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/latency.ts#L12)
+
+A general utility to test a list of URLs for availability and latency.
+Returns a list of verified URLs sorted by response time.
+
+## Parameters
+
+### urls
+
+`string`[]
+
+### options
+
+#### limit?
+
+`number`
+
+#### proxy?
+
+`string`
+
+#### testPath?
+
+`string`
+
+#### timeout?
+
+`number`
+
+## Returns
+
+`Promise`\<[`VerifiedUrl`](../interfaces/VerifiedUrl.md)[]\>

package/docs/globals.md

@@ -12,15 +12,33 @@
 ## Interfaces
 
 - [CustomTimeRange](interfaces/CustomTimeRange.md)
+- [ExtractOptions](interfaces/ExtractOptions.md)
+- [FetcherOptions](interfaces/FetcherOptions.md)
+- [FetchExtractorOptions](interfaces/FetchExtractorOptions.md)
+- [HtmlData](interfaces/HtmlData.md)
+- [MetadataResult](interfaces/MetadataResult.md)
 - [PaginationConfig](interfaces/PaginationConfig.md)
 - [SearchContext](interfaces/SearchContext.md)
 - [SearchOptions](interfaces/SearchOptions.md)
 - [StandardSearchResult](interfaces/StandardSearchResult.md)
+- [VerifiedUrl](interfaces/VerifiedUrl.md)
 
 ## Type Aliases
 
+- [MetadataType](type-aliases/MetadataType.md)
 - [SafeSearchLevel](type-aliases/SafeSearchLevel.md)
 - [SearchCategory](type-aliases/SearchCategory.md)
 - [SearcherConstructor](type-aliases/SearcherConstructor.md)
 - [SearchTimeRange](type-aliases/SearchTimeRange.md)
 - [SearchTimeRangePreset](type-aliases/SearchTimeRangePreset.md)
+
+## Functions
+
+- [extractDate](functions/extractDate.md)
+- [extractMetadataFrom](functions/extractMetadataFrom.md)
+- [fetchHeaders](functions/fetchHeaders.md)
+- [fetchPartial](functions/fetchPartial.md)
+- [normalizeDate](functions/normalizeDate.md)
+- [parseHeaders](functions/parseHeaders.md)
+- [parseHtml](functions/parseHtml.md)
+- [testUrlsByLatency](functions/testUrlsByLatency.md)

package/docs/interfaces/CustomTimeRange.md

@@ -6,7 +6,7 @@
 
 # Interface: CustomTimeRange
 
-Defined in: [web-searcher/src/types.ts:104](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L104)
+Defined in: [web-searcher/src/types.ts:113](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L113)
 
 ## Properties
 
@@ -14,7 +14,7 @@ Defined in: [web-searcher/src/types.ts:104](https://github.com/isdk/web-searcher
 
 > **from**: `string` \| `Date`
 
-Defined in: [web-searcher/src/types.ts:106](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L106)
+Defined in: [web-searcher/src/types.ts:115](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L115)
 
 Start date (Date object or string like 'YYYY-MM-DD').
 
@@ -24,6 +24,6 @@ Start date (Date object or string like 'YYYY-MM-DD').
 
 > `optional` **to**: `string` \| `Date`
 
-Defined in: [web-searcher/src/types.ts:108](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L108)
+Defined in: [web-searcher/src/types.ts:117](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L117)
 
 End date (Date object or string like 'YYYY-MM-DD'). Defaults to current date if omitted.

package/docs/interfaces/ExtractOptions.md

@@ -0,0 +1,54 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / ExtractOptions
+
+# Interface: ExtractOptions
+
+Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:7](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/date-extractor.ts#L7)
+
+Options for the extractDate function.
+
+## Extends
+
+- [`FetchExtractorOptions`](FetchExtractorOptions.md)
+
+## Properties
+
+### headers?
+
+> `optional` **headers**: `Record`\<`string`, `string`\>
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:8](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L8)
+
+Custom HTTP headers to include in the request.
+
+#### Inherited from
+
+[`FetchExtractorOptions`](FetchExtractorOptions.md).[`headers`](FetchExtractorOptions.md#headers)
+
+***
+
+### maxBytes?
+
+> `optional` **maxBytes**: `number`
+
+Defined in: [web-searcher/src/utils/extractor/date-extractor.ts:12](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/date-extractor.ts#L12)
+
+Maximum number of bytes to download from the URL.
+Defaults to 32768 (32KB), which is usually enough for the HTML <head>.
+
+***
+
+### timeout?
+
+> `optional` **timeout**: `number`
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:6](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L6)
+
+Timeout in milliseconds. Defaults vary by function (5s to 10s).
+
+#### Inherited from
+
+[`FetchExtractorOptions`](FetchExtractorOptions.md).[`timeout`](FetchExtractorOptions.md#timeout)

package/docs/interfaces/FetchExtractorOptions.md

@@ -0,0 +1,35 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / FetchExtractorOptions
+
+# Interface: FetchExtractorOptions
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:4](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L4)
+
+Options for network requests.
+
+## Extended by
+
+- [`ExtractOptions`](ExtractOptions.md)
+
+## Properties
+
+### headers?
+
+> `optional` **headers**: `Record`\<`string`, `string`\>
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:8](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L8)
+
+Custom HTTP headers to include in the request.
+
+***
+
+### timeout?
+
+> `optional` **timeout**: `number`
+
+Defined in: [web-searcher/src/utils/extractor/fetcher.ts:6](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/fetcher.ts#L6)
+
+Timeout in milliseconds. Defaults vary by function (5s to 10s).