@isdk/web-searcher 0.1.4 → 0.1.6

package/docs/interfaces/FetcherOptions.md

@@ -0,0 +1,436 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / FetcherOptions
+
+# Interface: FetcherOptions
+
+Defined in: web-fetcher/dist/index.d.ts:1109
+
+## Extends
+
+- `BaseFetcherProperties`
+
+## Properties
+
+### actions?
+
+> `optional` **actions**: `_RequireAtLeastOne`\<`FetchActionProperties`, `"name"` \| `"id"` \| `"action"`\>[]
+
+Defined in: web-fetcher/dist/index.d.ts:1110
+
+***
+
+### antibot?
+
+> `optional` **antibot**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1049
+
+#### Inherited from
+
+`BaseFetcherProperties.antibot`
+
+***
+
+### blockResources?
+
+> `optional` **blockResources**: `string`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1062
+
+#### Inherited from
+
+`BaseFetcherProperties.blockResources`
+
+***
+
+### browser?
+
+> `optional` **browser**: `object`
+
+Defined in: web-fetcher/dist/index.d.ts:1072
+
+#### engine?
+
+> `optional` **engine**: `BrowserEngine`
+
+浏览器引擎，默认为 playwright
+
+- `playwright`: 使用 Playwright 引擎
+- `puppeteer`: 使用 Puppeteer 引擎
+
+#### headless?
+
+> `optional` **headless**: `boolean`
+
+#### launchOptions?
+
+> `optional` **launchOptions**: `Record`\<`string`, `any`\>
+
+#### waitUntil?
+
+> `optional` **waitUntil**: `"load"` \| `"domcontentloaded"` \| `"networkidle"` \| `"commit"`
+
+#### Inherited from
+
+`BaseFetcherProperties.browser`
+
+***
+
+### cache?
+
+> `optional` **cache**: `FetchCacheOptions`
+
+Defined in: web-fetcher/dist/index.d.ts:1070
+
+Cache configuration for persistent HTTP caching.
+
+#### Inherited from
+
+`BaseFetcherProperties.cache`
+
+***
+
+### cookies?
+
+> `optional` **cookies**: `Cookie`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1052
+
+#### Inherited from
+
+`BaseFetcherProperties.cookies`
+
+***
+
+### debug?
+
+> `optional` **debug**: `string` \| `boolean` \| `string`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1050
+
+#### Inherited from
+
+`BaseFetcherProperties.debug`
+
+***
+
+### delayBetweenRequestsMs?
+
+> `optional` **delayBetweenRequestsMs**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1092
+
+#### Inherited from
+
+`BaseFetcherProperties.delayBetweenRequestsMs`
+
+***
+
+### enableSmart?
+
+> `optional` **enableSmart**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1044
+
+#### Inherited from
+
+`BaseFetcherProperties.enableSmart`
+
+***
+
+### engine?
+
+> `optional` **engine**: `string`
+
+Defined in: web-fetcher/dist/index.d.ts:1043
+
+抓取模式
+
+- `http`: 使用 HTTP 进行抓取
+- `browser`: 使用浏览器进行抓取
+- `auto`: auto 会走“智能探测”选择 http 或 browser, 但是如果没有启用 smart，并且在站点注册表中没有，那么则等价为 http.
+
+#### Inherited from
+
+`BaseFetcherProperties.engine`
+
+***
+
+### headers?
+
+> `optional` **headers**: `Record`\<`string`, `string`\>
+
+Defined in: web-fetcher/dist/index.d.ts:1051
+
+#### Inherited from
+
+`BaseFetcherProperties.headers`
+
+***
+
+### http?
+
+> `optional` **http**: `object`
+
+Defined in: web-fetcher/dist/index.d.ts:1084
+
+#### body?
+
+> `optional` **body**: `any`
+
+#### method?
+
+> `optional` **method**: `"GET"` \| `"POST"` \| `"PUT"` \| `"PATCH"` \| `"DELETE"`
+
+#### Inherited from
+
+`BaseFetcherProperties.http`
+
+***
+
+### ignoreSslErrors?
+
+> `optional` **ignoreSslErrors**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1071
+
+#### Inherited from
+
+`BaseFetcherProperties.ignoreSslErrors`
+
+***
+
+### maxConcurrency?
+
+> `optional` **maxConcurrency**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1090
+
+#### Inherited from
+
+`BaseFetcherProperties.maxConcurrency`
+
+***
+
+### maxRequestsPerMinute?
+
+> `optional` **maxRequestsPerMinute**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1091
+
+#### Inherited from
+
+`BaseFetcherProperties.maxRequestsPerMinute`
+
+***
+
+### onPause?
+
+> `optional` **onPause**: `OnFetchPauseCallback`
+
+Defined in: web-fetcher/dist/index.d.ts:1111
+
+***
+
+### output?
+
+> `optional` **output**: `object`
+
+Defined in: web-fetcher/dist/index.d.ts:1057
+
+#### cookies?
+
+> `optional` **cookies**: `boolean`
+
+#### sessionState?
+
+> `optional` **sessionState**: `boolean`
+
+#### Inherited from
+
+`BaseFetcherProperties.output`
+
+***
+
+### overrideSessionState?
+
+> `optional` **overrideSessionState**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1055
+
+#### Inherited from
+
+`BaseFetcherProperties.overrideSessionState`
+
+***
+
+### proxy?
+
+> `optional` **proxy**: `string` \| `string`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1061
+
+#### Inherited from
+
+`BaseFetcherProperties.proxy`
+
+***
+
+### requestHandlerTimeoutSecs?
+
+> `optional` **requestHandlerTimeoutSecs**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1089
+
+#### Inherited from
+
+`BaseFetcherProperties.requestHandlerTimeoutSecs`
+
+***
+
+### retries?
+
+> `optional` **retries**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1093
+
+#### Inherited from
+
+`BaseFetcherProperties.retries`
+
+***
+
+### sessionPoolOptions?
+
+> `optional` **sessionPoolOptions**: `SessionPoolOptions`
+
+Defined in: web-fetcher/dist/index.d.ts:1054
+
+#### Inherited from
+
+`BaseFetcherProperties.sessionPoolOptions`
+
+***
+
+### sessionState?
+
+> `optional` **sessionState**: `any`
+
+Defined in: web-fetcher/dist/index.d.ts:1053
+
+#### Inherited from
+
+`BaseFetcherProperties.sessionState`
+
+***
+
+### sites?
+
+> `optional` **sites**: `FetchSite`[]
+
+Defined in: web-fetcher/dist/index.d.ts:1094
+
+#### Inherited from
+
+`BaseFetcherProperties.sites`
+
+***
+
+### storage?
+
+> `optional` **storage**: `StorageOptions`
+
+Defined in: web-fetcher/dist/index.d.ts:1066
+
+Storage configuration for session isolation and persistence.
+
+#### Inherited from
+
+`BaseFetcherProperties.storage`
+
+***
+
+### syncStateOnUpgrade?
+
+> `optional` **syncStateOnUpgrade**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1045
+
+#### Inherited from
+
+`BaseFetcherProperties.syncStateOnUpgrade`
+
+***
+
+### throwHttpErrors?
+
+> `optional` **throwHttpErrors**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1056
+
+#### Inherited from
+
+`BaseFetcherProperties.throwHttpErrors`
+
+***
+
+### timeoutMs?
+
+> `optional` **timeoutMs**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1088
+
+#### Inherited from
+
+`BaseFetcherProperties.timeoutMs`
+
+***
+
+### upgradeOnJsContent?
+
+> `optional` **upgradeOnJsContent**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1046
+
+#### Inherited from
+
+`BaseFetcherProperties.upgradeOnJsContent`
+
+***
+
+### upgradeThresholdMs?
+
+> `optional` **upgradeThresholdMs**: `number`
+
+Defined in: web-fetcher/dist/index.d.ts:1047
+
+#### Inherited from
+
+`BaseFetcherProperties.upgradeThresholdMs`
+
+***
+
+### url?
+
+> `optional` **url**: `string`
+
+Defined in: web-fetcher/dist/index.d.ts:1095
+
+#### Inherited from
+
+`BaseFetcherProperties.url`
+
+***
+
+### useSiteRegistry?
+
+> `optional` **useSiteRegistry**: `boolean`
+
+Defined in: web-fetcher/dist/index.d.ts:1048
+
+#### Inherited from
+
+`BaseFetcherProperties.useSiteRegistry`

package/docs/interfaces/HtmlData.md

@@ -0,0 +1,53 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / HtmlData
+
+# Interface: HtmlData
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:4](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/parser.ts#L4)
+
+Represents structured data extracted from an HTML document.
+
+## Properties
+
+### jsonLd
+
+> **jsonLd**: `any`[]
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:8](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/parser.ts#L8)
+
+Array of parsed JSON-LD objects found in the document.
+
+***
+
+### meta
+
+> **meta**: `Record`\<`string`, `string`\>
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:6](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/parser.ts#L6)
+
+Map of meta tag names/properties to their content. Keys are lowercase.
+
+***
+
+### time
+
+> **time**: `object`[]
+
+Defined in: [web-searcher/src/utils/extractor/parser.ts:10](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/parser.ts#L10)
+
+Array of data from HTML <time> tags.
+
+#### datetime
+
+> **datetime**: `string` \| `null`
+
+The value of the 'datetime' attribute, if present.
+
+#### text
+
+> **text**: `string`
+
+The text content within the <time> tag, with HTML stripped.

package/docs/interfaces/MetadataResult.md

@@ -0,0 +1,27 @@
+[**@isdk/web-searcher**](../README.md)
+
+***
+
+[@isdk/web-searcher](../globals.md) / MetadataResult
+
+# Interface: MetadataResult
+
+Defined in: [web-searcher/src/utils/extractor/extractor.ts:7](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/extractor.ts#L7)
+
+Result object for generic metadata extraction.
+
+## Indexable
+
+\[`key`: `string`\]: `any`
+
+Placeholders for future metadata fields.
+
+## Properties
+
+### date?
+
+> `optional` **date**: `string` \| `null`
+
+Defined in: [web-searcher/src/utils/extractor/extractor.ts:9](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/utils/extractor/extractor.ts#L9)
+
+The extracted and normalized date, if any.

package/docs/interfaces/PaginationConfig.md

@@ -6,7 +6,7 @@
 
 # Interface: PaginationConfig
 
-Defined in: [web-searcher/src/types.ts:41](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L41)
+Defined in: [web-searcher/src/types.ts:41](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L41)
 
 Configuration for pagination strategies.
 Defines how the searcher should navigate to the next page of results.
@@ -17,7 +17,7 @@ Defines how the searcher should navigate to the next page of results.
 
 > `optional` **increment**: `number`
 
-Defined in: [web-searcher/src/types.ts:68](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L68)
+Defined in: [web-searcher/src/types.ts:68](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L68)
 
 The increment step for each page.
 - If the parameter represents an item offset (like Google's 'start'), this might be 10.
@@ -35,12 +35,12 @@ The increment step for each page.
 
 > `optional` **maxPages**: `number`
 
-Defined in: [web-searcher/src/types.ts:85](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L85)
+Defined in: [web-searcher/src/types.ts:85](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L85)
 
-The safety threshold for the maximum number of pages to fetch automatically 
+The safety threshold for the maximum number of pages to fetch automatically
 in a single search call.
 
-Even if the requested `limit` of results hasn't been reached, the searcher 
+Even if the requested `limit` of results hasn't been reached, the searcher
 will stop after this many pages to prevent infinite loops or excessive API usage.
 
 #### Default
@@ -55,7 +55,7 @@ will stop after this many pages to prevent infinite loops or excessive API usage
 
 > `optional` **nextButtonSelector**: `string`
 
-Defined in: [web-searcher/src/types.ts:74](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L74)
+Defined in: [web-searcher/src/types.ts:74](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L74)
 
 The CSS selector for the "Next" page button.
 Required if type is 'click-next'.
@@ -66,7 +66,7 @@ Required if type is 'click-next'.
 
 > `optional` **paramName**: `string`
 
-Defined in: [web-searcher/src/types.ts:54](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L54)
+Defined in: [web-searcher/src/types.ts:54](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L54)
 
 The name of the URL parameter used for pagination.
 Required if type is 'url-param'.
@@ -83,7 +83,7 @@ Required if type is 'url-param'.
 
 > `optional` **startValue**: `number`
 
-Defined in: [web-searcher/src/types.ts:60](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L60)
+Defined in: [web-searcher/src/types.ts:60](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L60)
 
 The starting value for the pagination parameter.
 
@@ -99,7 +99,7 @@ The starting value for the pagination parameter.
 
 > **type**: `"url-param"` \| `"click-next"`
 
-Defined in: [web-searcher/src/types.ts:47](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L47)
+Defined in: [web-searcher/src/types.ts:47](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L47)
 
 The type of pagination mechanism:
 - 'url-param': Pagination is handled by modifying URL parameters (e.g., `?page=2` or `?start=10`).

package/docs/interfaces/SearchContext.md

@@ -6,17 +6,43 @@
 
 # Interface: SearchContext
 
-Defined in: [web-searcher/src/types.ts:91](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L91)
+Defined in: [web-searcher/src/types.ts:91](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L91)
 
 Context object passed to the transform function.
 
+## Indexable
+
+\[`key`: `string`\]: `any`
+
+Allows for custom variables passed via search options.
+
 ## Properties
 
+### baseUrl?
+
+> `optional` **baseUrl**: `string`
+
+Defined in: [web-searcher/src/types.ts:105](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L105)
+
+The baseUrl used for this specific fetch (if multi-instance is enabled)
+
+***
+
+### engine?
+
+> `optional` **engine**: `string`
+
+Defined in: [web-searcher/src/types.ts:108](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L108)
+
+The name of the engine executing the search
+
+***
+
 ### limit?
 
 > `optional` **limit**: `number`
 
-Defined in: [web-searcher/src/types.ts:99](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L99)
+Defined in: [web-searcher/src/types.ts:99](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L99)
 
 The requested limit of results.
 
@@ -26,7 +52,7 @@ The requested limit of results.
 
 > **page**: `number`
 
-Defined in: [web-searcher/src/types.ts:96](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L96)
+Defined in: [web-searcher/src/types.ts:96](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L96)
 
 The current page index (0-based).
 
@@ -36,6 +62,6 @@ The current page index (0-based).
 
 > **query**: `string`
 
-Defined in: [web-searcher/src/types.ts:93](https://github.com/isdk/web-searcher.js/blob/7bcd8cca4a3a7fc201a5cf3e3b4283f267eadcea/src/types.ts#L93)
+Defined in: [web-searcher/src/types.ts:93](https://github.com/isdk/web-searcher.js/blob/0c4757eb75b3b7c5af0231806f11e7b3c3166736/src/types.ts#L93)
 
 The original search query.