@isdk/web-searcher 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.cn.md +20 -15
- package/README.md +20 -15
- package/docs/README.md +20 -15
- package/docs/classes/GoogleSearcher.md +17 -17
- package/docs/classes/WebSearcher.md +17 -17
- package/docs/interfaces/CustomTimeRange.md +3 -3
- package/docs/interfaces/PaginationConfig.md +6 -6
- package/docs/interfaces/SearchContext.md +4 -4
- package/docs/interfaces/SearchOptions.md +8 -8
- package/docs/interfaces/StandardSearchResult.md +5 -5
- package/docs/type-aliases/SafeSearchLevel.md +1 -1
- package/docs/type-aliases/SearchCategory.md +1 -1
- package/docs/type-aliases/SearchTimeRange.md +1 -1
- package/docs/type-aliases/SearchTimeRangePreset.md +1 -1
- package/docs/type-aliases/SearcherConstructor.md +1 -1
- package/package.json +16 -4
package/README.cn.md
CHANGED
|
@@ -11,36 +11,41 @@ Search 模块提供了一个基于类的高级框架,用于构建搜索引擎
|
|
|
11
11
|
- **数据清洗**: 解析原始 HTML 并处理重定向链接。
|
|
12
12
|
- **灵活性**: 轻松切换 HTTP(快速)和 Browser(抗反爬)模式。
|
|
13
13
|
|
|
14
|
-
本模块将这些通用模式封装在一个可复用的 `
|
|
14
|
+
本模块将这些通用模式封装在一个可复用的 `WebSearcher` 类中。
|
|
15
15
|
|
|
16
16
|
## 🚀 快速开始
|
|
17
17
|
|
|
18
18
|
### 1. 一次性搜索 (One-off Search)
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
> **⚠️ 关于 `GoogleSearcher` 的说明**:这些示例中使用的 `GoogleSearcher` 类仅作为**演示实现**用于教学目的。它不适用于生产环境。
|
|
21
|
+
>
|
|
22
|
+
> * 它缺乏大规模可靠抓取 Google 所需的高级反爬虫处理(如验证码破解、代理轮换)。
|
|
23
|
+
> * 由于 Google 频繁的 DOM 变更和 A/B 测试,提取的数据可能会出现**不准确或信息错位**的情况。
|
|
24
|
+
|
|
25
|
+
使用静态方法 `WebSearcher.search` 处理快速、用完即弃的任务。它会自动创建会话、抓取结果并进行清理。
|
|
21
26
|
|
|
22
27
|
```typescript
|
|
23
|
-
import {
|
|
24
|
-
import { GoogleSearcher } from '@isdk/web-fetcher/search/engines/google';
|
|
28
|
+
import { GoogleSearcher, WebSearcher } from '@isdk/web-fetcher';
|
|
25
29
|
|
|
26
30
|
// 注册引擎 (只需执行一次)
|
|
27
|
-
|
|
31
|
+
WebSearcher.register(GoogleSearcher);
|
|
28
32
|
|
|
29
33
|
// 搜索!
|
|
30
34
|
// 'limit' 参数确保我们会自动翻页直到获取 20 条结果。
|
|
31
35
|
// 注意:引擎名称区分大小写,且由类名自动提取(例如:'GoogleSearcher' -> 'Google')
|
|
32
|
-
const results = await
|
|
36
|
+
const results = await WebSearcher.search('Google', 'open source', { limit: 20 });
|
|
33
37
|
|
|
34
38
|
console.log(results);
|
|
35
39
|
```
|
|
36
40
|
|
|
37
41
|
### 2. 有状态会话 (Stateful Session)
|
|
38
42
|
|
|
39
|
-
由于 `
|
|
43
|
+
由于 `WebSearcher` 继承自 `FetchSession`,您可以实例化它以在多个请求之间保持 Cookie 和存储。这对于需要登录的搜索或通过模拟人类行为来避免反爬虫非常有用。
|
|
40
44
|
|
|
41
45
|
**配置优先级:**
|
|
42
46
|
创建会话时,选项按以下顺序合并:
|
|
43
|
-
|
|
47
|
+
|
|
48
|
+
1. **模板默认 (Template Default)**:在 WebSearcher 类中定义(结构化选项的优先级最高)。
|
|
44
49
|
2. **用户选项 (User Options)**:传递给构造函数的选项(可填充缺失的默认值,或在允许的情况下进行覆盖)。
|
|
45
50
|
|
|
46
51
|
*注:如果模板设置了 `engine: 'auto'`(默认值),则会尊重用户提供的 `engine` 选项。*
|
|
@@ -71,11 +76,11 @@ try {
|
|
|
71
76
|
|
|
72
77
|
## 🛠️ 实现一个新的搜索引擎
|
|
73
78
|
|
|
74
|
-
要支持一个新的网站,请创建一个继承自 `
|
|
79
|
+
要支持一个新的网站,请创建一个继承自 `WebSearcher` 的类。
|
|
75
80
|
|
|
76
81
|
### 步骤 1: 定义模板 (Template)
|
|
77
82
|
|
|
78
|
-
要支持一个新的网站,请创建一个继承自 `
|
|
83
|
+
要支持一个新的网站,请创建一个继承自 `WebSearcher` 的类。引擎名称默认由类名自动提取(例如:`MyBlogSearcher` -> `MyBlog`),但您可以通过静态属性自定义名称和别名。
|
|
79
84
|
|
|
80
85
|
`template` 属性定义了搜索的“蓝图”。它是一个标准的 `FetcherOptions` 对象,但支持**变量注入**。
|
|
81
86
|
|
|
@@ -87,10 +92,10 @@ try {
|
|
|
87
92
|
- `${limit}`: 请求的限制数量。
|
|
88
93
|
|
|
89
94
|
```typescript
|
|
90
|
-
import {
|
|
95
|
+
import { WebSearcher } from '@isdk/web-fetcher/search';
|
|
91
96
|
import { FetcherOptions } from '@isdk/web-fetcher/types';
|
|
92
97
|
|
|
93
|
-
export class MyBlogSearcher extends
|
|
98
|
+
export class MyBlogSearcher extends WebSearcher {
|
|
94
99
|
static name = 'blog'; // 自定义名称 (区分大小写)
|
|
95
100
|
static alias = ['myblog', 'news'];
|
|
96
101
|
|
|
@@ -120,7 +125,7 @@ export class MyBlogSearcher extends Searcher {
|
|
|
120
125
|
|
|
121
126
|
### 步骤 2: 配置分页 (Pagination)
|
|
122
127
|
|
|
123
|
-
告诉 `
|
|
128
|
+
告诉 `WebSearcher` 如何导航到下一页。实现 `pagination` 获取器。
|
|
124
129
|
|
|
125
130
|
#### 方案 A: URL 参数 (Offset/Page)
|
|
126
131
|
|
|
@@ -152,7 +157,7 @@ protected override get pagination() {
|
|
|
152
157
|
|
|
153
158
|
### 步骤 3: 转换与清洗数据 (Transform)
|
|
154
159
|
|
|
155
|
-
重写 `transform` 以清洗数据。由于 `
|
|
160
|
+
重写 `transform` 以清洗数据。由于 `WebSearcher` 本身就是一个 `FetchSession`,您还可以使用 `this` 发起额外的请求(如解析重定向)。
|
|
156
161
|
|
|
157
162
|
```typescript
|
|
158
163
|
protected override async transform(outputs: Record<string, any>) {
|
|
@@ -171,7 +176,7 @@ protected override async transform(outputs: Record<string, any>) {
|
|
|
171
176
|
|
|
172
177
|
### 自动分页与过滤
|
|
173
178
|
|
|
174
|
-
`
|
|
179
|
+
`WebSearcher` 是智能的。如果您请求 `limit: 10`,但第一页只返回了 5 条结果(或者如果您的 `transform` 过滤掉了一些结果),它会自动抓取下一页,直到满足限制。
|
|
175
180
|
|
|
176
181
|
### 用户自定义转换 (User-defined Transforms)
|
|
177
182
|
|
package/README.md
CHANGED
|
@@ -11,36 +11,41 @@ Building a robust search scraper involves more than just fetching a URL. You oft
|
|
|
11
11
|
- **Data Cleaning**: Parse raw HTML and resolve redirect links.
|
|
12
12
|
- **Flexibility**: Switch between HTTP (fast) and Browser (anti-bot) modes easily.
|
|
13
13
|
|
|
14
|
-
This module encapsulates these patterns into a reusable `
|
|
14
|
+
This module encapsulates these patterns into a reusable `WebSearcher` class.
|
|
15
15
|
|
|
16
16
|
## 🚀 Quick Start
|
|
17
17
|
|
|
18
18
|
### 1. One-off Search
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
> **⚠️ Note on `GoogleSearcher`**: The `GoogleSearcher` class used in these examples is a **demo implementation** included for educational purposes. It is not intended for production use.
|
|
21
|
+
>
|
|
22
|
+
> * It lacks advanced anti-bot handling (CAPTCHA solving, proxy rotation) required for scraping Google reliably at scale.
|
|
23
|
+
> * The extracted data may be **inaccurate or misaligned** due to Google's frequent DOM changes and A/B testing.
|
|
24
|
+
|
|
25
|
+
Use the static `WebSearcher.search` method for quick, disposable tasks. It automatically creates a session, fetches results, and cleans up.
|
|
21
26
|
|
|
22
27
|
```typescript
|
|
23
|
-
import {
|
|
24
|
-
import { GoogleSearcher } from '@isdk/web-fetcher/search/engines/google';
|
|
28
|
+
import { GoogleSearcher, WebSearcher } from '@isdk/web-fetcher';
|
|
25
29
|
|
|
26
30
|
// Register the engine (only needs to be done once)
|
|
27
|
-
|
|
31
|
+
WebSearcher.register(GoogleSearcher);
|
|
28
32
|
|
|
29
33
|
// Search!
|
|
30
34
|
// The 'limit' parameter ensures we fetch enough pages to get 20 results.
|
|
31
35
|
// Note: The engine name is case-sensitive and derived from the class name (e.g., 'GoogleSearcher' -> 'Google')
|
|
32
|
-
const results = await
|
|
36
|
+
const results = await WebSearcher.search('Google', 'open source', { limit: 20 });
|
|
33
37
|
|
|
34
38
|
console.log(results);
|
|
35
39
|
```
|
|
36
40
|
|
|
37
41
|
### 2. Stateful Session
|
|
38
42
|
|
|
39
|
-
Since `
|
|
43
|
+
Since `WebSearcher` extends `FetchSession`, you can instantiate it to keep cookies and storage alive across multiple requests. This is useful for authenticated searches or avoiding bot detection by behaving like a human.
|
|
40
44
|
|
|
41
45
|
**Configuration Precedence:**
|
|
42
46
|
When creating a session, options are merged in the following order:
|
|
43
|
-
|
|
47
|
+
|
|
48
|
+
1. **Template Default**: Defined in the WebSearcher class (highest priority for structural options).
|
|
44
49
|
2. **User Options**: Passed to the constructor (can fill missing defaults or override if allowed).
|
|
45
50
|
|
|
46
51
|
*Note: If the template sets `engine: 'auto'` (default), user-provided `engine` option will be respected.*
|
|
@@ -71,11 +76,11 @@ try {
|
|
|
71
76
|
|
|
72
77
|
## 🛠️ Implementing a New Search Engine
|
|
73
78
|
|
|
74
|
-
To support a new website, create a class that extends `
|
|
79
|
+
To support a new website, create a class that extends `WebSearcher`.
|
|
75
80
|
|
|
76
81
|
### Step 1: Define the Template
|
|
77
82
|
|
|
78
|
-
To support a new website, create a class that extends `
|
|
83
|
+
To support a new website, create a class that extends `WebSearcher`. The engine name is automatically derived from the class name (e.g., `MyBlogSearcher` -> `MyBlog`), but you can customize it and add aliases using static properties.
|
|
79
84
|
|
|
80
85
|
The `template` property defines the "Blueprint" for your search. It's a standard `FetcherOptions` object but supports **variable injection**.
|
|
81
86
|
|
|
@@ -87,10 +92,10 @@ Supported variables:
|
|
|
87
92
|
- `${limit}`: The requested limit.
|
|
88
93
|
|
|
89
94
|
```typescript
|
|
90
|
-
import {
|
|
95
|
+
import { WebSearcher } from '@isdk/web-fetcher/search';
|
|
91
96
|
import { FetcherOptions } from '@isdk/web-fetcher/types';
|
|
92
97
|
|
|
93
|
-
export class MyBlogSearcher extends
|
|
98
|
+
export class MyBlogSearcher extends WebSearcher {
|
|
94
99
|
static name = 'blog'; // Custom name (case-sensitive)
|
|
95
100
|
static alias = ['myblog', 'news'];
|
|
96
101
|
|
|
@@ -120,7 +125,7 @@ export class MyBlogSearcher extends Searcher {
|
|
|
120
125
|
|
|
121
126
|
### Step 2: Configure Pagination
|
|
122
127
|
|
|
123
|
-
Tell the `
|
|
128
|
+
Tell the `WebSearcher` how to navigate to the next page. Implement the `pagination` getter.
|
|
124
129
|
|
|
125
130
|
#### Option A: URL Parameters (Offset/Page)
|
|
126
131
|
|
|
@@ -152,7 +157,7 @@ protected override get pagination() {
|
|
|
152
157
|
|
|
153
158
|
### Step 3: Transform & Clean Data
|
|
154
159
|
|
|
155
|
-
Override `transform` to clean data. Since `
|
|
160
|
+
Override `transform` to clean data. Since `WebSearcher` is a `FetchSession`, you can also make extra requests (like resolving redirects) using `this`.
|
|
156
161
|
|
|
157
162
|
```typescript
|
|
158
163
|
protected override async transform(outputs: Record<string, any>) {
|
|
@@ -171,7 +176,7 @@ protected override async transform(outputs: Record<string, any>) {
|
|
|
171
176
|
|
|
172
177
|
### Auto-Pagination & Filtering
|
|
173
178
|
|
|
174
|
-
The `
|
|
179
|
+
The `WebSearcher` is smart. If you request `limit: 10`, but the first page only returns 5 results (or if your `transform` filters out results), it will automatically fetch the next page until the limit is met.
|
|
175
180
|
|
|
176
181
|
### User-defined Transforms
|
|
177
182
|
|
package/docs/README.md
CHANGED
|
@@ -15,36 +15,41 @@ Building a robust search scraper involves more than just fetching a URL. You oft
|
|
|
15
15
|
- **Data Cleaning**: Parse raw HTML and resolve redirect links.
|
|
16
16
|
- **Flexibility**: Switch between HTTP (fast) and Browser (anti-bot) modes easily.
|
|
17
17
|
|
|
18
|
-
This module encapsulates these patterns into a reusable `
|
|
18
|
+
This module encapsulates these patterns into a reusable `WebSearcher` class.
|
|
19
19
|
|
|
20
20
|
## 🚀 Quick Start
|
|
21
21
|
|
|
22
22
|
### 1. One-off Search
|
|
23
23
|
|
|
24
|
-
|
|
24
|
+
> **⚠️ Note on `GoogleSearcher`**: The `GoogleSearcher` class used in these examples is a **demo implementation** included for educational purposes. It is not intended for production use.
|
|
25
|
+
>
|
|
26
|
+
> * It lacks advanced anti-bot handling (CAPTCHA solving, proxy rotation) required for scraping Google reliably at scale.
|
|
27
|
+
> * The extracted data may be **inaccurate or misaligned** due to Google's frequent DOM changes and A/B testing.
|
|
28
|
+
|
|
29
|
+
Use the static `WebSearcher.search` method for quick, disposable tasks. It automatically creates a session, fetches results, and cleans up.
|
|
25
30
|
|
|
26
31
|
```typescript
|
|
27
|
-
import {
|
|
28
|
-
import { GoogleSearcher } from '@isdk/web-fetcher/search/engines/google';
|
|
32
|
+
import { GoogleSearcher, WebSearcher } from '@isdk/web-fetcher';
|
|
29
33
|
|
|
30
34
|
// Register the engine (only needs to be done once)
|
|
31
|
-
|
|
35
|
+
WebSearcher.register(GoogleSearcher);
|
|
32
36
|
|
|
33
37
|
// Search!
|
|
34
38
|
// The 'limit' parameter ensures we fetch enough pages to get 20 results.
|
|
35
39
|
// Note: The engine name is case-sensitive and derived from the class name (e.g., 'GoogleSearcher' -> 'Google')
|
|
36
|
-
const results = await
|
|
40
|
+
const results = await WebSearcher.search('Google', 'open source', { limit: 20 });
|
|
37
41
|
|
|
38
42
|
console.log(results);
|
|
39
43
|
```
|
|
40
44
|
|
|
41
45
|
### 2. Stateful Session
|
|
42
46
|
|
|
43
|
-
Since `
|
|
47
|
+
Since `WebSearcher` extends `FetchSession`, you can instantiate it to keep cookies and storage alive across multiple requests. This is useful for authenticated searches or avoiding bot detection by behaving like a human.
|
|
44
48
|
|
|
45
49
|
**Configuration Precedence:**
|
|
46
50
|
When creating a session, options are merged in the following order:
|
|
47
|
-
|
|
51
|
+
|
|
52
|
+
1. **Template Default**: Defined in the WebSearcher class (highest priority for structural options).
|
|
48
53
|
2. **User Options**: Passed to the constructor (can fill missing defaults or override if allowed).
|
|
49
54
|
|
|
50
55
|
*Note: If the template sets `engine: 'auto'` (default), user-provided `engine` option will be respected.*
|
|
@@ -75,11 +80,11 @@ try {
|
|
|
75
80
|
|
|
76
81
|
## 🛠️ Implementing a New Search Engine
|
|
77
82
|
|
|
78
|
-
To support a new website, create a class that extends `
|
|
83
|
+
To support a new website, create a class that extends `WebSearcher`.
|
|
79
84
|
|
|
80
85
|
### Step 1: Define the Template
|
|
81
86
|
|
|
82
|
-
To support a new website, create a class that extends `
|
|
87
|
+
To support a new website, create a class that extends `WebSearcher`. The engine name is automatically derived from the class name (e.g., `MyBlogSearcher` -> `MyBlog`), but you can customize it and add aliases using static properties.
|
|
83
88
|
|
|
84
89
|
The `template` property defines the "Blueprint" for your search. It's a standard `FetcherOptions` object but supports **variable injection**.
|
|
85
90
|
|
|
@@ -91,10 +96,10 @@ Supported variables:
|
|
|
91
96
|
- `${limit}`: The requested limit.
|
|
92
97
|
|
|
93
98
|
```typescript
|
|
94
|
-
import {
|
|
99
|
+
import { WebSearcher } from '@isdk/web-fetcher/search';
|
|
95
100
|
import { FetcherOptions } from '@isdk/web-fetcher/types';
|
|
96
101
|
|
|
97
|
-
export class MyBlogSearcher extends
|
|
102
|
+
export class MyBlogSearcher extends WebSearcher {
|
|
98
103
|
static name = 'blog'; // Custom name (case-sensitive)
|
|
99
104
|
static alias = ['myblog', 'news'];
|
|
100
105
|
|
|
@@ -124,7 +129,7 @@ export class MyBlogSearcher extends Searcher {
|
|
|
124
129
|
|
|
125
130
|
### Step 2: Configure Pagination
|
|
126
131
|
|
|
127
|
-
Tell the `
|
|
132
|
+
Tell the `WebSearcher` how to navigate to the next page. Implement the `pagination` getter.
|
|
128
133
|
|
|
129
134
|
#### Option A: URL Parameters (Offset/Page)
|
|
130
135
|
|
|
@@ -156,7 +161,7 @@ protected override get pagination() {
|
|
|
156
161
|
|
|
157
162
|
### Step 3: Transform & Clean Data
|
|
158
163
|
|
|
159
|
-
Override `transform` to clean data. Since `
|
|
164
|
+
Override `transform` to clean data. Since `WebSearcher` is a `FetchSession`, you can also make extra requests (like resolving redirects) using `this`.
|
|
160
165
|
|
|
161
166
|
```typescript
|
|
162
167
|
protected override async transform(outputs: Record<string, any>) {
|
|
@@ -175,7 +180,7 @@ protected override async transform(outputs: Record<string, any>) {
|
|
|
175
180
|
|
|
176
181
|
### Auto-Pagination & Filtering
|
|
177
182
|
|
|
178
|
-
The `
|
|
183
|
+
The `WebSearcher` is smart. If you request `limit: 10`, but the first page only returns 5 results (or if your `transform` filters out results), it will automatically fetch the next page until the limit is met.
|
|
179
184
|
|
|
180
185
|
### User-defined Transforms
|
|
181
186
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Class: GoogleSearcher
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/engines/google.ts:24](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/engines/google.ts:24](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/engines/google.ts#L24)
|
|
10
10
|
|
|
11
11
|
A sample implementation of a Google Search scraper.
|
|
12
12
|
|
|
@@ -115,7 +115,7 @@ Defined in: web-fetcher/dist/index.d.ts:2177
|
|
|
115
115
|
|
|
116
116
|
> `static` **\_isFactory**: `boolean` = `false`
|
|
117
117
|
|
|
118
|
-
Defined in: [web-searcher/src/searcher.ts:33](https://github.com/isdk/web-searcher.js/blob/
|
|
118
|
+
Defined in: [web-searcher/src/searcher.ts:33](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L33)
|
|
119
119
|
|
|
120
120
|
#### Inherited from
|
|
121
121
|
|
|
@@ -127,7 +127,7 @@ Defined in: [web-searcher/src/searcher.ts:33](https://github.com/isdk/web-search
|
|
|
127
127
|
|
|
128
128
|
> `static` **alias**: `string`[]
|
|
129
129
|
|
|
130
|
-
Defined in: [web-searcher/src/engines/google.ts:25](https://github.com/isdk/web-searcher.js/blob/
|
|
130
|
+
Defined in: [web-searcher/src/engines/google.ts:25](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/engines/google.ts#L25)
|
|
131
131
|
|
|
132
132
|
Engine alias(es). Can be a single string or an array of strings.
|
|
133
133
|
Useful for registering shorthand names (e.g., 'g' for 'Google').
|
|
@@ -142,7 +142,7 @@ Useful for registering shorthand names (e.g., 'g' for 'Google').
|
|
|
142
142
|
|
|
143
143
|
> `static` **createObject**: (`name`, ...`args`) => [`WebSearcher`](WebSearcher.md)
|
|
144
144
|
|
|
145
|
-
Defined in: [web-searcher/src/searcher.ts:78](https://github.com/isdk/web-searcher.js/blob/
|
|
145
|
+
Defined in: [web-searcher/src/searcher.ts:78](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L78)
|
|
146
146
|
|
|
147
147
|
Creates an instance of the registered search engine.
|
|
148
148
|
|
|
@@ -176,7 +176,7 @@ An instance of the search engine.
|
|
|
176
176
|
|
|
177
177
|
> `static` **forEach**: (`cb`) => `void`
|
|
178
178
|
|
|
179
|
-
Defined in: [web-searcher/src/searcher.ts:85](https://github.com/isdk/web-searcher.js/blob/
|
|
179
|
+
Defined in: [web-searcher/src/searcher.ts:85](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L85)
|
|
180
180
|
|
|
181
181
|
Iterates over all registered engines.
|
|
182
182
|
|
|
@@ -202,7 +202,7 @@ Callback function to invoke for each registered engine.
|
|
|
202
202
|
|
|
203
203
|
> `static` **get**: (`name`) => *typeof* [`WebSearcher`](WebSearcher.md)
|
|
204
204
|
|
|
205
|
-
Defined in: [web-searcher/src/searcher.ts:69](https://github.com/isdk/web-searcher.js/blob/
|
|
205
|
+
Defined in: [web-searcher/src/searcher.ts:69](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L69)
|
|
206
206
|
|
|
207
207
|
Retrieves a registered search engine class by name.
|
|
208
208
|
|
|
@@ -230,7 +230,7 @@ The search engine class constructor.
|
|
|
230
230
|
|
|
231
231
|
> `static` `optional` **name**: `string`
|
|
232
232
|
|
|
233
|
-
Defined in: [web-searcher/src/searcher.ts:40](https://github.com/isdk/web-searcher.js/blob/
|
|
233
|
+
Defined in: [web-searcher/src/searcher.ts:40](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L40)
|
|
234
234
|
|
|
235
235
|
Custom engine name. If not provided, it is derived from the class name.
|
|
236
236
|
For example, `GoogleSearcher` becomes `Google`.
|
|
@@ -245,7 +245,7 @@ For example, `GoogleSearcher` becomes `Google`.
|
|
|
245
245
|
|
|
246
246
|
> `static` **register**: (`ctor`, `options?`) => `boolean`
|
|
247
247
|
|
|
248
|
-
Defined in: [web-searcher/src/searcher.ts:54](https://github.com/isdk/web-searcher.js/blob/
|
|
248
|
+
Defined in: [web-searcher/src/searcher.ts:54](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L54)
|
|
249
249
|
|
|
250
250
|
Registers a search engine class.
|
|
251
251
|
|
|
@@ -279,7 +279,7 @@ Registration options. If a string is provided, it is used as the registered name
|
|
|
279
279
|
|
|
280
280
|
> `static` **setAliases**: (`ctor`, ...`aliases`) => `void`
|
|
281
281
|
|
|
282
|
-
Defined in: [web-searcher/src/searcher.ts:93](https://github.com/isdk/web-searcher.js/blob/
|
|
282
|
+
Defined in: [web-searcher/src/searcher.ts:93](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L93)
|
|
283
283
|
|
|
284
284
|
Sets aliases for a registered engine.
|
|
285
285
|
|
|
@@ -311,7 +311,7 @@ Aliases to add.
|
|
|
311
311
|
|
|
312
312
|
> `static` **unregister**: (`name?`) => `void`
|
|
313
313
|
|
|
314
|
-
Defined in: [web-searcher/src/searcher.ts:61](https://github.com/isdk/web-searcher.js/blob/
|
|
314
|
+
Defined in: [web-searcher/src/searcher.ts:61](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L61)
|
|
315
315
|
|
|
316
316
|
Unregisters a search engine.
|
|
317
317
|
|
|
@@ -339,7 +339,7 @@ The name or class to unregister.
|
|
|
339
339
|
|
|
340
340
|
> **get** **pagination**(): [`PaginationConfig`](../interfaces/PaginationConfig.md)
|
|
341
341
|
|
|
342
|
-
Defined in: [web-searcher/src/engines/google.ts:61](https://github.com/isdk/web-searcher.js/blob/
|
|
342
|
+
Defined in: [web-searcher/src/engines/google.ts:61](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/engines/google.ts#L61)
|
|
343
343
|
|
|
344
344
|
Configures pagination for Google Search results.
|
|
345
345
|
Uses the 'start' URL parameter, incrementing by 10 for each page.
|
|
@@ -360,7 +360,7 @@ Uses the 'start' URL parameter, incrementing by 10 for each page.
|
|
|
360
360
|
|
|
361
361
|
> **get** **template**(): `FetcherOptions`
|
|
362
362
|
|
|
363
|
-
Defined in: [web-searcher/src/engines/google.ts:32](https://github.com/isdk/web-searcher.js/blob/
|
|
363
|
+
Defined in: [web-searcher/src/engines/google.ts:32](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/engines/google.ts#L32)
|
|
364
364
|
|
|
365
365
|
Defines the fetch template for Google Search.
|
|
366
366
|
|
|
@@ -380,7 +380,7 @@ The fetcher configuration including the URL pattern and extraction rules.
|
|
|
380
380
|
|
|
381
381
|
> `protected` **createContext**(`options`): `FetchContext`
|
|
382
382
|
|
|
383
|
-
Defined in: [web-searcher/src/searcher.ts:155](https://github.com/isdk/web-searcher.js/blob/
|
|
383
|
+
Defined in: [web-searcher/src/searcher.ts:155](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L155)
|
|
384
384
|
|
|
385
385
|
#### Parameters
|
|
386
386
|
|
|
@@ -517,7 +517,7 @@ const { result, outputs } = await session.executeAll([
|
|
|
517
517
|
|
|
518
518
|
> `protected` **formatOptions**(`options`): `Record`\<`string`, `any`\>
|
|
519
519
|
|
|
520
|
-
Defined in: [web-searcher/src/engines/google.ts:82](https://github.com/isdk/web-searcher.js/blob/
|
|
520
|
+
Defined in: [web-searcher/src/engines/google.ts:82](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/engines/google.ts#L82)
|
|
521
521
|
|
|
522
522
|
Maps standard `SearchOptions` to Google's specific URL parameters.
|
|
523
523
|
|
|
@@ -591,7 +591,7 @@ A promise resolving to the session state, or undefined if no engine is initializ
|
|
|
591
591
|
|
|
592
592
|
> **search**(`query`, `options`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
593
593
|
|
|
594
|
-
Defined in: [web-searcher/src/searcher.ts:182](https://github.com/isdk/web-searcher.js/blob/
|
|
594
|
+
Defined in: [web-searcher/src/searcher.ts:182](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L182)
|
|
595
595
|
|
|
596
596
|
Executes a search query.
|
|
597
597
|
|
|
@@ -628,7 +628,7 @@ A promise resolving to an array of standardized search results.
|
|
|
628
628
|
|
|
629
629
|
> `protected` **transform**(`outputs`): `Promise`\<`any`[]\>
|
|
630
630
|
|
|
631
|
-
Defined in: [web-searcher/src/engines/google.ts:144](https://github.com/isdk/web-searcher.js/blob/
|
|
631
|
+
Defined in: [web-searcher/src/engines/google.ts:144](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/engines/google.ts#L144)
|
|
632
632
|
|
|
633
633
|
Cleans and normalizes the extracted results.
|
|
634
634
|
Specifically, it unwraps Google's redirect URLs (starting with `/url?q=`).
|
|
@@ -657,7 +657,7 @@ An array of cleaned search results.
|
|
|
657
657
|
|
|
658
658
|
> `static` **search**(`engineName`, `query`, `options`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
659
659
|
|
|
660
|
-
Defined in: [web-searcher/src/searcher.ts:106](https://github.com/isdk/web-searcher.js/blob/
|
|
660
|
+
Defined in: [web-searcher/src/searcher.ts:106](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L106)
|
|
661
661
|
|
|
662
662
|
Static helper to execute a one-off search.
|
|
663
663
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Abstract Class: WebSearcher
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/searcher.ts:31](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/searcher.ts:31](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L31)
|
|
10
10
|
|
|
11
11
|
The abstract base class for all search engines.
|
|
12
12
|
|
|
@@ -119,7 +119,7 @@ Defined in: web-fetcher/dist/index.d.ts:2177
|
|
|
119
119
|
|
|
120
120
|
> `static` **\_isFactory**: `boolean` = `false`
|
|
121
121
|
|
|
122
|
-
Defined in: [web-searcher/src/searcher.ts:33](https://github.com/isdk/web-searcher.js/blob/
|
|
122
|
+
Defined in: [web-searcher/src/searcher.ts:33](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L33)
|
|
123
123
|
|
|
124
124
|
***
|
|
125
125
|
|
|
@@ -127,7 +127,7 @@ Defined in: [web-searcher/src/searcher.ts:33](https://github.com/isdk/web-search
|
|
|
127
127
|
|
|
128
128
|
> `static` `optional` **alias**: `string` \| `string`[]
|
|
129
129
|
|
|
130
|
-
Defined in: [web-searcher/src/searcher.ts:45](https://github.com/isdk/web-searcher.js/blob/
|
|
130
|
+
Defined in: [web-searcher/src/searcher.ts:45](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L45)
|
|
131
131
|
|
|
132
132
|
Engine alias(es). Can be a single string or an array of strings.
|
|
133
133
|
Useful for registering shorthand names (e.g., 'g' for 'Google').
|
|
@@ -138,7 +138,7 @@ Useful for registering shorthand names (e.g., 'g' for 'Google').
|
|
|
138
138
|
|
|
139
139
|
> `static` **createObject**: (`name`, ...`args`) => `WebSearcher`
|
|
140
140
|
|
|
141
|
-
Defined in: [web-searcher/src/searcher.ts:78](https://github.com/isdk/web-searcher.js/blob/
|
|
141
|
+
Defined in: [web-searcher/src/searcher.ts:78](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L78)
|
|
142
142
|
|
|
143
143
|
Creates an instance of the registered search engine.
|
|
144
144
|
|
|
@@ -168,7 +168,7 @@ An instance of the search engine.
|
|
|
168
168
|
|
|
169
169
|
> `static` **forEach**: (`cb`) => `void`
|
|
170
170
|
|
|
171
|
-
Defined in: [web-searcher/src/searcher.ts:85](https://github.com/isdk/web-searcher.js/blob/
|
|
171
|
+
Defined in: [web-searcher/src/searcher.ts:85](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L85)
|
|
172
172
|
|
|
173
173
|
Iterates over all registered engines.
|
|
174
174
|
|
|
@@ -190,7 +190,7 @@ Callback function to invoke for each registered engine.
|
|
|
190
190
|
|
|
191
191
|
> `static` **get**: (`name`) => *typeof* `WebSearcher`
|
|
192
192
|
|
|
193
|
-
Defined in: [web-searcher/src/searcher.ts:69](https://github.com/isdk/web-searcher.js/blob/
|
|
193
|
+
Defined in: [web-searcher/src/searcher.ts:69](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L69)
|
|
194
194
|
|
|
195
195
|
Retrieves a registered search engine class by name.
|
|
196
196
|
|
|
@@ -214,7 +214,7 @@ The search engine class constructor.
|
|
|
214
214
|
|
|
215
215
|
> `static` `optional` **name**: `string`
|
|
216
216
|
|
|
217
|
-
Defined in: [web-searcher/src/searcher.ts:40](https://github.com/isdk/web-searcher.js/blob/
|
|
217
|
+
Defined in: [web-searcher/src/searcher.ts:40](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L40)
|
|
218
218
|
|
|
219
219
|
Custom engine name. If not provided, it is derived from the class name.
|
|
220
220
|
For example, `GoogleSearcher` becomes `Google`.
|
|
@@ -225,7 +225,7 @@ For example, `GoogleSearcher` becomes `Google`.
|
|
|
225
225
|
|
|
226
226
|
> `static` **register**: (`ctor`, `options?`) => `boolean`
|
|
227
227
|
|
|
228
|
-
Defined in: [web-searcher/src/searcher.ts:54](https://github.com/isdk/web-searcher.js/blob/
|
|
228
|
+
Defined in: [web-searcher/src/searcher.ts:54](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L54)
|
|
229
229
|
|
|
230
230
|
Registers a search engine class.
|
|
231
231
|
|
|
@@ -255,7 +255,7 @@ Registration options. If a string is provided, it is used as the registered name
|
|
|
255
255
|
|
|
256
256
|
> `static` **setAliases**: (`ctor`, ...`aliases`) => `void`
|
|
257
257
|
|
|
258
|
-
Defined in: [web-searcher/src/searcher.ts:93](https://github.com/isdk/web-searcher.js/blob/
|
|
258
|
+
Defined in: [web-searcher/src/searcher.ts:93](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L93)
|
|
259
259
|
|
|
260
260
|
Sets aliases for a registered engine.
|
|
261
261
|
|
|
@@ -283,7 +283,7 @@ Aliases to add.
|
|
|
283
283
|
|
|
284
284
|
> `static` **unregister**: (`name?`) => `void`
|
|
285
285
|
|
|
286
|
-
Defined in: [web-searcher/src/searcher.ts:61](https://github.com/isdk/web-searcher.js/blob/
|
|
286
|
+
Defined in: [web-searcher/src/searcher.ts:61](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L61)
|
|
287
287
|
|
|
288
288
|
Unregisters a search engine.
|
|
289
289
|
|
|
@@ -307,7 +307,7 @@ The name or class to unregister.
|
|
|
307
307
|
|
|
308
308
|
> **get** **pagination**(): [`PaginationConfig`](../interfaces/PaginationConfig.md) \| `undefined`
|
|
309
309
|
|
|
310
|
-
Defined in: [web-searcher/src/searcher.ts:151](https://github.com/isdk/web-searcher.js/blob/
|
|
310
|
+
Defined in: [web-searcher/src/searcher.ts:151](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L151)
|
|
311
311
|
|
|
312
312
|
Optional pagination configuration.
|
|
313
313
|
Defines how the searcher navigates to subsequent pages.
|
|
@@ -326,7 +326,7 @@ If undefined, the searcher will only fetch the first page.
|
|
|
326
326
|
|
|
327
327
|
> **get** `abstract` **template**(): `FetcherOptions`
|
|
328
328
|
|
|
329
|
-
Defined in: [web-searcher/src/searcher.ts:143](https://github.com/isdk/web-searcher.js/blob/
|
|
329
|
+
Defined in: [web-searcher/src/searcher.ts:143](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L143)
|
|
330
330
|
|
|
331
331
|
The declarative template for the fetch options.
|
|
332
332
|
|
|
@@ -356,7 +356,7 @@ get template() {
|
|
|
356
356
|
|
|
357
357
|
> `protected` **createContext**(`options`): `FetchContext`
|
|
358
358
|
|
|
359
|
-
Defined in: [web-searcher/src/searcher.ts:155](https://github.com/isdk/web-searcher.js/blob/
|
|
359
|
+
Defined in: [web-searcher/src/searcher.ts:155](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L155)
|
|
360
360
|
|
|
361
361
|
#### Parameters
|
|
362
362
|
|
|
@@ -493,7 +493,7 @@ const { result, outputs } = await session.executeAll([
|
|
|
493
493
|
|
|
494
494
|
> `protected` **formatOptions**(`options`): `Record`\<`string`, `any`\>
|
|
495
495
|
|
|
496
|
-
Defined in: [web-searcher/src/searcher.ts:308](https://github.com/isdk/web-searcher.js/blob/
|
|
496
|
+
Defined in: [web-searcher/src/searcher.ts:308](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L308)
|
|
497
497
|
|
|
498
498
|
Transforms standard options into engine-specific template variables.
|
|
499
499
|
|
|
@@ -561,7 +561,7 @@ A promise resolving to the session state, or undefined if no engine is initializ
|
|
|
561
561
|
|
|
562
562
|
> **search**(`query`, `options`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
563
563
|
|
|
564
|
-
Defined in: [web-searcher/src/searcher.ts:182](https://github.com/isdk/web-searcher.js/blob/
|
|
564
|
+
Defined in: [web-searcher/src/searcher.ts:182](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L182)
|
|
565
565
|
|
|
566
566
|
Executes a search query.
|
|
567
567
|
|
|
@@ -594,7 +594,7 @@ A promise resolving to an array of standardized search results.
|
|
|
594
594
|
|
|
595
595
|
> `protected` **transform**(`outputs`, `context`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
596
596
|
|
|
597
|
-
Defined in: [web-searcher/src/searcher.ts:290](https://github.com/isdk/web-searcher.js/blob/
|
|
597
|
+
Defined in: [web-searcher/src/searcher.ts:290](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L290)
|
|
598
598
|
|
|
599
599
|
Transform and clean the raw extracted results.
|
|
600
600
|
|
|
@@ -627,7 +627,7 @@ A promise resolving to an array of standardized search results.
|
|
|
627
627
|
|
|
628
628
|
> `static` **search**(`engineName`, `query`, `options`): `Promise`\<[`StandardSearchResult`](../interfaces/StandardSearchResult.md)[]\>
|
|
629
629
|
|
|
630
|
-
Defined in: [web-searcher/src/searcher.ts:106](https://github.com/isdk/web-searcher.js/blob/
|
|
630
|
+
Defined in: [web-searcher/src/searcher.ts:106](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L106)
|
|
631
631
|
|
|
632
632
|
Static helper to execute a one-off search.
|
|
633
633
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: CustomTimeRange
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:78](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:78](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L78)
|
|
10
10
|
|
|
11
11
|
## Properties
|
|
12
12
|
|
|
@@ -14,7 +14,7 @@ Defined in: [web-searcher/src/types.ts:78](https://github.com/isdk/web-searcher.
|
|
|
14
14
|
|
|
15
15
|
> **from**: `string` \| `Date`
|
|
16
16
|
|
|
17
|
-
Defined in: [web-searcher/src/types.ts:80](https://github.com/isdk/web-searcher.js/blob/
|
|
17
|
+
Defined in: [web-searcher/src/types.ts:80](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L80)
|
|
18
18
|
|
|
19
19
|
Start date (Date object or string like 'YYYY-MM-DD').
|
|
20
20
|
|
|
@@ -24,6 +24,6 @@ Start date (Date object or string like 'YYYY-MM-DD').
|
|
|
24
24
|
|
|
25
25
|
> `optional` **to**: `string` \| `Date`
|
|
26
26
|
|
|
27
|
-
Defined in: [web-searcher/src/types.ts:82](https://github.com/isdk/web-searcher.js/blob/
|
|
27
|
+
Defined in: [web-searcher/src/types.ts:82](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L82)
|
|
28
28
|
|
|
29
29
|
End date (Date object or string like 'YYYY-MM-DD'). Defaults to current date if omitted.
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: PaginationConfig
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:26](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:26](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L26)
|
|
10
10
|
|
|
11
11
|
Configuration for pagination strategies.
|
|
12
12
|
Defines how the searcher should navigate to the next page of results.
|
|
@@ -17,7 +17,7 @@ Defines how the searcher should navigate to the next page of results.
|
|
|
17
17
|
|
|
18
18
|
> `optional` **increment**: `number`
|
|
19
19
|
|
|
20
|
-
Defined in: [web-searcher/src/types.ts:53](https://github.com/isdk/web-searcher.js/blob/
|
|
20
|
+
Defined in: [web-searcher/src/types.ts:53](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L53)
|
|
21
21
|
|
|
22
22
|
The increment step for each page.
|
|
23
23
|
- If the parameter represents an item offset (like Google's 'start'), this might be 10.
|
|
@@ -35,7 +35,7 @@ The increment step for each page.
|
|
|
35
35
|
|
|
36
36
|
> `optional` **nextButtonSelector**: `string`
|
|
37
37
|
|
|
38
|
-
Defined in: [web-searcher/src/types.ts:59](https://github.com/isdk/web-searcher.js/blob/
|
|
38
|
+
Defined in: [web-searcher/src/types.ts:59](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L59)
|
|
39
39
|
|
|
40
40
|
The CSS selector for the "Next" page button.
|
|
41
41
|
Required if type is 'click-next'.
|
|
@@ -46,7 +46,7 @@ Required if type is 'click-next'.
|
|
|
46
46
|
|
|
47
47
|
> `optional` **paramName**: `string`
|
|
48
48
|
|
|
49
|
-
Defined in: [web-searcher/src/types.ts:39](https://github.com/isdk/web-searcher.js/blob/
|
|
49
|
+
Defined in: [web-searcher/src/types.ts:39](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L39)
|
|
50
50
|
|
|
51
51
|
The name of the URL parameter used for pagination.
|
|
52
52
|
Required if type is 'url-param'.
|
|
@@ -63,7 +63,7 @@ Required if type is 'url-param'.
|
|
|
63
63
|
|
|
64
64
|
> `optional` **startValue**: `number`
|
|
65
65
|
|
|
66
|
-
Defined in: [web-searcher/src/types.ts:45](https://github.com/isdk/web-searcher.js/blob/
|
|
66
|
+
Defined in: [web-searcher/src/types.ts:45](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L45)
|
|
67
67
|
|
|
68
68
|
The starting value for the pagination parameter.
|
|
69
69
|
|
|
@@ -79,7 +79,7 @@ The starting value for the pagination parameter.
|
|
|
79
79
|
|
|
80
80
|
> **type**: `"url-param"` \| `"click-next"`
|
|
81
81
|
|
|
82
|
-
Defined in: [web-searcher/src/types.ts:32](https://github.com/isdk/web-searcher.js/blob/
|
|
82
|
+
Defined in: [web-searcher/src/types.ts:32](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L32)
|
|
83
83
|
|
|
84
84
|
The type of pagination mechanism:
|
|
85
85
|
- 'url-param': Pagination is handled by modifying URL parameters (e.g., `?page=2` or `?start=10`).
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: SearchContext
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:65](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:65](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L65)
|
|
10
10
|
|
|
11
11
|
Context object passed to the transform function.
|
|
12
12
|
|
|
@@ -16,7 +16,7 @@ Context object passed to the transform function.
|
|
|
16
16
|
|
|
17
17
|
> `optional` **limit**: `number`
|
|
18
18
|
|
|
19
|
-
Defined in: [web-searcher/src/types.ts:73](https://github.com/isdk/web-searcher.js/blob/
|
|
19
|
+
Defined in: [web-searcher/src/types.ts:73](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L73)
|
|
20
20
|
|
|
21
21
|
The requested limit of results.
|
|
22
22
|
|
|
@@ -26,7 +26,7 @@ The requested limit of results.
|
|
|
26
26
|
|
|
27
27
|
> **page**: `number`
|
|
28
28
|
|
|
29
|
-
Defined in: [web-searcher/src/types.ts:70](https://github.com/isdk/web-searcher.js/blob/
|
|
29
|
+
Defined in: [web-searcher/src/types.ts:70](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L70)
|
|
30
30
|
|
|
31
31
|
The current page index (0-based).
|
|
32
32
|
|
|
@@ -36,6 +36,6 @@ The current page index (0-based).
|
|
|
36
36
|
|
|
37
37
|
> **query**: `string`
|
|
38
38
|
|
|
39
|
-
Defined in: [web-searcher/src/types.ts:67](https://github.com/isdk/web-searcher.js/blob/
|
|
39
|
+
Defined in: [web-searcher/src/types.ts:67](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L67)
|
|
40
40
|
|
|
41
41
|
The original search query.
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: SearchOptions
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:94](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:94](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L94)
|
|
10
10
|
|
|
11
11
|
Options provided when executing a search.
|
|
12
12
|
|
|
@@ -22,7 +22,7 @@ Any other custom variables to be injected into the template.
|
|
|
22
22
|
|
|
23
23
|
> `optional` **category**: [`SearchCategory`](../type-aliases/SearchCategory.md)
|
|
24
24
|
|
|
25
|
-
Defined in: [web-searcher/src/types.ts:108](https://github.com/isdk/web-searcher.js/blob/
|
|
25
|
+
Defined in: [web-searcher/src/types.ts:108](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L108)
|
|
26
26
|
|
|
27
27
|
The category of results to return.
|
|
28
28
|
Default: 'all' (web search)
|
|
@@ -33,7 +33,7 @@ Default: 'all' (web search)
|
|
|
33
33
|
|
|
34
34
|
> `optional` **language**: `string`
|
|
35
35
|
|
|
36
|
-
Defined in: [web-searcher/src/types.ts:118](https://github.com/isdk/web-searcher.js/blob/
|
|
36
|
+
Defined in: [web-searcher/src/types.ts:118](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L118)
|
|
37
37
|
|
|
38
38
|
Language code (ISO 639-1) for the interface or results (e.g., 'en', 'zh-CN').
|
|
39
39
|
|
|
@@ -43,7 +43,7 @@ Language code (ISO 639-1) for the interface or results (e.g., 'en', 'zh-CN').
|
|
|
43
43
|
|
|
44
44
|
> `optional` **limit**: `number`
|
|
45
45
|
|
|
46
|
-
Defined in: [web-searcher/src/types.ts:96](https://github.com/isdk/web-searcher.js/blob/
|
|
46
|
+
Defined in: [web-searcher/src/types.ts:96](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L96)
|
|
47
47
|
|
|
48
48
|
The maximum number of results to retrieve.
|
|
49
49
|
|
|
@@ -53,7 +53,7 @@ The maximum number of results to retrieve.
|
|
|
53
53
|
|
|
54
54
|
> `optional` **region**: `string`
|
|
55
55
|
|
|
56
|
-
Defined in: [web-searcher/src/types.ts:113](https://github.com/isdk/web-searcher.js/blob/
|
|
56
|
+
Defined in: [web-searcher/src/types.ts:113](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L113)
|
|
57
57
|
|
|
58
58
|
Region code (ISO 3166-1 alpha-2) to bias results (e.g., 'US', 'CN', 'JP').
|
|
59
59
|
|
|
@@ -63,7 +63,7 @@ Region code (ISO 3166-1 alpha-2) to bias results (e.g., 'US', 'CN', 'JP').
|
|
|
63
63
|
|
|
64
64
|
> `optional` **safeSearch**: [`SafeSearchLevel`](../type-aliases/SafeSearchLevel.md)
|
|
65
65
|
|
|
66
|
-
Defined in: [web-searcher/src/types.ts:124](https://github.com/isdk/web-searcher.js/blob/
|
|
66
|
+
Defined in: [web-searcher/src/types.ts:124](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L124)
|
|
67
67
|
|
|
68
68
|
Safe search filtering level.
|
|
69
69
|
Default: engine dependent (usually 'moderate' or 'strict' by default).
|
|
@@ -74,7 +74,7 @@ Default: engine dependent (usually 'moderate' or 'strict' by default).
|
|
|
74
74
|
|
|
75
75
|
> `optional` **timeRange**: [`SearchTimeRange`](../type-aliases/SearchTimeRange.md)
|
|
76
76
|
|
|
77
|
-
Defined in: [web-searcher/src/types.ts:102](https://github.com/isdk/web-searcher.js/blob/
|
|
77
|
+
Defined in: [web-searcher/src/types.ts:102](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L102)
|
|
78
78
|
|
|
79
79
|
Date range for the search results.
|
|
80
80
|
Default: 'all'
|
|
@@ -85,7 +85,7 @@ Default: 'all'
|
|
|
85
85
|
|
|
86
86
|
> `optional` **transform**: (`results`, `context`) => [`StandardSearchResult`](StandardSearchResult.md)[] \| `Promise`\<[`StandardSearchResult`](StandardSearchResult.md)[]\>
|
|
87
87
|
|
|
88
|
-
Defined in: [web-searcher/src/types.ts:130](https://github.com/isdk/web-searcher.js/blob/
|
|
88
|
+
Defined in: [web-searcher/src/types.ts:130](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L130)
|
|
89
89
|
|
|
90
90
|
A custom transform function to filter or modify results at runtime.
|
|
91
91
|
This runs AFTER the engine-level transform.
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
# Interface: StandardSearchResult
|
|
8
8
|
|
|
9
|
-
Defined in: [web-searcher/src/types.ts:5](https://github.com/isdk/web-searcher.js/blob/
|
|
9
|
+
Defined in: [web-searcher/src/types.ts:5](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L5)
|
|
10
10
|
|
|
11
11
|
Interface representing a standardized search result item.
|
|
12
12
|
This ensures consistency across different search engines.
|
|
@@ -23,7 +23,7 @@ Allows for engine-specific extra fields (e.g., rank, author, date).
|
|
|
23
23
|
|
|
24
24
|
> `optional` **image**: `string`
|
|
25
25
|
|
|
26
|
-
Defined in: [web-searcher/src/types.ts:16](https://github.com/isdk/web-searcher.js/blob/
|
|
26
|
+
Defined in: [web-searcher/src/types.ts:16](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L16)
|
|
27
27
|
|
|
28
28
|
An optional image URL associated with the result.
|
|
29
29
|
|
|
@@ -33,7 +33,7 @@ An optional image URL associated with the result.
|
|
|
33
33
|
|
|
34
34
|
> `optional` **snippet**: `string`
|
|
35
35
|
|
|
36
|
-
Defined in: [web-searcher/src/types.ts:13](https://github.com/isdk/web-searcher.js/blob/
|
|
36
|
+
Defined in: [web-searcher/src/types.ts:13](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L13)
|
|
37
37
|
|
|
38
38
|
A brief snippet or description of the result.
|
|
39
39
|
|
|
@@ -43,7 +43,7 @@ A brief snippet or description of the result.
|
|
|
43
43
|
|
|
44
44
|
> **title**: `string`
|
|
45
45
|
|
|
46
|
-
Defined in: [web-searcher/src/types.ts:7](https://github.com/isdk/web-searcher.js/blob/
|
|
46
|
+
Defined in: [web-searcher/src/types.ts:7](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L7)
|
|
47
47
|
|
|
48
48
|
The title of the search result.
|
|
49
49
|
|
|
@@ -53,6 +53,6 @@ The title of the search result.
|
|
|
53
53
|
|
|
54
54
|
> **url**: `string`
|
|
55
55
|
|
|
56
|
-
Defined in: [web-searcher/src/types.ts:10](https://github.com/isdk/web-searcher.js/blob/
|
|
56
|
+
Defined in: [web-searcher/src/types.ts:10](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L10)
|
|
57
57
|
|
|
58
58
|
The URL of the search result.
|
|
@@ -8,4 +8,4 @@
|
|
|
8
8
|
|
|
9
9
|
> **SafeSearchLevel** = `"off"` \| `"moderate"` \| `"strict"`
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/types.ts:89](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/types.ts:89](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L89)
|
|
@@ -8,4 +8,4 @@
|
|
|
8
8
|
|
|
9
9
|
> **SearchCategory** = `"all"` \| `"images"` \| `"videos"` \| `"news"`
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/types.ts:87](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/types.ts:87](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L87)
|
|
@@ -8,4 +8,4 @@
|
|
|
8
8
|
|
|
9
9
|
> **SearchTimeRange** = [`SearchTimeRangePreset`](SearchTimeRangePreset.md) \| [`CustomTimeRange`](../interfaces/CustomTimeRange.md)
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/types.ts:85](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/types.ts:85](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L85)
|
|
@@ -8,4 +8,4 @@
|
|
|
8
8
|
|
|
9
9
|
> **SearchTimeRangePreset** = `"all"` \| `"day"` \| `"week"` \| `"month"` \| `"year"`
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/types.ts:76](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/types.ts:76](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/types.ts#L76)
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **SearcherConstructor** = (`options?`) => [`WebSearcher`](../classes/WebSearcher.md)
|
|
10
10
|
|
|
11
|
-
Defined in: [web-searcher/src/searcher.ts:10](https://github.com/isdk/web-searcher.js/blob/
|
|
11
|
+
Defined in: [web-searcher/src/searcher.ts:10](https://github.com/isdk/web-searcher.js/blob/6ce291d521b8526526b386fab6dda19d36d0bece/src/searcher.ts#L10)
|
|
12
12
|
|
|
13
13
|
Constructor definition for Searcher subclasses.
|
|
14
14
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@isdk/web-searcher",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.2",
|
|
4
4
|
"description": "A high-level framework for building search engine scrapers, supporting multi-page navigation, session persistence, and result standardization.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Riceball LEE <snowyu.lee@gmail.com>",
|
|
@@ -54,11 +54,17 @@
|
|
|
54
54
|
],
|
|
55
55
|
"keywords": [
|
|
56
56
|
"web",
|
|
57
|
-
"
|
|
57
|
+
"search",
|
|
58
|
+
"searcher",
|
|
59
|
+
"scraper",
|
|
58
60
|
"crawler",
|
|
59
61
|
"spider",
|
|
60
|
-
"
|
|
61
|
-
"
|
|
62
|
+
"google",
|
|
63
|
+
"automation",
|
|
64
|
+
"headless",
|
|
65
|
+
"pagination",
|
|
66
|
+
"data-mining",
|
|
67
|
+
"fetcher"
|
|
62
68
|
],
|
|
63
69
|
"maintainers": [
|
|
64
70
|
"Riceball LEE <snowyu.lee@gmail.com>"
|
|
@@ -66,6 +72,12 @@
|
|
|
66
72
|
"engines": {
|
|
67
73
|
"node": ">=20.11.1"
|
|
68
74
|
},
|
|
75
|
+
"bugs": {
|
|
76
|
+
"url": "https://github.com/isdk/web-searcher.js/issues"
|
|
77
|
+
},
|
|
78
|
+
"publishConfig": {
|
|
79
|
+
"access": "public"
|
|
80
|
+
},
|
|
69
81
|
"types": "./dist/index.d.ts",
|
|
70
82
|
"scripts": {
|
|
71
83
|
"build-fast": "tsup",
|