felo-ai 0.2.6 → 0.2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -0
- package/README.en.md +6 -6
- package/README.md +23 -17
- package/{felo-web-extract → felo-web-fetch}/README.md +15 -15
- package/{felo-web-extract → felo-web-fetch}/SKILL.md +27 -27
- package/{felo-web-extract/scripts/run_web_extract.mjs → felo-web-fetch/scripts/run_web_fetch.mjs} +5 -5
- package/felo-youtube-subtitling/SKILL.md +1 -1
- package/package.json +3 -3
- package/src/cli.js +6 -6
- package/src/{webExtract.js → webFetch.js} +6 -6
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [0.2.7] - 2025-03-06
|
|
9
|
+
|
|
10
|
+
### Breaking Changes
|
|
11
|
+
|
|
12
|
+
- **CLI 命令与技能重命名:web extract → web fetch**
|
|
13
|
+
- 命令 `felo web-extract` 已更名为 `felo web-fetch`,请更新脚本与文档。
|
|
14
|
+
- 独立脚本路径由 `felo-web-extract/scripts/run_web_extract.mjs` 改为 `felo-web-fetch/scripts/run_web_fetch.mjs`。
|
|
15
|
+
- 技能/目录名由 `felo-web-extract` 改为 `felo-web-fetch`;触发词示例:`/felo-web-fetch`、`use felo web fetch`。
|
|
16
|
+
- 后端 API 路径未变(仍为 `POST /v2/web/extract`),仅产品对外名称改为「Web Fetch」。
|
|
17
|
+
|
|
18
|
+
### Changed
|
|
19
|
+
|
|
20
|
+
- 产品名称统一为「Web Fetch」:README、SKILL、package 描述与关键词已同步更新。
|
|
21
|
+
- `felo-youtube-subtitling` 技能中「网页内容」相关引用已更新为 `felo-web-fetch`。
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## [0.2.6] - (previous)
|
|
26
|
+
|
|
27
|
+
Earlier releases: search, slides, web extract, youtube-subtitling features.
|
|
28
|
+
|
|
29
|
+
[0.2.7]: https://github.com/Felo-Inc/felo-skills/compare/v0.2.6...v0.2.7
|
|
30
|
+
[0.2.6]: https://github.com/Felo-Inc/felo-skills/releases/tag/v0.2.6
|
package/README.en.md
CHANGED
|
@@ -56,7 +56,7 @@ Get your API key at [felo.ai](https://felo.ai) (Settings → API Keys).
|
|
|
56
56
|
|---------|-------------|
|
|
57
57
|
| `felo search "<query>"` | Real-time search |
|
|
58
58
|
| `felo slides "<prompt>"` | Generate PPT |
|
|
59
|
-
| `felo web-
|
|
59
|
+
| `felo web-fetch --url <url>` | Fetch webpage content (markdown/text/html) |
|
|
60
60
|
| `felo youtube-subtitling -v <url-or-id>` | Fetch YouTube video subtitles |
|
|
61
61
|
| `felo config set FELO_API_KEY <key>` | Save API key |
|
|
62
62
|
| `felo config get/list/path/unset` | View / list / path / remove config |
|
|
@@ -81,12 +81,12 @@ felo slides "Q4 2024 business review, 10 pages" --poll-timeout 300
|
|
|
81
81
|
npx felo-ai slides "Tokyo travel guide, 5 slides"
|
|
82
82
|
```
|
|
83
83
|
|
|
84
|
-
**Web
|
|
84
|
+
**Web fetch**
|
|
85
85
|
|
|
86
86
|
```bash
|
|
87
|
-
felo web-
|
|
88
|
-
felo web-
|
|
89
|
-
node felo-web-
|
|
87
|
+
felo web-fetch --url "https://example.com"
|
|
88
|
+
felo web-fetch --url "https://example.com" --format text --readability
|
|
89
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --format markdown
|
|
90
90
|
```
|
|
91
91
|
|
|
92
92
|
**YouTube subtitling**
|
|
@@ -111,7 +111,7 @@ After setting `FELO_API_KEY`, ask Claude things like “What’s the weather in
|
|
|
111
111
|
|
|
112
112
|
**Slides (PPT)** — `npx @claude/skills add felo-slides`, then `/felo-slides your topic`. Same `FELO_API_KEY`. [Details →](./felo-slides/README.md)
|
|
113
113
|
|
|
114
|
-
**Web
|
|
114
|
+
**Web Fetch** — `felo web-fetch --url "https://example.com"` or run `node felo-web-fetch/scripts/run_web_fetch.mjs` from repo. [Details →](./felo-web-fetch/README.md)
|
|
115
115
|
|
|
116
116
|
**YouTube Subtitling** — `felo youtube-subtitling -v "URL_or_VIDEO_ID"` or run `node felo-youtube-subtitling/scripts/run_youtube_subtitling.mjs` from repo. [Details →](./felo-youtube-subtitling/README.md)
|
|
117
117
|
|
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Ask anything. Get current answers. Generate slides from a prompt.**
|
|
4
4
|
|
|
5
|
-
[npm package: **felo-ai**](https://www.npmjs.com/package/felo-ai) — Real-time search, PPT generation, web
|
|
5
|
+
[npm package: **felo-ai**](https://www.npmjs.com/package/felo-ai) — Real-time search, PPT generation, web fetch, and YouTube subtitles from the terminal. Also works as Claude Code skills. Supports Chinese, English, Japanese, and Korean.
|
|
6
6
|
|
|
7
7
|
[](https://www.npmjs.com/package/felo-ai) []()
|
|
8
8
|
|
|
@@ -51,7 +51,7 @@ Get your API key from [felo.ai](https://felo.ai) (Settings → API Keys). Enviro
|
|
|
51
51
|
| ------------------------------------ | ----------------------------------------------------- |
|
|
52
52
|
| `felo search "<query>"` | Search for current info (weather, news, prices, etc.) |
|
|
53
53
|
| `felo slides "<prompt>"` | Generate PPT; returns link when done |
|
|
54
|
-
| `felo web-
|
|
54
|
+
| `felo web-fetch --url <url>` | Fetch webpage content (markdown/text/html) |
|
|
55
55
|
| `felo youtube-subtitling -v <url-or-id>` | Fetch YouTube video subtitles by video URL or ID |
|
|
56
56
|
| `felo config set FELO_API_KEY <key>` | Save API key to config |
|
|
57
57
|
| `felo config get FELO_API_KEY` | Print stored key |
|
|
@@ -79,29 +79,29 @@ felo slides "Q4 2024 business review, 10 pages" --poll-timeout 300
|
|
|
79
79
|
npx felo-ai slides "Tokyo travel guide, 5 slides"
|
|
80
80
|
```
|
|
81
81
|
|
|
82
|
-
**Web
|
|
82
|
+
**Web fetch** (after `npm install -g felo-ai`)
|
|
83
83
|
|
|
84
84
|
```bash
|
|
85
85
|
# Packaged CLI
|
|
86
|
-
felo web-
|
|
87
|
-
felo web-
|
|
88
|
-
felo web-
|
|
89
|
-
felo web-
|
|
90
|
-
npx felo-ai web-
|
|
86
|
+
felo web-fetch --url "https://example.com"
|
|
87
|
+
felo web-fetch --url "https://example.com/article" --format markdown --readability
|
|
88
|
+
felo web-fetch --url "https://example.com" --target-selector "article.main" --format text
|
|
89
|
+
felo web-fetch --url "https://example.com" -j
|
|
90
|
+
npx felo-ai web-fetch --url "https://example.com" --format markdown
|
|
91
91
|
|
|
92
92
|
# From repo: run script directly (no install)
|
|
93
|
-
node felo-web-
|
|
94
|
-
node felo-web-
|
|
93
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --format markdown
|
|
94
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --readability -f text
|
|
95
95
|
```
|
|
96
96
|
|
|
97
97
|
**How to pass parameters**
|
|
98
98
|
|
|
99
99
|
| Parameter | CLI option | Example | Description |
|
|
100
100
|
|-----------|------------|---------|--------------|
|
|
101
|
-
| URL (required) | `-u`, `--url` | `--url "https://example.com"` | Page to
|
|
101
|
+
| URL (required) | `-u`, `--url` | `--url "https://example.com"` | Page to fetch |
|
|
102
102
|
| Output format | `-f`, `--format` | `--format text` or `-f markdown` | `html`, `text`, or `markdown` (default: markdown) |
|
|
103
|
-
| Target element | `--target-selector` | `--target-selector "article.main"` | CSS selector; only this element is
|
|
104
|
-
| Wait for element | `--wait-for-selector` | `--wait-for-selector ".content"` | Wait for selector before
|
|
103
|
+
| Target element | `--target-selector` | `--target-selector "article.main"` | CSS selector; only this element is fetched |
|
|
104
|
+
| Wait for element | `--wait-for-selector` | `--wait-for-selector ".content"` | Wait for selector before fetching (e.g. dynamic pages) |
|
|
105
105
|
| Readability | `--readability` | `--readability` | Main article content only (no nav/ads) |
|
|
106
106
|
| Crawl mode | `--crawl-mode` | `--crawl-mode fine` | `fast` (default) or `fine` |
|
|
107
107
|
| Timeout (seconds) | `-t`, `--timeout` | `--timeout 120` or `-t 90` | Request timeout (default: 60) |
|
|
@@ -110,8 +110,8 @@ node felo-web-extract/scripts/run_web_extract.mjs --url "https://example.com" --
|
|
|
110
110
|
Examples with multiple options:
|
|
111
111
|
|
|
112
112
|
```bash
|
|
113
|
-
felo web-
|
|
114
|
-
felo web-
|
|
113
|
+
felo web-fetch -u "https://example.com" -f text --readability -t 90
|
|
114
|
+
felo web-fetch --url "https://example.com" --target-selector "#main" --wait-for-selector ".loaded" --format markdown --json
|
|
115
115
|
```
|
|
116
116
|
|
|
117
117
|
Same `FELO_API_KEY` as search/slides.
|
|
@@ -138,7 +138,7 @@ Options: `-v/--video-code` (required: **YouTube video URL** or video ID), `-l/--
|
|
|
138
138
|
- **Request timeout?** Use `felo search "query" --timeout 120` (default 60 seconds). 5xx errors are retried automatically with backoff.
|
|
139
139
|
- **Slides taking long?** Use `felo slides "topic" --poll-timeout 300` (default 1200s) to limit wait.
|
|
140
140
|
- **Where is config stored?** Run `felo config path` to see the file (e.g. `~/.felo/config.json`).
|
|
141
|
-
- **Web
|
|
141
|
+
- **Web fetch after install?** Use `felo web-fetch --url "<page url>"`. Other params: `--format markdown|text|html`, `--readability`, `--target-selector "selector"`, `--wait-for-selector "selector"`, `--crawl-mode fast|fine`, `--timeout 120`, `--json`. See the "How to pass parameters" table above. Same API key as other commands.
|
|
142
142
|
- **YouTube subtitles?** Use `felo youtube-subtitling -v "<url or video_id>"` (full YouTube link or 11-char ID). Optional: `-l/--language`, `--with-time`, `-j/--json`. See [felo-youtube-subtitling](./felo-youtube-subtitling/README.md).
|
|
143
143
|
|
|
144
144
|
---
|
|
@@ -181,7 +181,7 @@ Ask Claude: "What's the weather in Tokyo today?"
|
|
|
181
181
|
|
|
182
182
|
**Felo Slides (PPT):** In terminal run `felo slides "your topic"`. In Claude Code install with `npx @claude/skills add felo-slides`, then use `/felo-slides your topic`. See [felo-slides](./felo-slides/README.md).
|
|
183
183
|
|
|
184
|
-
**Felo Web
|
|
184
|
+
**Felo Web Fetch:** In terminal run `felo web-fetch --url "https://example.com"` (see [felo-web-fetch](./felo-web-fetch/README.md)). In Claude Code you can install the skill and use it to fetch webpage content from a URL.
|
|
185
185
|
|
|
186
186
|
**Felo YouTube Subtitling:** In terminal run `felo youtube-subtitling -v "URL_or_VIDEO_ID"` (see [felo-youtube-subtitling](./felo-youtube-subtitling/README.md)). Fetches subtitles/captions; accepts full YouTube link or video ID.
|
|
187
187
|
|
|
@@ -399,6 +399,12 @@ Run CLI tests: `npm test`
|
|
|
399
399
|
|
|
400
400
|
---
|
|
401
401
|
|
|
402
|
+
## Version history
|
|
403
|
+
|
|
404
|
+
See [CHANGELOG.md](./CHANGELOG.md) for release notes (e.g. breaking changes such as `web-extract` → `web-fetch` in v0.2.7).
|
|
405
|
+
|
|
406
|
+
---
|
|
407
|
+
|
|
402
408
|
## License
|
|
403
409
|
|
|
404
410
|
MIT — see [LICENSE](./felo-search/LICENSE) in the repo for details.
|
|
@@ -1,10 +1,10 @@
|
|
|
1
|
-
# Felo Web
|
|
1
|
+
# Felo Web Fetch Skill
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Fetch webpage content from a URL using the [Felo Web Extract API](https://openapi.felo.ai/docs/api-reference/v2/web-extract.html).
|
|
4
4
|
|
|
5
5
|
## Features
|
|
6
6
|
|
|
7
|
-
-
|
|
7
|
+
- Fetch content from any URL as **html**, **text**, or **markdown** (`--format`)
|
|
8
8
|
- **Target one element** with a CSS selector (`--target-selector`, e.g. `article.main`, `#content`)
|
|
9
9
|
- Optional **readability** mode for main article content only
|
|
10
10
|
- Crawl modes: `fast` (default) or `fine`
|
|
@@ -26,28 +26,28 @@ export FELO_API_KEY="your-api-key-here"
|
|
|
26
26
|
$env:FELO_API_KEY="your-api-key-here"
|
|
27
27
|
```
|
|
28
28
|
|
|
29
|
-
### 2) Run
|
|
29
|
+
### 2) Run fetch
|
|
30
30
|
|
|
31
31
|
```bash
|
|
32
|
-
#
|
|
33
|
-
node felo-web-
|
|
32
|
+
# Fetch as Markdown (default)
|
|
33
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com/article"
|
|
34
34
|
|
|
35
35
|
# With readability for clean article text
|
|
36
|
-
node felo-web-
|
|
36
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --readability --format markdown
|
|
37
37
|
|
|
38
38
|
# Full JSON response
|
|
39
|
-
node felo-web-
|
|
39
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --json
|
|
40
40
|
|
|
41
41
|
# Only a specific element (CSS selector) and output format
|
|
42
|
-
node felo-web-
|
|
42
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --target-selector "article.main" --format markdown
|
|
43
43
|
```
|
|
44
44
|
|
|
45
|
-
## Using the packaged CLI (`felo web-
|
|
45
|
+
## Using the packaged CLI (`felo web-fetch`)
|
|
46
46
|
|
|
47
47
|
After `npm install -g felo-ai`, you can run:
|
|
48
48
|
|
|
49
49
|
```bash
|
|
50
|
-
felo web-
|
|
50
|
+
felo web-fetch --url "https://example.com"
|
|
51
51
|
```
|
|
52
52
|
|
|
53
53
|
**All parameters (how to pass)**
|
|
@@ -66,13 +66,13 @@ felo web-extract --url "https://example.com"
|
|
|
66
66
|
**Examples with multiple options**
|
|
67
67
|
|
|
68
68
|
```bash
|
|
69
|
-
felo web-
|
|
70
|
-
felo web-
|
|
71
|
-
felo web-
|
|
69
|
+
felo web-fetch -u "https://example.com" -f text --readability
|
|
70
|
+
felo web-fetch --url "https://example.com" --target-selector "#content" --format markdown --timeout 90
|
|
71
|
+
felo web-fetch --url "https://example.com" --wait-for-selector "main" --readability -j
|
|
72
72
|
```
|
|
73
73
|
|
|
74
74
|
## When to use (Agent)
|
|
75
75
|
|
|
76
|
-
Trigger keywords:
|
|
76
|
+
Trigger keywords: fetch webpage, scrape URL, fetch page content, url to markdown, `/felo-web-fetch`.
|
|
77
77
|
|
|
78
78
|
See [SKILL.md](SKILL.md) for full agent instructions and API parameters.
|
|
@@ -1,23 +1,23 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: felo-web-
|
|
3
|
-
description: "
|
|
2
|
+
name: felo-web-fetch
|
|
3
|
+
description: "Fetch web page content from a URL using Felo Web Extract API. Use when users ask to scrape/capture/fetch webpage content, get article text from URL, convert page to markdown/text, or when explicit commands like /felo-web-fetch are used. Supports html, text, markdown output and readability mode."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
# Felo Web
|
|
6
|
+
# Felo Web Fetch Skill
|
|
7
7
|
|
|
8
8
|
## When to Use
|
|
9
9
|
|
|
10
10
|
Trigger this skill when the user wants to:
|
|
11
11
|
|
|
12
|
-
-
|
|
12
|
+
- Fetch or scrape content from a webpage URL
|
|
13
13
|
- Get article/main text from a link
|
|
14
14
|
- Convert a webpage to Markdown or plain text
|
|
15
15
|
- Capture readable content from a URL for summarization or processing
|
|
16
16
|
|
|
17
17
|
Trigger keywords (examples):
|
|
18
18
|
|
|
19
|
-
-
|
|
20
|
-
- Explicit: `/felo-web-
|
|
19
|
+
- fetch webpage, scrape URL, fetch page content, web fetch, url to markdown
|
|
20
|
+
- Explicit: `/felo-web-fetch`, "use felo web fetch"
|
|
21
21
|
- Same intent in other languages (e.g. 网页抓取, 提取网页内容) also triggers this skill
|
|
22
22
|
|
|
23
23
|
Do NOT use for:
|
|
@@ -55,13 +55,13 @@ $env:FELO_API_KEY="your-api-key-here"
|
|
|
55
55
|
**Script** (from repo):
|
|
56
56
|
|
|
57
57
|
```bash
|
|
58
|
-
node felo-web-
|
|
58
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com/article" [options]
|
|
59
59
|
```
|
|
60
60
|
|
|
61
61
|
**Packaged CLI** (after `npm install -g felo-ai`): same options, with short forms allowed:
|
|
62
62
|
|
|
63
63
|
```bash
|
|
64
|
-
felo web-
|
|
64
|
+
felo web-fetch -u "https://example.com" [options]
|
|
65
65
|
# Short forms: -u (url), -f (format), -t (timeout, seconds), -j (json)
|
|
66
66
|
```
|
|
67
67
|
|
|
@@ -69,10 +69,10 @@ Options:
|
|
|
69
69
|
|
|
70
70
|
| Option | Default | Description |
|
|
71
71
|
|--------|---------|-------------|
|
|
72
|
-
| `--url` | (required) | Webpage URL to
|
|
72
|
+
| `--url` | (required) | Webpage URL to fetch |
|
|
73
73
|
| `--format` | markdown | Output format: `html`, `text`, `markdown` |
|
|
74
|
-
| `--target-selector` | - | CSS selector:
|
|
75
|
-
| `--wait-for-selector` | - | Wait for this selector before
|
|
74
|
+
| `--target-selector` | - | CSS selector: fetch only this element (e.g. `article.main`, `#content`) |
|
|
75
|
+
| `--wait-for-selector` | - | Wait for this selector before fetching (e.g. dynamic content) |
|
|
76
76
|
| `--readability` | false | Enable readability processing (main content only) |
|
|
77
77
|
| `--crawl-mode` | fast | `fast` or `fine` |
|
|
78
78
|
| `--timeout` | 60000 (script) / 60 (CLI) | Request timeout: script uses **milliseconds**, CLI uses **seconds** (e.g. `-t 90`) |
|
|
@@ -82,35 +82,35 @@ Options:
|
|
|
82
82
|
|
|
83
83
|
When the user wants a **specific part** of the page or a **specific output format**, phrase the command like this:
|
|
84
84
|
|
|
85
|
-
- **Output format**: "
|
|
86
|
-
- **Target one element**: "Only the **main article**" / "Just the **content inside** `#main`" / "
|
|
85
|
+
- **Output format**: "Fetch as **text**" / "Get **markdown**" / "Return **html**" → use `--format text`, `--format markdown`, or `--format html`.
|
|
86
|
+
- **Target one element**: "Only the **main article**" / "Just the **content inside** `#main`" / "Fetch only **article.main-content**" → use `--target-selector "article.main"` or the selector they give (e.g. `#main`, `.main-content`, `article .post`).
|
|
87
87
|
|
|
88
88
|
Examples of user intents and equivalent commands:
|
|
89
89
|
|
|
90
90
|
| User intent | Command |
|
|
91
91
|
|-------------|---------|
|
|
92
|
-
| "
|
|
92
|
+
| "Fetch this page as plain text" | `--url "..." --format text` |
|
|
93
93
|
| "Get only the main content area" | `--url "..." --target-selector "main"` or `article` |
|
|
94
|
-
| "
|
|
94
|
+
| "Fetch the div with id=content as markdown" | `--url "..." --target-selector "#content" --format markdown` |
|
|
95
95
|
| "Just the article body, as HTML" | `--url "..." --target-selector "article .body" --format html` |
|
|
96
96
|
|
|
97
97
|
Examples:
|
|
98
98
|
|
|
99
99
|
```bash
|
|
100
|
-
# Basic:
|
|
101
|
-
node felo-web-
|
|
100
|
+
# Basic: fetch as Markdown
|
|
101
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com"
|
|
102
102
|
|
|
103
103
|
# Article-style with readability
|
|
104
|
-
node felo-web-
|
|
104
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com/article" --readability --format markdown
|
|
105
105
|
|
|
106
106
|
# Raw HTML
|
|
107
|
-
node felo-web-
|
|
107
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --format html --json
|
|
108
108
|
|
|
109
109
|
# Only the element matching a CSS selector (e.g. main article)
|
|
110
|
-
node felo-web-
|
|
110
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --target-selector "article.main" --format markdown
|
|
111
111
|
|
|
112
112
|
# Specific output format + target selector
|
|
113
|
-
node felo-web-
|
|
113
|
+
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --target-selector "#content" --format text
|
|
114
114
|
```
|
|
115
115
|
|
|
116
116
|
### Option B: Call API with curl
|
|
@@ -132,14 +132,14 @@ curl -X POST "https://openapi.felo.ai/v2/web/extract" \
|
|
|
132
132
|
|
|
133
133
|
| Parameter | Type | Required | Default | Description |
|
|
134
134
|
|-----------|------|----------|---------|-------------|
|
|
135
|
-
| url | string | Yes | - | Webpage URL to
|
|
135
|
+
| url | string | Yes | - | Webpage URL to fetch |
|
|
136
136
|
| crawl_mode | string | No | fast | `fast` or `fine` |
|
|
137
137
|
| output_format | string | No | html | `html`, `text`, `markdown` |
|
|
138
138
|
| with_readability | boolean | No | - | Use readability (main content) |
|
|
139
139
|
| with_links_summary | boolean | No | - | Include links summary |
|
|
140
140
|
| with_images_summary | boolean | No | - | Include images summary |
|
|
141
141
|
| target_selector | string | No | - | CSS selector for target element |
|
|
142
|
-
| wait_for_selector | string | No | - | Wait for selector before
|
|
142
|
+
| wait_for_selector | string | No | - | Wait for selector before fetch |
|
|
143
143
|
| timeout | integer | No | - | Timeout in milliseconds |
|
|
144
144
|
| with_cache | boolean | No | true | Use cache |
|
|
145
145
|
|
|
@@ -157,7 +157,7 @@ Success (200):
|
|
|
157
157
|
}
|
|
158
158
|
```
|
|
159
159
|
|
|
160
|
-
|
|
160
|
+
Fetched content is in `data.content`; structure depends on `output_format`.
|
|
161
161
|
|
|
162
162
|
### Error codes
|
|
163
163
|
|
|
@@ -165,13 +165,13 @@ Extracted content is in `data.content`; structure depends on `output_format`.
|
|
|
165
165
|
|------|------|-------------|
|
|
166
166
|
| 400 | - | Parameter validation failed |
|
|
167
167
|
| 401 | INVALID_API_KEY | API key invalid or revoked |
|
|
168
|
-
| 500/502 | WEB_EXTRACT_FAILED |
|
|
168
|
+
| 500/502 | WEB_EXTRACT_FAILED | Fetch failed (server or page error) |
|
|
169
169
|
|
|
170
170
|
## Output Format
|
|
171
171
|
|
|
172
172
|
On success (script without `--json`):
|
|
173
173
|
|
|
174
|
-
- Print the
|
|
174
|
+
- Print the fetched content only (for direct use or piping).
|
|
175
175
|
|
|
176
176
|
With `--json`:
|
|
177
177
|
|
|
@@ -180,7 +180,7 @@ With `--json`:
|
|
|
180
180
|
Error response to user:
|
|
181
181
|
|
|
182
182
|
```markdown
|
|
183
|
-
## Web
|
|
183
|
+
## Web Fetch Failed
|
|
184
184
|
|
|
185
185
|
- Error: <code or message>
|
|
186
186
|
- URL: <requested url>
|
package/{felo-web-extract/scripts/run_web_extract.mjs → felo-web-fetch/scripts/run_web_fetch.mjs}
RENAMED
|
@@ -28,13 +28,13 @@ function usage() {
|
|
|
28
28
|
console.error(
|
|
29
29
|
[
|
|
30
30
|
'Usage:',
|
|
31
|
-
' node felo-web-
|
|
31
|
+
' node felo-web-fetch/scripts/run_web_fetch.mjs --url <url> [options]',
|
|
32
32
|
'',
|
|
33
33
|
'Options:',
|
|
34
|
-
' --url <url> Page URL to
|
|
34
|
+
' --url <url> Page URL to fetch (required)',
|
|
35
35
|
' --format <format> Output format: html, text, markdown (default: markdown)',
|
|
36
36
|
' --target-selector <s> CSS selector for target element only',
|
|
37
|
-
' --wait-for-selector <s> Wait for selector before
|
|
37
|
+
' --wait-for-selector <s> Wait for selector before fetch',
|
|
38
38
|
' --readability Enable readability (main content only)',
|
|
39
39
|
' --crawl-mode <mode> fast or fine (default: fast)',
|
|
40
40
|
' --timeout <ms> Request timeout in ms (default: 60000)',
|
|
@@ -210,7 +210,7 @@ async function main() {
|
|
|
210
210
|
if (isEmpty) {
|
|
211
211
|
stopSpinner(spinnerId);
|
|
212
212
|
process.stderr.write(
|
|
213
|
-
`No content
|
|
213
|
+
`No content fetched from ${args.url}. The page may be empty, blocked, or the selector did not match.\n`
|
|
214
214
|
);
|
|
215
215
|
process.exit(1);
|
|
216
216
|
}
|
|
@@ -226,7 +226,7 @@ main().catch((err) => {
|
|
|
226
226
|
const i = argv.findIndex((a) => a === '--url' || a === '-u');
|
|
227
227
|
if (i >= 0 && argv[i + 1]) url = argv[i + 1];
|
|
228
228
|
process.stderr.write(
|
|
229
|
-
`Web
|
|
229
|
+
`Web fetch failed${url ? ` for ${url}` : ''}: ${err?.message || err}\n`
|
|
230
230
|
);
|
|
231
231
|
process.exit(1);
|
|
232
232
|
});
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "felo-ai",
|
|
3
|
-
"version": "0.2.
|
|
4
|
-
"description": "Felo AI CLI - real-time search, PPT generation, web
|
|
3
|
+
"version": "0.2.7",
|
|
4
|
+
"description": "Felo AI CLI - real-time search, PPT generation, web fetch, and YouTube subtitles from the terminal",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/cli.js",
|
|
7
7
|
"bin": {
|
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
"felo-ai",
|
|
16
16
|
"search",
|
|
17
17
|
"slides",
|
|
18
|
-
"web-
|
|
18
|
+
"web-fetch",
|
|
19
19
|
"youtube-subtitles",
|
|
20
20
|
"cli",
|
|
21
21
|
"ai"
|
package/src/cli.js
CHANGED
|
@@ -4,7 +4,7 @@ import { createRequire } from "module";
|
|
|
4
4
|
import { Command } from "commander";
|
|
5
5
|
import { search } from "./search.js";
|
|
6
6
|
import { slides } from "./slides.js";
|
|
7
|
-
import {
|
|
7
|
+
import { webFetch } from "./webFetch.js";
|
|
8
8
|
import { youtubeSubtitling } from "./youtubeSubtitling.js";
|
|
9
9
|
import * as config from "./config.js";
|
|
10
10
|
|
|
@@ -177,9 +177,9 @@ configCmd
|
|
|
177
177
|
});
|
|
178
178
|
|
|
179
179
|
program
|
|
180
|
-
.command("web-
|
|
181
|
-
.description("
|
|
182
|
-
.requiredOption("-u, --url <url>", "page URL to
|
|
180
|
+
.command("web-fetch")
|
|
181
|
+
.description("Fetch webpage content from a URL (markdown, text, or html)")
|
|
182
|
+
.requiredOption("-u, --url <url>", "page URL to fetch")
|
|
183
183
|
.option(
|
|
184
184
|
"-f, --format <format>",
|
|
185
185
|
"output format: html, text, markdown",
|
|
@@ -191,7 +191,7 @@ program
|
|
|
191
191
|
)
|
|
192
192
|
.option(
|
|
193
193
|
"--wait-for-selector <selector>",
|
|
194
|
-
"wait for selector before
|
|
194
|
+
"wait for selector before fetching"
|
|
195
195
|
)
|
|
196
196
|
.option("--readability", "use readability (main content only)")
|
|
197
197
|
.option("--crawl-mode <mode>", "crawl mode: fast or fine", "fast")
|
|
@@ -199,7 +199,7 @@ program
|
|
|
199
199
|
.option("-j, --json", "output full API response as JSON")
|
|
200
200
|
.action(async (opts) => {
|
|
201
201
|
const timeoutMs = parseInt(opts.timeout, 10) * 1000;
|
|
202
|
-
const code = await
|
|
202
|
+
const code = await webFetch({
|
|
203
203
|
url: opts.url,
|
|
204
204
|
format: opts.format,
|
|
205
205
|
targetSelector: opts.targetSelector,
|
|
@@ -49,7 +49,7 @@ function stringifyContent(content) {
|
|
|
49
49
|
return String(content);
|
|
50
50
|
}
|
|
51
51
|
|
|
52
|
-
async function
|
|
52
|
+
async function fetchContent(apiBase, apiKey, body, timeoutMs) {
|
|
53
53
|
const controller = new AbortController();
|
|
54
54
|
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
|
55
55
|
try {
|
|
@@ -86,10 +86,10 @@ async function fetchExtract(apiBase, apiKey, body, timeoutMs) {
|
|
|
86
86
|
}
|
|
87
87
|
|
|
88
88
|
/**
|
|
89
|
-
* Run web
|
|
89
|
+
* Run web fetch and print result. Returns exit code (0 or 1).
|
|
90
90
|
* @param {Object} opts - { url, format, targetSelector, waitForSelector, readability, timeoutMs, json }
|
|
91
91
|
*/
|
|
92
|
-
export async function
|
|
92
|
+
export async function webFetch(opts) {
|
|
93
93
|
const apiKey = await getApiKey();
|
|
94
94
|
if (!apiKey) {
|
|
95
95
|
console.error(NO_KEY_MESSAGE.trim());
|
|
@@ -119,7 +119,7 @@ export async function webExtract(opts) {
|
|
|
119
119
|
if (opts.waitForSelector) body.wait_for_selector = opts.waitForSelector;
|
|
120
120
|
|
|
121
121
|
try {
|
|
122
|
-
const payload = await
|
|
122
|
+
const payload = await fetchContent(apiBase, apiKey, body, timeoutMs);
|
|
123
123
|
const content = payload?.data?.content;
|
|
124
124
|
|
|
125
125
|
if (opts.json) {
|
|
@@ -131,7 +131,7 @@ export async function webExtract(opts) {
|
|
|
131
131
|
const isEmpty = out == null || String(out).trim() === '';
|
|
132
132
|
if (isEmpty) {
|
|
133
133
|
process.stderr.write(
|
|
134
|
-
`No content
|
|
134
|
+
`No content fetched from ${opts.url}. The page may be empty, blocked, or the selector did not match.\n`
|
|
135
135
|
);
|
|
136
136
|
return 1;
|
|
137
137
|
}
|
|
@@ -139,7 +139,7 @@ export async function webExtract(opts) {
|
|
|
139
139
|
return 0;
|
|
140
140
|
} catch (err) {
|
|
141
141
|
process.stderr.write(
|
|
142
|
-
`Web
|
|
142
|
+
`Web fetch failed for ${opts.url}: ${err?.message || err}\n`
|
|
143
143
|
);
|
|
144
144
|
return 1;
|
|
145
145
|
} finally {
|