@just-every/mcp-read-website-fast 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +44 -3
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,12 +1,27 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @just-every/mcp-read-website-fast
|
|
2
|
+
|
|
3
|
+
Fast, token-efficient web content extraction for AI agents - converts websites to clean Markdown.
|
|
4
|
+
|
|
5
|
+
[](https://www.npmjs.com/package/@just-every/mcp-read-website-fast)
|
|
6
|
+
[](https://github.com/just-every/mcp-read-website-fast/actions)
|
|
7
|
+
|
|
8
|
+
## Overview
|
|
2
9
|
|
|
3
10
|
Existing MCP web crawlers are slow and consume large quantities of tokens. This pauses the development process and provides incomplete results as LLMs need to parse whole web pages.
|
|
4
11
|
|
|
5
12
|
This MCP package fetches web pages locally, strips noise, and converts content to clean Markdown while preserving links. Designed for Claude Code, IDEs and LLM pipelines with minimal token footprint. Crawl sites locally with minimal dependencies.
|
|
6
13
|
|
|
7
|
-
##
|
|
14
|
+
## Features
|
|
8
15
|
|
|
9
|
-
|
|
16
|
+
- **Fast startup** using official MCP SDK with lazy loading for optimal performance
|
|
17
|
+
- **Content extraction** using Mozilla Readability (same as Firefox Reader View)
|
|
18
|
+
- **HTML to Markdown** conversion with Turndown + GFM support
|
|
19
|
+
- **Smart caching** with SHA-256 hashed URLs
|
|
20
|
+
- **Polite crawling** with robots.txt support and rate limiting
|
|
21
|
+
- **Concurrent fetching** with configurable depth crawling
|
|
22
|
+
- **Stream-first design** for low memory usage
|
|
23
|
+
- **Link preservation** for knowledge graphs
|
|
24
|
+
- **Optional chunking** for downstream processing
|
|
10
25
|
|
|
11
26
|
## Installation
|
|
12
27
|
|
|
@@ -160,6 +175,32 @@ npm run typecheck
|
|
|
160
175
|
npm run lint
|
|
161
176
|
```
|
|
162
177
|
|
|
178
|
+
## Contributing
|
|
179
|
+
|
|
180
|
+
Contributions are welcome! Please:
|
|
181
|
+
|
|
182
|
+
1. Fork the repository
|
|
183
|
+
2. Create a feature branch
|
|
184
|
+
3. Add tests for new functionality
|
|
185
|
+
4. Submit a pull request
|
|
186
|
+
|
|
187
|
+
## Troubleshooting
|
|
188
|
+
|
|
189
|
+
### Cache Issues
|
|
190
|
+
```bash
|
|
191
|
+
npm run dev clear-cache
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
### Timeout Errors
|
|
195
|
+
- Increase timeout with `-t` flag
|
|
196
|
+
- Check network connectivity
|
|
197
|
+
- Verify URL is accessible
|
|
198
|
+
|
|
199
|
+
### Content Not Extracted
|
|
200
|
+
- Some sites block automated access
|
|
201
|
+
- Try custom user agent with `-u` flag
|
|
202
|
+
- Check if site requires JavaScript (not supported)
|
|
203
|
+
|
|
163
204
|
## License
|
|
164
205
|
|
|
165
206
|
MIT
|
package/package.json
CHANGED