webcrawl 1.2.0 → 1.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +133 -14
  2. package/package.json +1 -1
package/README.md CHANGED
@@ -1,43 +1,162 @@
1
- # webcrawl — Webcrawl Anything CLI
1
+ # Webcrawl Anything
2
2
 
3
3
  **Crawl, search, and extract structured evidence from any website, SPA, PDF, Office file, YouTube video, GitHub repo, ArXiv paper, shopping offer, travel fare, or CVE advisory — from a single CLI and MCP server.**
4
4
 
5
- ## Quickstart
5
+ [![npm version](https://img.shields.io/npm/v/webcrawl?color=0a0a0a)](https://www.npmjs.com/package/webcrawl)
6
+ [![MCP Server](https://img.shields.io/npm/v/@webcrawl/mcp?label=%40webcrawl%2Fmcp&color=2563eb)](https://www.npmjs.com/package/@webcrawl/mcp)
7
+ [![Cloud API](https://img.shields.io/badge/cloud-webcrawl.app-10b981)](https://webcrawl.app)
8
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/brainfish-ai/webcrawl-anything/blob/main/LICENSE)
9
+
10
+ ---
11
+
12
+ ### What You Can Crawl & Search
13
+
14
+ | Target | Returns | Requirements |
15
+ |---|---|---|
16
+ | **Websites & JS SPAs** (`web`) | Clean Markdown, semantic headings, tables, code blocks, links, OpenGraph/JSON-LD, and SHA-256 content hashes | Zero-key local (`impit` TLS impersonation + local Chrome) or [`webcrawl.app`](https://webcrawl.app) cloud stealth browsers |
17
+ | **PDF & Office Docs** (`pdf-office`) | Page-numbered Markdown, tables, slide notes, and automatic OCR on scanned PDF pages (`pagesOcr` + honest `pagesSkipped`) | Built-in (`@firecrawl/pdf-inspector`, `pdf-parse`, `mammoth`, `officeparser`, `tesseract.js`) |
18
+ | **YouTube & Media** (`youtube-media`) | Timestamped transcript Markdown, video title/channel/duration metadata, and spoken audio transcription | Built-in zero-key YouTube multi-client caption extractor; optional audio transcription provider for raw MP3/MP4 |
19
+ | **Shopping & Products** (`shopping`) | Multi-retailer price comparison, currency-normalized offers, stock status, ratings, review counts, and shipping | Zero-key local HTML/JSON-LD product extractor (`-c shopping`); optional `SERPER_API_KEY` |
20
+ | **Flights & Hotels** (`travel`) | Structured flight routes (`origin`, `destination`, `airline`, `stops`, `price`) and hotel nightly rates, stars, and guest ratings | Zero-key local search + schema.org extractor (`-c flights` / `-c hotels`) |
21
+ | **Real Estate & Local Business** (`real-estate`) | Property listings (`bedrooms`, `bathrooms`, `parking`, `suburb`, `portal`) and local business hours, phone, address, and reviews | Zero-key local extractor (`-c real-estate --sub buy\|rent\|sold` / `-c business`) |
22
+ | **Security & CVEs** (`cve`) | Authoritative CVE & GHSA records with CVSS v3.1/v4.0 scores, CWEs, affected/fixed package versions, and CISA KEV flags | Built-in live NVD 2.0 + OSV.dev + CISA KEV enrichment (`-c security --sub cve`) |
23
+ | **GitHub Repos, Issues & Releases** (`github`) | Structured repo metadata (stars, forks, license, topics, language), README Markdown, or issue/PR/release discussion threads | Built-in zero-key HTML & GitHub extractor (`npx webcrawl scrape https://github.com/owner/repo`) |
24
+ | **ArXiv & Academic Papers** (`arxiv`) | Paper title, authors, abstract, primary category, submission date, direct PDF link, and full parsed paper Markdown | Built-in ArXiv abstract & PDF inspector (`npx webcrawl scrape https://arxiv.org/abs/...`) |
25
+ | **llms.txt & Sitemaps** (`llms-txt`) | Structured documentation manifest, section hierarchy, and crawlable doc URLs | Built-in (`npx webcrawl scrape https://docs.example.com/llms.txt` or `npx webcrawl map`) |
26
+
27
+ Run `npx webcrawl targets` at any time to inspect all registered targets from your terminal.
28
+
29
+ ---
30
+
31
+ ### Quickstart
6
32
 
7
33
  One command registers the `@webcrawl/mcp` server and installs the `webcrawl-anything` Agent Skill (`SKILL.md`) for Claude Code, Cursor, and Codex:
8
34
 
9
35
  ```bash
10
36
  npx webcrawl setup
11
- npx webcrawl doctor
12
37
  ```
13
38
 
14
- ## CLI Commands
39
+ Or target specific coding agents:
40
+
41
+ ```bash
42
+ npx webcrawl setup --agent claude --agent cursor --agent codex --agent gemini --agent windsurf
43
+ ```
44
+
45
+ Check your local engine, stealth browser, and parser readiness:
15
46
 
16
47
  ```bash
17
- # List all supported targets and their requirements
18
- npx webcrawl targets
48
+ npx webcrawl doctor
49
+ ```
50
+
51
+ ---
52
+
53
+ ### Zero-Key CLI Usage
19
54
 
20
- # Scrape any website, GitHub repo, ArXiv paper, PDF, or YouTube video
55
+ Webcrawl Anything runs **out-of-the-box locally with zero API keys** using the built-in `impit` TLS-impersonated fetcher, document/media parsers, BM25 + semantic ranker, and vertical extractors:
56
+
57
+ ```bash
58
+ # Scrape any website, GitHub repo, ArXiv paper, PDF, or YouTube video into clean Markdown
21
59
  npx webcrawl scrape https://github.com/torvalds/linux
22
60
  npx webcrawl scrape https://arxiv.org/abs/1706.03762
61
+ npx webcrawl scrape https://www.youtube.com/watch?v=dQw4w9WgXcQ
23
62
 
24
- # Search the web or structured verticals
63
+ # Search the live web or structured verticals
25
64
  npx webcrawl search "react 19 compiler" -n 5
26
65
  npx webcrawl search "sony wh-1000xm5" -c shopping
27
66
  npx webcrawl search "boutique hotels in shibuya" -c hotels
67
+ npx webcrawl search "3 bedroom house in marrickville" -c real-estate --sub buy
28
68
  npx webcrawl search "CVE-2024-3094 xz utils" -c security --sub cve
29
69
 
30
- # Discover URLs and crawl multi-page documentation
70
+ # Discover every URL on a domain (sitemap.xml + robots.txt + link graph)
31
71
  npx webcrawl map https://docs.stripe.com --filter webhook -n 50
72
+
73
+ # Crawl up to 25 pages in parallel with target evidence
32
74
  npx webcrawl crawl https://docs.stripe.com/webhooks -n 5
33
75
 
34
- # Connect to webcrawl.app cloud for managed stealth browsers & global cache
76
+ # Extract structured JSON from any page
77
+ npx webcrawl extract https://news.ycombinator.com --prompt "top 5 stories with points"
78
+ ```
79
+
80
+ ---
81
+
82
+ ### Example Prompts (Once Connected to Your Agent)
83
+
84
+ Once `npx webcrawl setup` has registered the MCP server and skill, ask Claude Code, Cursor, or Codex directly:
85
+
86
+ - *"Search for CVE-2024-3094 in the security category and show me the CVSS score, CISA KEV status, and fixed versions."*
87
+ - *"Scrape `https://arxiv.org/abs/2305.18290` and summarize the DPO objective and experimental results."*
88
+ - *"Compare retail prices and stock availability for the Apple Mac Mini M4 across US retailers using `web_search` with `category: shopping`."*
89
+ - *"Map `https://modelcontextprotocol.io`, pick the specification pages, and crawl them into Markdown."*
90
+ - *"Parse the PDF at `https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf` and verify whether any pages required OCR."*
91
+
92
+ ---
93
+
94
+ ### Local Engine vs. [`webcrawl.app`](https://webcrawl.app) Cloud
95
+
96
+ Webcrawl Anything uses the **exact same TypeScript engine (`src/`)** locally and in the hosted [`webcrawl.app`](https://webcrawl.app) cloud API. You can start with zero keys locally and upgrade to cloud with one command whenever you need managed stealth browsers or sub-50ms global cache hits:
97
+
98
+ | Capability | Open-Source Local (`--mode local`) | [`webcrawl.app`](https://webcrawl.app) Cloud (`--mode cloud`) |
99
+ |---|---|---|
100
+ | **API Key Required** | **None (Zero-Key)** | `WEBCRAWL_API_KEY` (`npx webcrawl login`) |
101
+ | **Core Engine & Target Plugins** | 100% shared (`src/engine`, `src/targets`, `src/extract`) | 100% shared |
102
+ | **PDF, Office, YouTube & CVE Parsers** | Built-in local execution | Hosted + background async workers (`/v1/runs/:id`) |
103
+ | **Anti-Bot & Cloudflare Bypass** | Local `impit` TLS + local Chrome | Managed residential stealth browser fleet + Cloudflare rendering |
104
+ | **Search Fan-Out** | Built-in DuckDuckGo/Bing + BYO keys | Managed multi-provider fan-out (Serper, Brave, Jina, Tavily) |
105
+ | **Global Content Cache** | Local in-memory LRU | Sub-50ms global SHA-256 content-addressed cache |
106
+
107
+ To connect your CLI and MCP server to [`webcrawl.app`](https://webcrawl.app) cloud:
108
+
109
+ ```bash
35
110
  npx webcrawl login
36
111
  npx webcrawl whoami
37
112
  ```
38
113
 
39
- ## Documentation & Source
114
+ ---
115
+
116
+ ### Compact MCP Schema Profile for Context-Constrained Agents
117
+
118
+ By default, `npx webcrawl setup` configures `WEBCRAWL_MCP_SCHEMA_PROFILE=compact` so MCP tool schemas use minimal token overhead inside Claude Code, Cursor, and local LLM context windows while keeping full server-side validation.
119
+
120
+ To switch to verbose tool parameter descriptions:
121
+
122
+ ```bash
123
+ npx webcrawl setup --schema-profile full
124
+ ```
125
+
126
+ ---
127
+
128
+ ### Architecture & Shared Engine Flywheel
129
+
130
+ ```text
131
+ webcrawl/
132
+ ├── src/
133
+ │ ├── targets/ # Pluggable Target & Extractor Registry (web, pdf-office, youtube, github, arxiv, llms-txt, shopping, travel, real-estate, cve)
134
+ │ ├── engine/ # Multi-tier fetcher (impit TLS impersonation → stealth browser → HTML-to-Markdown + SHA-256 cache)
135
+ │ ├── parse/ # Native PDF inspector + OCR fallback, DOCX/XLSX/PPTX parser, YouTube caption extractor
136
+ │ ├── extract/ # Vertical structured extractors (shopping, flights, hotels, real-estate, business, CVE/NVD/OSV)
137
+ │ ├── search/ # Multi-provider search fan-out + vertical pipelines
138
+ │ ├── ranker/ # Hybrid BM25 + semantic vector + RRF + MMR diversity reranker
139
+ │ └── local-engine.ts # In-process zero-key backend powering CLI & MCP locally
140
+ ├── packages/
141
+ │ ├── cli/ # `webcrawl` CLI (`npx webcrawl search|scrape|map|crawl|extract|rerank|targets|setup|doctor`)
142
+ │ ├── mcp/ # `@webcrawl/mcp` stdio & Streamable HTTP MCP server
143
+ │ └── client/ # `@webcrawl/client` zero-dependency TypeScript SDK
144
+ └── api/
145
+ └── index.ts # Production `webcrawl.app` API powered by the exact same `src/` engine
146
+ ```
147
+
148
+ Every community improvement to a target plugin (`src/targets/`), vertical extractor (`src/extract/`), or document parser (`src/parse/`) immediately powers both local CLI/MCP users and the hosted [`webcrawl.app`](https://webcrawl.app) platform.
149
+
150
+ ---
151
+
152
+ ### Contributing a New Target or Extractor
153
+
154
+ Want to add a new target (e.g., Reddit threads, Hacker News discussions, SEC EDGAR filings, npm/PyPI packages, or a regional real-estate/shopping portal)?
155
+
156
+ See **[CONTRIBUTING.md](https://github.com/brainfish-ai/webcrawl-anything/blob/main/CONTRIBUTING.md)** for the 3-step guide to registering a `CrawlTargetPlugin` in `src/targets/` with a fixture test in `src/__tests__/targets.test.ts`.
157
+
158
+ ---
159
+
160
+ ### License
40
161
 
41
- - **GitHub Repository:** [brainfish-ai/webcrawl-anything](https://github.com/brainfish-ai/webcrawl-anything)
42
- - **Cloud Platform:** [https://webcrawl.app](https://webcrawl.app)
43
- - **License:** MIT
162
+ MIT
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "webcrawl",
3
- "version": "1.2.0",
3
+ "version": "1.2.1",
4
4
  "description": "Webcrawl Anything CLI — crawl, search, and extract structured evidence from any website, SPA, PDF, Office file, YouTube video, GitHub repo, ArXiv paper, or CVE",
5
5
  "type": "module",
6
6
  "license": "MIT",