webforai 3.0.0 → 4.1.0

package/README.md

@@ -10,8 +10,8 @@
 </p>
 
 <p align="center">
-  A esm-native library that converts HTML to Markdown & Useful Utilities with simple, lightweight and epic quality.
-<p>
+  Convert web pages and local HTML to clean, LLM-ready Markdown — with a TypeScript library, the <code>npx webforai</code> CLI, or a hosted API.
+</p>
 
 <p align="center">
   <a href="https://www.npmjs.com/package/webforai">
@@ -30,7 +30,17 @@
 
 ## Documentation
 
-[Head to the documentation](https://webforai.dev/) to read and learn more about Webforai.
+Everything is documented at **[webforai.dev](https://webforai.dev)**. Start from what you want to do:
+
+| I want to… | Use | Start here |
+| --- | --- | --- |
+| convert HTML in my own code (Node.js, browsers, Workers) | the `webforai` library | [Getting started](https://webforai.dev/getting-started) |
+| get Markdown for a URL from the shell or an AI agent | the CLI, `npx webforai <url>` | [CLI](https://webforai.dev/cli) |
+| call a hosted API that runs the browsers and proxies for me | webforai platform | [platform.webforai.dev](https://platform.webforai.dev) (sign-up, API keys, usage) · [platform docs](https://webforai.dev/platform) |
+| run my own copy of that platform | [`apps/platform`](https://github.com/inaridiy/webforai/tree/main/apps/platform) | [Deploying your own instance](https://github.com/inaridiy/webforai/tree/main/apps/platform#deploying-your-own-instance) |
+
+The library, the CLI and the typed platform client (`webforai/platform`) all ship in the one
+`webforai` npm package.
 
 ## Overview
 
@@ -40,7 +50,9 @@ npx webforai@latest https://example.com/article
 
 # machine-readable output, hosted-platform fetching, agent installation
 npx webforai@latest https://example.com --json
-npx webforai@latest https://example.com --engine browser   # platform, renders JS
+npx webforai@latest https://example.com --engine auto      # platform, renders JS when needed
+npx webforai@latest crawl https://docs.example.com -o docs --llms-txt   # site → .md files + llms.txt
+npx webforai@latest batch --file urls.txt -o pages         # many URLs → one .md per page
 npx webforai@latest skill --install                        # Agent Skill via `npx skills add`
 ```
 
@@ -58,6 +70,35 @@ const html = await loadHtml(url);
 const markdown = htmlToMarkdown(html, { baseUrl: url });
 ```
 
+The CLI keeps Markdown/JSON on stdout and API failures on stderr, including the error code
+and `retryAfter` when supplied; `invalid_api_key`, `payment_required` and `spend_cap_reached`
+name the platform dashboard. `crawl` and `batch` run hosted jobs (API key required) and write
+one Markdown file per page. `--engine` and `--region` select the platform loader; combining
+them with an explicit local loader exits with usage error `2`. The interactive wizard applies
+the same validation and honors `WEBFORAI_PLATFORM_URL`.
+
+## kiwame: learned main-content extraction
+
+Pages without a site adapter go through **kiwame** (極め, an appraiser's verdict), a small learned
+block classifier: the page is split
+into text blocks, each scored by gradient-boosted trees over cheap features (text statistics,
+structure, neighbours, and what the Readability-style heuristic would keep), and the tree is
+pruned to the kept blocks. The model is ~100 KB of plain TypeScript arrays — no WASM, no extra
+dependency — and runs comfortably inside a Cloudflare Worker.
+
+On the [WCEB](https://github.com/chatnoir-eu/web-content-extraction-benchmark) benchmark it raised
+token F1 from 0.874 to 0.892. The previous heuristic remains available:
+
+```ts
+import { htmlToMarkdown, createAutoExtractor, takumiExtractor } from "webforai";
+
+// Site adapters, then the heuristic extractor (the pre-4.1 default).
+const markdown = htmlToMarkdown(html, { url, extractors: [createAutoExtractor({ fallback: takumiExtractor })] });
+```
+
+On the CLI, `--extractor kiwame` runs the classifier without site adapters and
+`--extractor takumi` the heuristic alone.
+
 ## What v3 changes
 
 **Better extraction.** The main-content extractor scores candidate containers the way Readability
@@ -110,7 +151,8 @@ self-hostable SaaS built on this library: a metered crawl→Markdown HTTP API on
 (Workers + Workflows + Containers + Browser Run) with Better Auth accounts/API keys and
 Stripe usage-based billing. Use it through the typed `webforai/platform` client
 (`import { createPlatformClient } from "webforai/platform"`) or the CLI
-(`npx webforai <url> --engine browser`); docs live at
+(`npx webforai <url> --engine auto`). Sign up and manage API keys at
+[platform.webforai.dev](https://platform.webforai.dev); the docs live at
 [webforai.dev/platform](https://webforai.dev/platform). See its README and the design ledger
 in `docs/specs/platform`.