webforai 4.0.0 → 4.1.1

package/README.md

@@ -37,108 +37,113 @@ Everything is documented at **[webforai.dev](https://webforai.dev)**. Start from
 | convert HTML in my own code (Node.js, browsers, Workers) | the `webforai` library | [Getting started](https://webforai.dev/getting-started) |
 | get Markdown for a URL from the shell or an AI agent | the CLI, `npx webforai <url>` | [CLI](https://webforai.dev/cli) |
 | call a hosted API that runs the browsers and proxies for me | webforai platform | [platform.webforai.dev](https://platform.webforai.dev) (sign-up, API keys, usage) · [platform docs](https://webforai.dev/platform) |
-| run my own copy of that platform | [`apps/platform`](https://github.com/inaridiy/webforai/tree/main/apps/platform) | [Deploying your own instance](https://github.com/inaridiy/webforai/tree/main/apps/platform#deploying-your-own-instance) |
 
 The library, the CLI and the typed platform client (`webforai/platform`) all ship in the one
-`webforai` npm package.
+`webforai` npm package:
 
-## Overview
+| Import | What it is |
+| --- | --- |
+| `webforai` | `htmlToMarkdown` and the conversion API |
+| `webforai/platform` | typed client for the hosted platform |
+| `webforai/loaders/fetch` | load HTML over `fetch` |
+| `webforai/loaders/playwright` | load JavaScript-rendered pages (needs `playwright-core`) |
+
+## Quick start
+
+### CLI
 
 ```bash
 # URL or local HTML file → Markdown on stdout (add -o out.md to write a file)
 npx webforai@latest https://example.com/article
 
-# machine-readable output, hosted-platform fetching, agent installation
+# machine-readable output
 npx webforai@latest https://example.com --json
-npx webforai@latest https://example.com --engine auto      # platform, renders JS when needed
+```
+
+These commands use the [hosted platform](#platform-hosted-api) and need an API key:
+
+```bash
+npx webforai@latest https://example.com --engine auto      # renders JavaScript when needed
 npx webforai@latest crawl https://docs.example.com -o docs --llms-txt   # site → .md files + llms.txt
 npx webforai@latest batch --file urls.txt -o pages         # many URLs → one .md per page
-npx webforai@latest skill --install                        # Agent Skill via `npx skills add`
 ```
 
-or
-
-```ts
-import { htmlToMarkdown, htmlToMdast } from "webforai";
-import { loadHtml } from "webforai/loaders/playwright";
+To let an AI agent (Claude Code, Cursor, …) call the CLI, install it as an Agent Skill with
+`npx webforai@latest skill --install`. Flags, exit codes and error output are covered in the
+[CLI docs](https://webforai.dev/cli).
 
-// Load html from url
-const url = "https://www.npmjs.com/package/webforai";
-const html = await loadHtml(url);
+### Library
 
-// Convert html to markdown
-const markdown = htmlToMarkdown(html, { baseUrl: url });
+```bash
+npm i webforai
 ```
 
-The CLI keeps Markdown/JSON on stdout and API failures on stderr, including the error code
-and `retryAfter` when supplied; `invalid_api_key`, `payment_required` and `spend_cap_reached`
-name the platform dashboard. `crawl` and `batch` run hosted jobs (API key required) and write
-one Markdown file per page. `--engine` and `--region` select the platform loader; combining
-them with an explicit local loader exits with usage error `2`. The interactive wizard applies
-the same validation and honors `WEBFORAI_PLATFORM_URL`.
-
-## What v3 changes
-
-**Better extraction.** The main-content extractor scores candidate containers the way Readability
-does — paragraph density, link density, class weighting, ancestor propagation — and adds signals
-the 2010 heuristics predate: semantic markup (`<article>`, `<main>`, `role="main"`, schema.org
-`articleBody`), the rendered geometry a browser loader can supply, and CJK-aware length
-thresholds. Across a 60-page corpus of real sites, the share of output lines that were bare links
-— the clearest sign navigation leaked in — dropped from 10.6% to 5.0%.
-
-**Site adapters.** Pages that generic scoring cannot read are handled directly, matched by
-hostname or by a platform fingerprint so self-hosted instances are covered too: GitHub, Stack
-Overflow, npm, MDN, Zenn, Qiita, Medium, Substack, note, Hatena, WordPress, Wikipedia, Reddit,
-YouTube, Hacker News, Docusaurus, VitePress, MkDocs, Read the Docs and GitBook. An adapter that
-produces too little is ignored in favour of generic extraction, so a site redesign degrades
-rather than breaks.
-
 ```ts
-// Pass adapters: false for generic extraction only.
-import { createAutoExtractor } from "webforai";
+import { htmlToMarkdown } from "webforai";
 
-const markdown = htmlToMarkdown(html, {
-  url,
-  extractors: [createAutoExtractor({ adapters: false })],
-});
+const url = "https://example.com/article";
+const html = await (await fetch(url)).text();
+
+// url selects a site-specific adapter (GitHub, MDN, …) and fills metadata;
+// baseUrl resolves relative links
+const markdown = htmlToMarkdown(html, { url, baseUrl: url });
 ```
 
-**Page metadata.**
+For pages that need JavaScript to render, load them with the Playwright loader
+(`import { loadHtml } from "webforai/loaders/playwright"`, after `npx playwright-core install`);
+see [Loaders](https://webforai.dev/docs/loaders).
 
-```ts
-import { htmlToMarkdownWithMetadata } from "webforai";
+## What you get
 
-const { markdown, metadata } = htmlToMarkdownWithMetadata(html, { url });
-// metadata: { title, author, published, siteName, canonicalUrl, lang, ... }
+- **Main content only.** Navigation, ads and footers are dropped. Pages are first matched
+  against site adapters; every other page goes through **kiwame**, a small built-in learned
+  classifier (plain TypeScript, no WASM, runs in a Cloudflare Worker). See
+  [How it works](https://webforai.dev/how-it-works) for the model, the benchmark and how to switch
+  back to the heuristic extractor.
+- **Site adapters** for GitHub, Stack Overflow, npm, MDN, Zenn, Qiita, Medium, Substack, note,
+  Hatena, WordPress, Wikipedia, Reddit, YouTube, Hacker News, Docusaurus, VitePress, MkDocs,
+  Read the Docs and GitBook — matched by hostname or by a platform fingerprint, so self-hosted
+  instances are covered too. An adapter that produces too little falls back to generic
+  extraction, so a site redesign degrades rather than breaks.
+- **Page metadata** as data or as YAML front matter:
 
-// Or prepend it as YAML front matter:
-const withFrontmatter = htmlToMarkdown(html, { url, frontmatter: true });
-```
+  ```ts
+  import { htmlToMarkdown, htmlToMarkdownWithMetadata } from "webforai";
+
+  const { markdown, metadata } = htmlToMarkdownWithMetadata(html, { url });
+  // metadata: { title, author, published, siteName, canonicalUrl, lang, ... }
+
+  const withFrontmatter = htmlToMarkdown(html, { url, frontmatter: true });
+  ```
 
-**Markup fixes.** Lazily-loaded images (`data-src`, `srcset`) now resolve to a real URL instead
-of `![alt]()`; KaTeX and MathJax expressions are no longer emitted twice; `role="heading"`
-elements become real headings; superscripts and subscripts stay distinguishable; and definition
-lists keep their term/definition pairing.
+- **Markup that survives conversion**: lazily-loaded images, KaTeX/MathJax, code blocks and tabs,
+  tables, definition lists, and superscripts/subscripts.
 
-See the [changeset](/.changeset) for the full list, including the breaking changes.
+Release notes and breaking changes are in the
+[CHANGELOG](https://github.com/inaridiy/webforai/blob/main/packages/webforai/CHANGELOG.md).
 
 ## Platform (hosted API)
 
-[`apps/platform`](https://github.com/inaridiy/webforai/tree/main/apps/platform) is an OSS,
-self-hostable SaaS built on this library: a metered crawl→Markdown HTTP API on Cloudflare
-(Workers + Workflows + Containers + Browser Run) with Better Auth accounts/API keys and
-Stripe usage-based billing. Use it through the typed `webforai/platform` client
-(`import { createPlatformClient } from "webforai/platform"`) or the CLI
-(`npx webforai <url> --engine auto`). Sign up and manage API keys at
-[platform.webforai.dev](https://platform.webforai.dev); the docs live at
-[webforai.dev/platform](https://webforai.dev/platform). See its README and the design ledger
-in `docs/specs/platform`.
+webforai platform runs the browsers, proxies and crawl queues for you and returns the same
+Markdown as the library. It includes 1,000 free credits a month; sign up and create an API key
+at [platform.webforai.dev](https://platform.webforai.dev).
+
+```ts
+import { createPlatformClient } from "webforai/platform";
+
+const platform = createPlatformClient({ apiKey: process.env.WEBFORAI_API_KEY });
+const page = await platform.scrape({ url: "https://example.com/article" });
+page.markdown;
+```
+
+See the [platform docs](https://webforai.dev/platform) for the API reference and
+[billing](https://webforai.dev/platform/billing).
 
 ## Support
 
 - [GitHub Sponsors](https://github.com/sponsors/inaridiy)
-- [inaridiy.eth](https://x.com/inaridiy)
+- [@inaridiy on X](https://x.com/inaridiy)
 
 ## License
 
-[Apache 2.0](/LICENSE) License
+[Apache 2.0](https://github.com/inaridiy/webforai/blob/main/LICENSE)