webforai 4.1.0 → 4.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +69 -86
  2. package/dist/bin.js +1 -1
  3. package/package.json +1 -1
package/README.md CHANGED
@@ -37,130 +37,113 @@ Everything is documented at **[webforai.dev](https://webforai.dev)**. Start from
37
37
  | convert HTML in my own code (Node.js, browsers, Workers) | the `webforai` library | [Getting started](https://webforai.dev/getting-started) |
38
38
  | get Markdown for a URL from the shell or an AI agent | the CLI, `npx webforai <url>` | [CLI](https://webforai.dev/cli) |
39
39
  | call a hosted API that runs the browsers and proxies for me | webforai platform | [platform.webforai.dev](https://platform.webforai.dev) (sign-up, API keys, usage) · [platform docs](https://webforai.dev/platform) |
40
- | run my own copy of that platform | [`apps/platform`](https://github.com/inaridiy/webforai/tree/main/apps/platform) | [Deploying your own instance](https://github.com/inaridiy/webforai/tree/main/apps/platform#deploying-your-own-instance) |
41
40
 
42
41
  The library, the CLI and the typed platform client (`webforai/platform`) all ship in the one
43
- `webforai` npm package.
42
+ `webforai` npm package:
44
43
 
45
- ## Overview
44
+ | Import | What it is |
45
+ | --- | --- |
46
+ | `webforai` | `htmlToMarkdown` and the conversion API |
47
+ | `webforai/platform` | typed client for the hosted platform |
48
+ | `webforai/loaders/fetch` | load HTML over `fetch` |
49
+ | `webforai/loaders/playwright` | load JavaScript-rendered pages (needs `playwright-core`) |
50
+
51
+ ## Quick start
52
+
53
+ ### CLI
46
54
 
47
55
  ```bash
48
56
  # URL or local HTML file → Markdown on stdout (add -o out.md to write a file)
49
57
  npx webforai@latest https://example.com/article
50
58
 
51
- # machine-readable output, hosted-platform fetching, agent installation
59
+ # machine-readable output
52
60
  npx webforai@latest https://example.com --json
53
- npx webforai@latest https://example.com --engine auto # platform, renders JS when needed
61
+ ```
62
+
63
+ These commands use the [hosted platform](#platform-hosted-api) and need an API key:
64
+
65
+ ```bash
66
+ npx webforai@latest https://example.com --engine auto # renders JavaScript when needed
54
67
  npx webforai@latest crawl https://docs.example.com -o docs --llms-txt # site → .md files + llms.txt
55
68
  npx webforai@latest batch --file urls.txt -o pages # many URLs → one .md per page
56
- npx webforai@latest skill --install # Agent Skill via `npx skills add`
57
69
  ```
58
70
 
59
- or
71
+ To let an AI agent (Claude Code, Cursor, …) call the CLI, install it as an Agent Skill with
72
+ `npx webforai@latest skill --install`. Flags, exit codes and error output are covered in the
73
+ [CLI docs](https://webforai.dev/cli).
60
74
 
61
- ```ts
62
- import { htmlToMarkdown, htmlToMdast } from "webforai";
63
- import { loadHtml } from "webforai/loaders/playwright";
64
-
65
- // Load html from url
66
- const url = "https://www.npmjs.com/package/webforai";
67
- const html = await loadHtml(url);
75
+ ### Library
68
76
 
69
- // Convert html to markdown
70
- const markdown = htmlToMarkdown(html, { baseUrl: url });
77
+ ```bash
78
+ npm i webforai
71
79
  ```
72
80
 
73
- The CLI keeps Markdown/JSON on stdout and API failures on stderr, including the error code
74
- and `retryAfter` when supplied; `invalid_api_key`, `payment_required` and `spend_cap_reached`
75
- name the platform dashboard. `crawl` and `batch` run hosted jobs (API key required) and write
76
- one Markdown file per page. `--engine` and `--region` select the platform loader; combining
77
- them with an explicit local loader exits with usage error `2`. The interactive wizard applies
78
- the same validation and honors `WEBFORAI_PLATFORM_URL`.
81
+ ```ts
82
+ import { htmlToMarkdown } from "webforai";
79
83
 
80
- ## kiwame: learned main-content extraction
84
+ const url = "https://example.com/article";
85
+ const html = await (await fetch(url)).text();
81
86
 
82
- Pages without a site adapter go through **kiwame** (極め, an appraiser's verdict), a small learned
83
- block classifier: the page is split
84
- into text blocks, each scored by gradient-boosted trees over cheap features (text statistics,
85
- structure, neighbours, and what the Readability-style heuristic would keep), and the tree is
86
- pruned to the kept blocks. The model is ~100 KB of plain TypeScript arrays — no WASM, no extra
87
- dependency — and runs comfortably inside a Cloudflare Worker.
87
+ // url selects a site-specific adapter (GitHub, MDN, …) and fills metadata;
88
+ // baseUrl resolves relative links
89
+ const markdown = htmlToMarkdown(html, { url, baseUrl: url });
90
+ ```
88
91
 
89
- On the [WCEB](https://github.com/chatnoir-eu/web-content-extraction-benchmark) benchmark it raised
90
- token F1 from 0.874 to 0.892. The previous heuristic remains available:
92
+ For pages that need JavaScript to render, load them with the Playwright loader
93
+ (`import { loadHtml } from "webforai/loaders/playwright"`, after `npx playwright-core install`);
94
+ see [Loaders](https://webforai.dev/docs/loaders).
91
95
 
92
- ```ts
93
- import { htmlToMarkdown, createAutoExtractor, takumiExtractor } from "webforai";
96
+ ## What you get
94
97
 
95
- // Site adapters, then the heuristic extractor (the pre-4.1 default).
96
- const markdown = htmlToMarkdown(html, { url, extractors: [createAutoExtractor({ fallback: takumiExtractor })] });
97
- ```
98
+ - **Main content only.** Navigation, ads and footers are dropped. Pages are first matched
99
+ against site adapters; every other page goes through **kiwame**, a small built-in learned
100
+ classifier (plain TypeScript, no WASM, runs in a Cloudflare Worker). See
101
+ [How it works](https://webforai.dev/how-it-works) for the model, the benchmark and how to switch
102
+ back to the heuristic extractor.
103
+ - **Site adapters** for GitHub, Stack Overflow, npm, MDN, Zenn, Qiita, Medium, Substack, note,
104
+ Hatena, WordPress, Wikipedia, Reddit, YouTube, Hacker News, Docusaurus, VitePress, MkDocs,
105
+ Read the Docs and GitBook — matched by hostname or by a platform fingerprint, so self-hosted
106
+ instances are covered too. An adapter that produces too little falls back to generic
107
+ extraction, so a site redesign degrades rather than breaks.
108
+ - **Page metadata** as data or as YAML front matter:
98
109
 
99
- On the CLI, `--extractor kiwame` runs the classifier without site adapters and
100
- `--extractor takumi` the heuristic alone.
110
+ ```ts
111
+ import { htmlToMarkdown, htmlToMarkdownWithMetadata } from "webforai";
101
112
 
102
- ## What v3 changes
113
+ const { markdown, metadata } = htmlToMarkdownWithMetadata(html, { url });
114
+ // metadata: { title, author, published, siteName, canonicalUrl, lang, ... }
103
115
 
104
- **Better extraction.** The main-content extractor scores candidate containers the way Readability
105
- does — paragraph density, link density, class weighting, ancestor propagation — and adds signals
106
- the 2010 heuristics predate: semantic markup (`<article>`, `<main>`, `role="main"`, schema.org
107
- `articleBody`), the rendered geometry a browser loader can supply, and CJK-aware length
108
- thresholds. Across a 60-page corpus of real sites, the share of output lines that were bare links
109
- — the clearest sign navigation leaked in — dropped from 10.6% to 5.0%.
116
+ const withFrontmatter = htmlToMarkdown(html, { url, frontmatter: true });
117
+ ```
110
118
 
111
- **Site adapters.** Pages that generic scoring cannot read are handled directly, matched by
112
- hostname or by a platform fingerprint so self-hosted instances are covered too: GitHub, Stack
113
- Overflow, npm, MDN, Zenn, Qiita, Medium, Substack, note, Hatena, WordPress, Wikipedia, Reddit,
114
- YouTube, Hacker News, Docusaurus, VitePress, MkDocs, Read the Docs and GitBook. An adapter that
115
- produces too little is ignored in favour of generic extraction, so a site redesign degrades
116
- rather than breaks.
119
+ - **Markup that survives conversion**: lazily-loaded images, KaTeX/MathJax, code blocks and tabs,
120
+ tables, definition lists, and superscripts/subscripts.
117
121
 
118
- ```ts
119
- // Pass adapters: false for generic extraction only.
120
- import { createAutoExtractor } from "webforai";
122
+ Release notes and breaking changes are in the
123
+ [CHANGELOG](https://github.com/inaridiy/webforai/blob/main/packages/webforai/CHANGELOG.md).
121
124
 
122
- const markdown = htmlToMarkdown(html, {
123
- url,
124
- extractors: [createAutoExtractor({ adapters: false })],
125
- });
126
- ```
125
+ ## Platform (hosted API)
127
126
 
128
- **Page metadata.**
127
+ webforai platform runs the browsers, proxies and crawl queues for you and returns the same
128
+ Markdown as the library. It includes 1,000 free credits a month; sign up and create an API key
129
+ at [platform.webforai.dev](https://platform.webforai.dev).
129
130
 
130
131
  ```ts
131
- import { htmlToMarkdownWithMetadata } from "webforai";
132
+ import { createPlatformClient } from "webforai/platform";
132
133
 
133
- const { markdown, metadata } = htmlToMarkdownWithMetadata(html, { url });
134
- // metadata: { title, author, published, siteName, canonicalUrl, lang, ... }
135
-
136
- // Or prepend it as YAML front matter:
137
- const withFrontmatter = htmlToMarkdown(html, { url, frontmatter: true });
134
+ const platform = createPlatformClient({ apiKey: process.env.WEBFORAI_API_KEY });
135
+ const page = await platform.scrape({ url: "https://example.com/article" });
136
+ page.markdown;
138
137
  ```
139
138
 
140
- **Markup fixes.** Lazily-loaded images (`data-src`, `srcset`) now resolve to a real URL instead
141
- of `![alt]()`; KaTeX and MathJax expressions are no longer emitted twice; `role="heading"`
142
- elements become real headings; superscripts and subscripts stay distinguishable; and definition
143
- lists keep their term/definition pairing.
144
-
145
- See the [changeset](/.changeset) for the full list, including the breaking changes.
146
-
147
- ## Platform (hosted API)
148
-
149
- [`apps/platform`](https://github.com/inaridiy/webforai/tree/main/apps/platform) is an OSS,
150
- self-hostable SaaS built on this library: a metered crawl→Markdown HTTP API on Cloudflare
151
- (Workers + Workflows + Containers + Browser Run) with Better Auth accounts/API keys and
152
- Stripe usage-based billing. Use it through the typed `webforai/platform` client
153
- (`import { createPlatformClient } from "webforai/platform"`) or the CLI
154
- (`npx webforai <url> --engine auto`). Sign up and manage API keys at
155
- [platform.webforai.dev](https://platform.webforai.dev); the docs live at
156
- [webforai.dev/platform](https://webforai.dev/platform). See its README and the design ledger
157
- in `docs/specs/platform`.
139
+ See the [platform docs](https://webforai.dev/platform) for the API reference and
140
+ [billing](https://webforai.dev/platform/billing).
158
141
 
159
142
  ## Support
160
143
 
161
144
  - [GitHub Sponsors](https://github.com/sponsors/inaridiy)
162
- - [inaridiy.eth](https://x.com/inaridiy)
145
+ - [@inaridiy on X](https://x.com/inaridiy)
163
146
 
164
147
  ## License
165
148
 
166
- [Apache 2.0](/LICENSE) License
149
+ [Apache 2.0](https://github.com/inaridiy/webforai/blob/main/LICENSE)
package/dist/bin.js CHANGED
@@ -6,7 +6,7 @@ import { CommanderError, program } from "commander";
6
6
  // package.json
7
7
  var package_default = {
8
8
  name: "webforai",
9
- version: "4.1.0",
9
+ version: "4.1.1",
10
10
  description: "Convert web pages and local HTML to clean, LLM-ready Markdown: library, CLI (npx webforai) and hosted-API client.",
11
11
  author: "inaridiy",
12
12
  license: "Apache-2.0",
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "webforai",
3
- "version": "4.1.0",
3
+ "version": "4.1.1",
4
4
  "description": "Convert web pages and local HTML to clean, LLM-ready Markdown: library, CLI (npx webforai) and hosted-API client.",
5
5
  "author": "inaridiy",
6
6
  "license": "Apache-2.0",