webforai 1.4.1 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +1 -1
- package/README.md +61 -3
- package/dist/bin.js +1049 -0
- package/dist/cjs/cli/bin.js +34 -0
- package/dist/cjs/cli/commands/handlers/index.js +84 -0
- package/dist/cjs/cli/commands/handlers/loadHtml.js +127 -0
- package/dist/cjs/cli/constants.js +34 -0
- package/dist/cjs/cli/helpers/assertContinue.js +34 -0
- package/dist/cjs/cli/helpers/inputOutputPath.js +89 -0
- package/dist/cjs/cli/helpers/inputSourcePath.js +64 -0
- package/dist/cjs/cli/helpers/selectExtractMode.js +39 -0
- package/dist/cjs/cli/helpers/selectLoader.js +44 -0
- package/dist/cjs/cli/utils.js +122 -0
- package/dist/cjs/html-to-markdown.test.js +6 -34
- package/dist/cjs/index.js +5 -0
- package/dist/cli/bin.js +12 -0
- package/dist/cli/commands/handlers/index.js +51 -0
- package/dist/cli/commands/handlers/loadHtml.js +94 -0
- package/dist/cli/constants.js +9 -0
- package/dist/cli/helpers/assertContinue.js +11 -0
- package/dist/cli/helpers/inputOutputPath.js +56 -0
- package/dist/cli/helpers/inputSourcePath.js +31 -0
- package/dist/cli/helpers/selectExtractMode.js +16 -0
- package/dist/cli/helpers/selectLoader.js +21 -0
- package/dist/cli/utils.js +85 -0
- package/dist/html-to-markdown.test.js +6 -12
- package/dist/index.js +3 -0
- package/dist/types/cli/bin.d.ts +1 -0
- package/dist/types/cli/bin.js.map +1 -0
- package/dist/types/cli/commands/handlers/index.d.ts +13 -0
- package/dist/types/cli/commands/handlers/index.js.map +1 -0
- package/dist/types/cli/commands/handlers/loadHtml.d.ts +2 -0
- package/dist/types/cli/commands/handlers/loadHtml.js.map +1 -0
- package/dist/types/cli/constants.d.ts +5 -0
- package/dist/types/cli/constants.js.map +1 -0
- package/dist/types/cli/helpers/assertContinue.d.ts +1 -0
- package/dist/types/cli/helpers/assertContinue.js.map +1 -0
- package/dist/types/cli/helpers/inputOutputPath.d.ts +1 -0
- package/dist/types/cli/helpers/inputOutputPath.js.map +1 -0
- package/dist/types/cli/helpers/inputSourcePath.d.ts +1 -0
- package/dist/types/cli/helpers/inputSourcePath.js.map +1 -0
- package/dist/types/cli/helpers/selectExtractMode.d.ts +1 -0
- package/dist/types/cli/helpers/selectExtractMode.js.map +1 -0
- package/dist/types/cli/helpers/selectLoader.d.ts +1 -0
- package/dist/types/cli/helpers/selectLoader.js.map +1 -0
- package/dist/types/cli/utils.d.ts +5 -0
- package/dist/types/cli/utils.js.map +1 -0
- package/dist/types/index.d.ts +1 -0
- package/dist/types/index.js.map +1 -1
- package/package.json +11 -21
package/LICENSE
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
MIT License
|
|
2
2
|
|
|
3
|
-
Copyright (c) 2023 inaridiy
|
|
3
|
+
Copyright (c) 2023 - present, inaridiy and webforai contributors
|
|
4
4
|
|
|
5
5
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
6
|
of this software and associated documentation files (the "Software"), to deal
|
package/README.md
CHANGED
|
@@ -10,6 +10,7 @@ A library that provides a web interface for AI
|
|
|
10
10
|
- Ultra-precise Markdown segmentation based on AST
|
|
11
11
|
- Ultra-precise HTML retrieval functions using headless browsers
|
|
12
12
|
- The core functionality is edge-native (runs on Cloudflare Workers!!!)
|
|
13
|
+
- Command-line interface for easy use without coding
|
|
13
14
|
|
|
14
15
|
## Demo
|
|
15
16
|
|
|
@@ -18,14 +19,25 @@ There is a demo API for Html2Markdown deployed on CloudflareWorker. Please acces
|
|
|
18
19
|
- [NPM Package page](https://webforai.inaridiy.workers.dev/?url=https://www.npmjs.com/package/webforai)
|
|
19
20
|
- [Wikipedia of Cloudflare (AI Mode)](https://webforai.inaridiy.workers.dev/?url=https://en.wikipedia.org/wiki/Cloudflare&mode=ai)
|
|
20
21
|
|
|
21
|
-
##
|
|
22
|
-
|
|
23
|
-
Just install and execute scripts
|
|
22
|
+
## Installation
|
|
24
23
|
|
|
24
|
+
### As a library
|
|
25
|
+
To use WebforAI as a library in your project, install it along with playwright:
|
|
25
26
|
```bash
|
|
26
27
|
pnpm i webforai playwright
|
|
27
28
|
```
|
|
28
29
|
|
|
30
|
+
### As a CLI tool
|
|
31
|
+
To use WebforAI as a command-line tool, install it globally:
|
|
32
|
+
```bash
|
|
33
|
+
npm install -g webforai
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
## Quick Start
|
|
37
|
+
|
|
38
|
+
### Using as a library
|
|
39
|
+
Just install and execute scripts
|
|
40
|
+
|
|
29
41
|
```js
|
|
30
42
|
import { promises as fs } from "fs";
|
|
31
43
|
import { htmlToMarkdown, htmlToMdast } from "webforai";
|
|
@@ -41,6 +53,18 @@ await fs.writeFile("output.md", markdown);
|
|
|
41
53
|
|
|
42
54
|
other examples are in [examples](./examples/simple/src/index.ts)
|
|
43
55
|
|
|
56
|
+
### Using as a CLI tool
|
|
57
|
+
After installing globally, you can use the webforai command:
|
|
58
|
+
```bash
|
|
59
|
+
webforai https://www.npmjs.com/package/webforai output.md
|
|
60
|
+
```
|
|
61
|
+
This will fetch the HTML from the specified URL, convert it to Markdown, and save it to output.md.
|
|
62
|
+
|
|
63
|
+
For more CLI options, run:
|
|
64
|
+
```bash
|
|
65
|
+
webforai --help
|
|
66
|
+
```
|
|
67
|
+
|
|
44
68
|
## Examples
|
|
45
69
|
|
|
46
70
|
- [Simple Example](https://github.com/inaridiy/webforai/tree/main/examples/simple/src/index.ts)
|
|
@@ -68,3 +92,37 @@ Convert Mdast to Markdown. If `solveLinks` is specified, the relative links in t
|
|
|
68
92
|
|
|
69
93
|
**`loadHtml(url: string, options?: LoadHtmlOptions): Promise<string>`**
|
|
70
94
|
Load HTML from the specified URL. This function uses Playwright internally.
|
|
95
|
+
|
|
96
|
+
### CLI Commands
|
|
97
|
+
|
|
98
|
+
**`webforai <source?> <outputPath?> [options]`**
|
|
99
|
+
Converts the HTML at the specified URL or path to Markdown and saves it to an output file.
|
|
100
|
+
Arguments and options can be specified in the interactive interface even if they are not specified.
|
|
101
|
+
- **`source`**
|
|
102
|
+
The URL or path to the HTML file to convert.
|
|
103
|
+
- **`outputPath`**
|
|
104
|
+
The path to save the output Markdown file.
|
|
105
|
+
|
|
106
|
+
#### Options
|
|
107
|
+
- **`--mode <mode>`**
|
|
108
|
+
Specify the mode to use for conversion. Options are `default` and `ai`. Default is `default`.
|
|
109
|
+
- **`--loader <loader>`**
|
|
110
|
+
Specify the loader to use for fetching HTML. Options are `fetch`, `playwright` and `puppeteer`. Default is `fetch`.
|
|
111
|
+
- **`--baseUrl <baseUrl>`**
|
|
112
|
+
Specify the base URL to use for relative links in the output Markdown.
|
|
113
|
+
- **`-o --stdout`**
|
|
114
|
+
Output the converted Markdown to the console instead of saving it to a file.
|
|
115
|
+
- **`-d --debug`**
|
|
116
|
+
Enable debug mode. This will output additional information to the console.
|
|
117
|
+
|
|
118
|
+
**`webforai --help`**
|
|
119
|
+
Displays help information for the CLI tool.
|
|
120
|
+
|
|
121
|
+
## Contributing
|
|
122
|
+
1. Fork it!
|
|
123
|
+
2. Create your feature branch: `git checkout -b my-new-feature`
|
|
124
|
+
3. Add your changes: `git add .`
|
|
125
|
+
4. Commit your changes: `git commit -am 'Add some feature'`
|
|
126
|
+
5. Add a changelog: `pnpm changeset`
|
|
127
|
+
6. Push to the branch: `git push origin my-new-feature`
|
|
128
|
+
7. Submit a pull request :sunglasses:
|