@aiquants/html-to-markdown 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 AIQuants
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,130 @@
+# @aiquants/html-to-markdown
+
+A tool to dynamically fetch a web page from a given URL and convert it to Markdown.
+
+This tool fetches the fully rendered HTML after JavaScript execution and converts it to Markdown. It supports complex pages, including footnotes and table structures.
+
+## Key Features
+
+- **Dynamic Content Fetching**: Accurately fetches content from dynamic sites like SPAs (Single Page Applications) using Playwright.
+- **High-Fidelity Conversion**: Performs powerful AST (Abstract Syntax Tree)-based conversion using `rehype` and `remark`.
+- **Preserves Table Content**: Retains HTML tags within table cells (`<td>`, `<th>`) as much as possible to maintain rich formatting.
+- **Link Normalization**: Automatically converts relative links on the page to absolute links to prevent broken links.
+- **Dual Interface**: Usable as both a Node.js library and a command-line tool.
+
+## Installation
+
+```bash
+npm install @aiquants/html-to-markdown
+```
+
+## Usage
+
+### As a Command-Line Tool
+
+You can run the tool directly using `npx` without installation. The converted Markdown will be saved in the `.outputs/raw` directory.
+
+```bash
+npx @aiquants/html-to-markdown <URL> [--locale <locale>]
+```
+
+**Example:**
+
+```bash
+# Convert a Wikipedia page with the Japanese locale
+npx @aiquants/html-to-markdown https://ja.wikipedia.org/wiki/Node.js --locale ja-JP
+
+# Convert an English page (locale defaults to en-US)
+npx @aiquants/html-to-markdown https://en.wikipedia.org/wiki/Node.js
+```
+
+### As a Library
+
+```typescript
+import { htmlToMarkdown } from '@aiquants/html-to-markdown';
+import fs from 'fs';
+
+async function main() {
+  const url = 'https://en.wikipedia.org/wiki/Node.js';
+  const options = {
+    locale: 'en-US', // 'en-US' (default) or 'ja-JP'
+  };
+
+  try {
+    const markdown = await htmlToMarkdown(url, options);
+    fs.writeFileSync('output.md', markdown);
+    console.log('Markdown file has been saved as output.md');
+  } catch (error) {
+    console.error('Error converting HTML to Markdown:', error);
+  }
+}
+
+main();
+```
+
+## API
+
+### `htmlToMarkdown(url, options?)`
+
+Converts the HTML content of a given URL to Markdown.
+
+- `url` (`string`, **required**): The URL of the web page to convert.
+- `options` (`object`, optional): Options for the conversion process.
+
+#### `options` object
+
+- `locale` (`string`, optional): Specifies the locale to use for the browser context and console messages.
+  - `'en-US'` (default)
+  - `'ja-JP'`
+
+## How It Works
+
+This tool follows these steps for conversion:
+
+1. **Fetch HTML (Playwright)**:
+    - Launches a browser with Playwright using the specified locale.
+    - Navigates to the page and waits until the `networkidle` event, ensuring dynamic content is fully loaded before fetching the HTML.
+
+2. **Pre-process HTML (Rehype)**:
+    - `rehype-parse`: Parses the HTML into a HAST (HTML Abstract Syntax Tree).
+    - `rehype-raw`: Preserves elements like `script` and `style`.
+    - `rehypeSanitizeHtml` (custom plugin): Removes empty comment nodes and unnecessary whitespace.
+    - `rehypeAbsoluteLinks` (custom plugin): Converts relative paths in `href` and `src` attributes to absolute paths.
+    - `rehypeWikipediaFootnotes` (custom plugin): Transforms Wikipedia footnotes into standard Markdown format.
+    - `rehypeSlug` & `rehypeAutolinkHeadings`: Adds IDs to headings and automatically generates anchor links.
+
+3. **Convert to Markdown (Remark)**:
+    - `rehype-remark`: Converts the HAST to an MDAST (Markdown Abstract Syntax Tree), with custom handling for links (`<a>`) and table cells (`<td>`, `<th>`).
+    - `remark-gfm`: Adds support for GitHub Flavored Markdown (GFM), including tables and strikethrough.
+    - `remark-stringify`: Serializes the MDAST into a Markdown string.
+
+## Dependencies and Licenses
+
+This project is built upon the following open-source software. We are grateful to the developers of these libraries.
+
+| Package                  | License    |
+| ------------------------ | ---------- |
+| github-slugger           | MIT        |
+| hast                     | MIT        |
+| hast-util-to-html        | MIT        |
+| playwright               | Apache-2.0 |
+| rehype-parse             | MIT        |
+| rehype-raw               | MIT        |
+| rehype-remark            | MIT        |
+| rehype-slug              | MIT        |
+| remark-gfm               | MIT        |
+| remark-stringify         | MIT        |
+| unified                  | MIT        |
+| unist-util-visit         | MIT        |
+| yargs-parser             | ISC        |
+
+*This list is generated based on the `dependencies` in `package.json`. For the most accurate and up-to-date license information, please refer to the individual packages.*
+
+## Author
+
+- **GitHub**: [fehde-k](https://github.com/fehde-k)
+- **X (formerly Twitter)**: [@fehdek](https://x.com/fehdek)
+
+## License
+
+MIT