into-md 0.1.0

package/docs/SPEC.md

@@ -0,0 +1,201 @@
+# into-md
+
+A CLI tool that fetches web pages and converts them to clean markdown, optimized for providing context to LLMs.
+
+## Overview
+
+`into-md` fetches a single URL, extracts the main content using readability heuristics, and outputs clean markdown suitable for LLM consumption. It preserves images with context, converts tables to structured JSON, and includes standard metadata.
+
+## Installation
+
+```bash
+npm install -g into-md
+# or
+bunx into-md <url>
+```
+
+## Usage
+
+```bash
+into-md <url> [options]
+```
+
+### Examples
+
+```bash
+# Output to stdout
+into-md https://example.com/article
+
+# Save to file
+into-md https://example.com/article -o article.md
+
+# Use headless browser for JS-rendered content
+into-md https://spa-site.com/page --js
+
+# Skip content extraction, convert full page
+into-md https://example.com --raw
+
+# With authentication cookies
+into-md https://private-site.com/page --cookies cookies.txt
+
+# Verbose output
+into-md https://example.com/article -v
+```
+
+## Options
+
+| Flag                    | Description                                               | Default         |
+| ----------------------- | --------------------------------------------------------- | --------------- |
+| `-o, --output <file>`   | Write output to file instead of stdout                    | stdout          |
+| `--js`                  | Use headless browser (Playwright) for JS-rendered content | disabled        |
+| `--raw`                 | Skip content extraction, convert entire HTML              | disabled        |
+| `--cookies <file>`      | Path to cookies file for authenticated requests           | none            |
+| `--user-agent <string>` | Custom User-Agent header                                  | browser-like UA |
+| `--encoding <encoding>` | Force character encoding (auto-detected by default)       | auto            |
+| `--strip-links`         | Remove hyperlinks, keep only anchor text                  | disabled        |
+| `--exclude <selectors>` | CSS selectors to exclude (comma-separated)                | none            |
+| `--timeout <ms>`        | Request timeout in milliseconds                           | 30000           |
+| `--no-cache`            | Bypass response cache                                     | cache enabled   |
+| `-v, --verbose`         | Show detailed progress information                        | minimal         |
+| `-h, --help`            | Show help                                                 | -               |
+| `--version`             | Show version                                              | -               |
+
+## Output Format
+
+### Frontmatter
+
+Standard metadata is included as YAML frontmatter when available:
+
+```yaml
+---
+title: "Article Title"
+description: "Meta description from the page"
+author: "Author Name"
+date: "2024-01-15"
+source: "https://example.com/article"
+---
+```
+
+### Content Structure
+
+- **Headings**: Preserved as-is from source (original hierarchy maintained)
+- **Text formatting**: Semantic formatting preserved (bold, italic, strikethrough); decorative formatting (colors, underlines) stripped
+- **Links**: Preserved as markdown links by default; all relative URLs converted to absolute
+- **Code blocks**: Language auto-detected and tagged for syntax highlighting
+
+### Images
+
+Images include alt text, URL, and surrounding context:
+
+```markdown
+![Diagram showing the system architecture](https://example.com/images/arch.png)
+_Figure 1: The system uses a microservices architecture with three main components._
+```
+
+### Tables
+
+Tables are converted to fenced JSON blocks for reliable LLM parsing:
+
+```json
+{
+	"caption": "Quarterly Revenue",
+	"headers": ["Quarter", "Revenue", "Growth"],
+	"rows": [
+		{ "Quarter": "Q1", "Revenue": "$1.2M", "Growth": "12%" },
+		{ "Quarter": "Q2", "Revenue": "$1.5M", "Growth": "25%" }
+	]
+}
+```
+
+### Embedded Content
+
+Embeds (iframes, videos, tweets) are replaced with links:
+
+```markdown
+[Embedded video: https://youtube.com/watch?v=xyz123]
+```
+
+## Content Extraction
+
+By default, `into-md` uses readability-style heuristics to:
+
+- Extract main article/content area
+- Remove navigation, headers, footers, sidebars
+- Strip ads, cookie banners, and promotional content
+- Filter out irrelevant widgets and scripts
+
+Use `--exclude` to fine-tune extraction with additional CSS selectors:
+
+```bash
+into-md https://example.com --exclude ".comments, .related-posts, #newsletter-signup"
+```
+
+Use `--raw` to bypass extraction and convert the entire page.
+
+## Caching
+
+Responses are cached locally by default to avoid redundant fetches. Cache location: `~/.cache/into-md/`
+
+- Default TTL: 1 hour
+- Use `--no-cache` to fetch fresh content
+- Cache is keyed by URL
+
+## Size Warnings
+
+If the output exceeds 100KB, a warning is printed to stderr:
+
+```
+Warning: Output is 156KB. Large documents may exceed LLM context limits.
+```
+
+## Authentication
+
+For pages requiring authentication, export cookies from your browser and pass them via `--cookies`:
+
+```bash
+into-md https://private-docs.company.com/page --cookies ~/cookies.txt
+```
+
+Cookie file format: Netscape/Mozilla cookie file format (compatible with browser extensions like EditThisCookie).
+
+## Error Handling
+
+- **403/Blocked**: Clear error message suggesting `--user-agent` option
+- **Timeouts**: Respects `--timeout` flag, defaults to 30 seconds
+- **Encoding issues**: Auto-detects from headers/meta, converts to UTF-8; use `--encoding` to override
+
+## Technical Stack
+
+- **Runtime**: Bun
+- **Language**: TypeScript
+- **HTML Parsing**: cheerio
+- **Markdown Conversion**: turndown
+- **Content Extraction**: @mozilla/readability
+- **Headless Browser**: playwright (optional, for `--js` mode)
+- **CLI Framework**: commander or yargs
+
+## Project Structure
+
+```
+into-md/
+├── src/
+│   ├── index.ts          # CLI entry point
+│   ├── fetcher.ts        # URL fetching (static + headless)
+│   ├── extractor.ts      # Content extraction with readability
+│   ├── converter.ts      # HTML to markdown conversion
+│   ├── tables.ts         # Table to JSON conversion
+│   ├── images.ts         # Image context extraction
+│   ├── metadata.ts       # Frontmatter generation
+│   └── cache.ts          # Response caching
+├── package.json
+├── tsconfig.json
+└── SPEC.md
+```
+
+## Future Considerations (Out of Scope for v1)
+
+- Batch processing of multiple URLs
+- Same-domain crawling with depth control
+- Config file for persistent preferences
+- Prebuilt binaries via GitHub releases
+- Full authentication support (headers, basic auth)

package/package.json

@@ -0,0 +1,39 @@
+{
+  "name": "into-md",
+  "private": false,
+  "bin": {
+    "into-md": "dist/index.mjs"
+  },
+  "type": "module",
+  "module": "src/index.ts",
+  "scripts": {
+    "start": "bun run src/index.ts",
+    "build": "tsdown",
+    "build:watch": "tsdown --watch",
+    "test": "bun test",
+    "lint": "ultracite check",
+    "fix": "ultracite fix",
+    "fix:unsafe": "ultracite fix --unsafe",
+    "typecheck": "tsc --noEmit"
+  },
+  "dependencies": {
+    "@mozilla/readability": "^0.5.0",
+    "@types/turndown": "^5.0.6",
+    "cheerio": "^1.0.0",
+    "commander": "^12.1.0",
+    "jsdom": "^24.1.0",
+    "playwright": "^1.42.1",
+    "turndown": "^7.2.0"
+  },
+  "devDependencies": {
+    "@biomejs/biome": "2.3.11",
+    "@types/bun": "latest",
+    "oxlint": "^1.38.0",
+    "tsdown": "^0.19.0",
+    "ultracite": "7.0.11"
+  },
+  "peerDependencies": {
+    "typescript": "^5"
+  },
+  "version": "0.1.0"
+}

package/src/cache.ts

@@ -0,0 +1,79 @@
+import { createHash } from "node:crypto";
+import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
+import { dirname, join } from "node:path";
+
+export interface CacheOptions {
+  enabled: boolean;
+  ttlMs: number;
+  cacheDir?: string;
+}
+
+export interface CachedResponse {
+  url: string;
+  fetchedAt: number;
+  content: string;
+}
+
+const defaultCacheDir = join(
+  process.env.HOME ?? process.cwd(),
+  ".cache",
+  "into-md"
+);
+
+const DEFAULT_TTL_MS = 60 * 60 * 1000;
+
+const buildCachePath = (url: string, cacheDir = defaultCacheDir): string => {
+  const hash = createHash("sha256").update(url).digest("hex");
+  return join(cacheDir, `${hash}.json`);
+};
+
+export async function readFromCache(
+  url: string,
+  options?: Partial<CacheOptions>
+): Promise<CachedResponse | null> {
+  const {
+    enabled = true,
+    ttlMs = DEFAULT_TTL_MS,
+    cacheDir = defaultCacheDir,
+  } = options ?? {};
+
+  if (!enabled) {
+    return null;
+  }
+
+  const target = buildCachePath(url, cacheDir);
+  try {
+    const [file, info] = await Promise.all([
+      readFile(target, "utf8"),
+      stat(target),
+    ]);
+    const payload = JSON.parse(file) as CachedResponse;
+    const isFresh = info.mtimeMs + ttlMs > Date.now();
+    if (!isFresh) {
+      return null;
+    }
+    if (payload.url !== url) {
+      return null;
+    }
+    return payload;
+  } catch {
+    return null;
+  }
+}
+
+export async function writeToCache(
+  url: string,
+  content: string,
+  options?: Partial<CacheOptions>
+): Promise<void> {
+  const { enabled = true, cacheDir = defaultCacheDir } = options ?? {};
+
+  if (!enabled) {
+    return;
+  }
+
+  const target = buildCachePath(url, cacheDir);
+  await mkdir(dirname(target), { recursive: true });
+  const payload: CachedResponse = { content, fetchedAt: Date.now(), url };
+  await writeFile(target, JSON.stringify(payload, null, 2), "utf8");
+}

package/src/converter.ts

@@ -0,0 +1,96 @@
+import { load } from "cheerio";
+import TurndownService from "turndown";
+
+import { getBodyHtml, toAbsoluteUrl } from "./utils";
+
+export interface ConvertOptions {
+  baseUrl: string;
+  stripLinks?: boolean;
+}
+
+function prepareDom(html: string, baseUrl: string): string {
+  const $ = load(html);
+
+  for (const el of $("a[href]").toArray()) {
+    const $el = $(el);
+    const absolute = toAbsoluteUrl($el.attr("href"), baseUrl);
+    if (absolute) {
+      $el.attr("href", absolute);
+    }
+  }
+
+  for (const el of $("img[src]").toArray()) {
+    const $el = $(el);
+    const absolute = toAbsoluteUrl($el.attr("src"), baseUrl);
+    if (absolute) {
+      $el.attr("src", absolute);
+    }
+  }
+
+  $("script, style").remove();
+  return getBodyHtml($);
+}
+
+export function convertHtmlToMarkdown(
+  html: string,
+  options: ConvertOptions
+): string {
+  const prepared = prepareDom(html, options.baseUrl);
+  const turndown = new TurndownService({
+    bulletListMarker: "-",
+    codeBlockStyle: "fenced",
+    headingStyle: "atx",
+  });
+
+  turndown.addRule("stripLinks", {
+    filter: "a",
+    replacement: (content, node) => {
+      if (options.stripLinks) {
+        return content;
+      }
+      const href = (node as HTMLElement).getAttribute("href");
+      if (!href) {
+        return content;
+      }
+      return `[${content}](${href})`;
+    },
+  });
+
+  turndown.addRule("imagesWithCaption", {
+    filter: "img",
+    replacement: (_, node) => {
+      const element = node as HTMLElement;
+      const src = element.getAttribute("src") ?? "";
+      const alt = element.getAttribute("alt") ?? "";
+      const caption = element.getAttribute("data-into-md-caption");
+      const imageLine = `![${alt}](${src})`;
+      if (caption) {
+        return `${imageLine}\n*${caption}*`;
+      }
+      return imageLine;
+    },
+  });
+
+  turndown.addRule("tableJson", {
+    filter: (node) =>
+      node.nodeName === "PRE" &&
+      (node as HTMLElement).getAttribute("data-into-md-table") === "true",
+    replacement: (_content, node) => {
+      const text = (node as HTMLElement).textContent?.trim() ?? "";
+      return `\`\`\`json\n${text}\n\`\`\``;
+    },
+  });
+
+  turndown.addRule("embeds", {
+    filter: ["iframe", "embed", "video"],
+    replacement: (_, node) => {
+      const src = (node as HTMLElement).getAttribute("src") ?? "";
+      if (!src) {
+        return "";
+      }
+      return `[Embedded content: ${src}]`;
+    },
+  });
+
+  return turndown.turndown(prepared);
+}

package/src/extractor.ts

@@ -0,0 +1,85 @@
+import { Readability } from "@mozilla/readability";
+import { JSDOM } from "jsdom";
+
+export interface ExtractOptions {
+  raw?: boolean;
+  excludeSelectors?: string[];
+  baseUrl: string;
+}
+
+export interface ExtractedContent {
+  html: string;
+  metadata: {
+    title?: string;
+    description?: string;
+    author?: string;
+    source: string;
+  };
+}
+
+function removeNodes(document: Document, selectors: string[]) {
+  for (const selector of selectors) {
+    for (const node of Array.from(document.querySelectorAll(selector))) {
+      node.remove();
+    }
+  }
+}
+
+function extractMetadata(document: Document, source: string) {
+  const title =
+    document.querySelector("title")?.textContent ??
+    document
+      .querySelector('meta[property="og:title"]')
+      ?.getAttribute("content") ??
+    undefined;
+
+  const description =
+    document
+      .querySelector('meta[name="description"]')
+      ?.getAttribute("content") ??
+    document
+      .querySelector('meta[property="og:description"]')
+      ?.getAttribute("content") ??
+    undefined;
+
+  const author =
+    document.querySelector('meta[name="author"]')?.getAttribute("content") ??
+    document
+      .querySelector('meta[property="article:author"]')
+      ?.getAttribute("content") ??
+    undefined;
+
+  return { author, description, source, title: title ?? undefined };
+}
+
+export function extractContent(
+  html: string,
+  { raw = false, excludeSelectors = [], baseUrl }: ExtractOptions
+): ExtractedContent {
+  const dom = new JSDOM(html, { url: baseUrl });
+  const { document } = dom.window;
+
+  if (excludeSelectors.length) {
+    removeNodes(document, excludeSelectors);
+  }
+
+  if (raw) {
+    const metadata = extractMetadata(document, baseUrl);
+    return { html: document.documentElement.outerHTML, metadata };
+  }
+
+  const clone = document.cloneNode(true) as Document;
+  const reader = new Readability(clone);
+  const article = reader.parse();
+
+  const contentHtml =
+    article?.content ?? document.querySelector("body")?.innerHTML ?? "";
+  const metadata = extractMetadata(document, baseUrl);
+  if (article?.title && !metadata.title) {
+    metadata.title = article.title;
+  }
+  if (article?.byline && !metadata.author) {
+    metadata.author = article.byline;
+  }
+  return { html: contentHtml, metadata };
+}