mdengine 0.3.0__tar.gz → 0.4.0__tar.gz

{mdengine-0.3.0 → mdengine-0.4.0}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: mdengine
-Version: 0.3.0
+Version: 0.4.0
 Summary: Convert PDF, Office, images, audio, video, text/JSON/XML, ZIP archives, and web URLs to Markdown.
 Author: mdengine contributors
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/vishal7090/md-generator
 Project-URL: Repository, https://github.com/vishal7090/md-generator
 Project-URL: Issues, https://github.com/vishal7090/md-generator/issues
-Keywords: markdown,pdf,docx,pptx,xlsx,ocr,zip,url,html
+Keywords: markdown,pdf,docx,pptx,xlsx,ocr,zip,url,html,youtube,playwright
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: Topic :: Text Processing :: Markup :: Markdown
@@ -74,6 +74,19 @@ Requires-Dist: imageio-ffmpeg>=0.5.1; extra == "audio"
 Provides-Extra: video
 Requires-Dist: openai-whisper>=20231117; extra == "video"
 Requires-Dist: imageio-ffmpeg>=0.5.1; extra == "video"
+Provides-Extra: youtube
+Requires-Dist: youtube-transcript-api>=0.6.0; extra == "youtube"
+Requires-Dist: httpx>=0.27.0; extra == "youtube"
+Requires-Dist: beautifulsoup4>=4.12.0; extra == "youtube"
+Requires-Dist: lxml>=5.0.0; extra == "youtube"
+Provides-Extra: playwright
+Requires-Dist: playwright>=1.49.0; extra == "playwright"
+Requires-Dist: httpx>=0.27.0; extra == "playwright"
+Requires-Dist: beautifulsoup4>=4.12.0; extra == "playwright"
+Requires-Dist: lxml>=5.0.0; extra == "playwright"
+Requires-Dist: markdownify>=0.14.0; extra == "playwright"
+Requires-Dist: readability-lxml>=0.8.1; extra == "playwright"
+Requires-Dist: Pillow>=10.0.0; extra == "playwright"
 Provides-Extra: api
 Requires-Dist: fastapi>=0.115.0; extra == "api"
 Requires-Dist: uvicorn[standard]>=0.32.0; extra == "api"
@@ -86,9 +99,13 @@ Requires-Dist: fastmcp>=2.3.0; extra == "mcp"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0.0; extra == "dev"
 Requires-Dist: httpx>=0.27.0; extra == "dev"
+Requires-Dist: beautifulsoup4>=4.12.0; extra == "dev"
+Requires-Dist: lxml>=5.0.0; extra == "dev"
+Requires-Dist: youtube-transcript-api>=0.6.0; extra == "dev"
 Requires-Dist: mdengine[api]; extra == "dev"
 Requires-Dist: mdengine[mcp]; extra == "dev"
 Requires-Dist: mdengine[url]; extra == "dev"
+Requires-Dist: mdengine[playwright]; extra == "dev"
 Provides-Extra: all
 Requires-Dist: pymupdf>=1.24.0; extra == "all"
 Requires-Dist: pdfplumber>=0.11.0; extra == "all"
@@ -114,6 +131,7 @@ Requires-Dist: fastmcp>=2.3.0; extra == "all"
 Requires-Dist: beautifulsoup4>=4.12.0; extra == "all"
 Requires-Dist: lxml_html_clean>=0.4.0; extra == "all"
 Requires-Dist: readability-lxml>=0.8.1; extra == "all"
+Requires-Dist: youtube-transcript-api>=0.6.0; extra == "all"
 Dynamic: license-file
 
 # mdengine
@@ -238,10 +256,13 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-url` | `md_generator.url.converter:main` | `md-url https://example.com/doc ./web-out --artifact-layout` |
 | `md-audio` | `md_generator.media.audio.converter:main` | `md-audio clip.mp3 transcript.md --model base` |
 | `md-video` | `md_generator.media.video.converter:main` | `md-video clip.mp4 transcript.md --model base` |
+| `md-youtube` | `md_generator.media.youtube.converter:main` | `md-youtube "https://youtu.be/…" out.md --transcript-lang en` |
 | `md-audio-api` | `md_generator.media.audio.api.run:main` | REST + MCP on port **8011** (see [Audio and video to Markdown](#audio-and-video-to-markdown)) |
 | `md-video-api` | `md_generator.media.video.api.run:main` | REST + MCP on port **8012** |
+| `md-youtube-api` | `md_generator.media.youtube.api.run:main` | REST + MCP on port **8013** (JSON `url` body; see same section) |
 | `md-audio-mcp` | `md_generator.media.audio.api.mcp_server:main` | Standalone MCP (`--transport stdio` \| `sse` \| `streamable-http`) |
 | `md-video-mcp` | `md_generator.media.video.api.mcp_server:main` | Same for video |
+| `md-youtube-mcp` | `md_generator.media.youtube.api.mcp_server:main` | Same for YouTube (`youtube_url_to_markdown`) |
 
 Every command accepts **`-h` / `--help`** for full flags (artifact layout, OCR, ZIP options, etc.).
 
@@ -455,6 +476,7 @@ Library code lives under **`md_generator.media`**: shared probing in [`document_
 ```bash
 pip install "mdengine[audio,api,mcp]"    # audio CLI + HTTP + MCP
 pip install "mdengine[video,api,mcp]"   # video CLI + HTTP + MCP (same ML stack as audio)
+pip install "mdengine[youtube,api,mcp]" # YouTube URL → Markdown (captions + metadata; optional Whisper fallback)
 ```
 
 ### Python library
@@ -483,6 +505,18 @@ md = svc.to_markdown(Path("input.mp4"), title=None)
 svc.write_markdown(Path("input.mp4"), Path("out/transcript.md"))
 ```
 
+**YouTube** — captions API + page metadata (BeautifulSoup / oEmbed); optional **yt-dlp + Whisper** when captions are missing (`mdengine[audio]` and `yt-dlp` on `PATH`, or `MD_YOUTUBE_YTDLP`):
+
+```python
+from md_generator.media.youtube import YouTubeToMarkdownService
+
+svc = YouTubeToMarkdownService(whisper_model="base")
+md = svc.to_markdown("https://www.youtube.com/watch?v=VIDEO_ID", transcript_languages=["en"])
+svc.write_markdown("https://youtu.be/VIDEO_ID", Path("out/youtube.md"))
+```
+
+For file-based pipelines, [`YouTubeConverter`](src/md_generator/media/youtube/converter.py) reads a `.url` / `.yturl` / `.youtube` file (or a `.txt` whose first non-comment line is a YouTube URL) and implements [`DocumentConverter`](src/md_generator/media/document_converter.py).
+
 Public symbols are also re-exported from [`md_generator.media`](src/md_generator/media/__init__.py) for ffprobe helpers (`ffprobe_json`, `VideoProbeResult`, …).
 
 ### REST API (FastAPI)
@@ -497,13 +531,15 @@ Each service exposes the same job pattern as other converters:
 | `GET /convert/jobs/{job_id}/download` | Markdown when `done`; workspace removed after download. |
 
 **Audio** defaults: `MD_AUDIO_MAX_UPLOAD_MB=200`, `MD_AUDIO_MAX_SYNC_UPLOAD_MB=40`, `MD_AUDIO_API_PORT=8011`.  
-**Video** defaults: `MD_VIDEO_MAX_UPLOAD_MB=500`, `MD_VIDEO_MAX_SYNC_UPLOAD_MB=80`, `MD_VIDEO_API_PORT=8012`.
+**Video** defaults: `MD_VIDEO_MAX_UPLOAD_MB=500`, `MD_VIDEO_MAX_SYNC_UPLOAD_MB=80`, `MD_VIDEO_API_PORT=8012`.  
+**YouTube** uses **JSON** (not multipart): `POST /convert/sync` and `POST /convert/jobs` accept `{"url":"https://www.youtube.com/watch?v=…","title":null,"transcript_languages":["en"],"enable_audio_fallback":true,"whisper_model":"base","language":null}`. Defaults: `MD_YOUTUBE_API_PORT=8013`, `MD_YOUTUBE_JOB_TTL_SECONDS`, `MD_YOUTUBE_CORS_ORIGINS`, `MD_YOUTUBE_TEMP_DIR`.
 
 Run with the bundled runners (each call builds the app with **`factory=True`** for a clean MCP session manager):
 
 ```bash
 md-audio-api --host 127.0.0.1 --port 8011
 md-video-api --host 127.0.0.1 --port 8012
+md-youtube-api --host 127.0.0.1 --port 8013
 ```
 
 Or with Uvicorn directly (the ASGI app is built by **`create_app()`** so each worker gets its own MCP session manager):
@@ -511,6 +547,7 @@ Or with Uvicorn directly (the ASGI app is built by **`create_app()`** so each wo
 ```bash
 uvicorn md_generator.media.audio.api.main:create_app --factory --host 127.0.0.1 --port 8011
 uvicorn md_generator.media.video.api.main:create_app --factory --host 127.0.0.1 --port 8012
+uvicorn md_generator.media.youtube.api.main:create_app --factory --host 127.0.0.1 --port 8013
 ```
 
 The module also defines **`app = create_app()`** for a single-process target: `uvicorn md_generator.media.audio.api.main:app` (no `--factory`).
@@ -519,7 +556,7 @@ Swagger is at **`/docs`** when the app is running.
 
 ### MCP (stdio, SSE, streamable HTTP)
 
-1. **With FastAPI** — start `md-audio-api` or `md-video-api`; mount Streamable HTTP MCP at **`http://<host>:<port>/mcp`** (same host as REST).
+1. **With FastAPI** — start `md-audio-api`, `md-video-api`, or `md-youtube-api`; mount Streamable HTTP MCP at **`http://<host>:<port>/mcp`** (same host as REST).
 2. **Standalone** — process speaks MCP only:
 
 ```bash
@@ -527,16 +564,18 @@ md-audio-mcp --transport stdio
 md-audio-mcp --transport sse
 md-audio-mcp --transport streamable-http
 md-video-mcp --transport stdio
+md-youtube-mcp --transport stdio
 ```
 
 **Audio MCP tools:** `transcribe_audio_path`, `transcribe_audio_base64`.  
-**Video MCP tools:** `transcribe_video_path`, `transcribe_video_base64`.
+**Video MCP tools:** `transcribe_video_path`, `transcribe_video_base64`.  
+**YouTube MCP tool:** `youtube_url_to_markdown`.
 
-Equivalent modules: `python -m md_generator.media.audio.api.mcp_server`, `python -m md_generator.media.video.api.mcp_server`.
+Equivalent modules: `python -m md_generator.media.audio.api.mcp_server`, `python -m md_generator.media.video.api.mcp_server`, `python -m md_generator.media.youtube.api.mcp_server`.
 
 ### Thin shims (repo clone)
 
-[`audio-to-md/converter.py`](audio-to-md/converter.py) and [`video-to-md/converter.py`](video-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video`. Tests and `pytest.ini` live under `audio-to-md/tests/` and `video-to-md/tests/`.
+[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`.
 
 ---
 
@@ -565,8 +604,10 @@ Install `mdengine[api]` plus the format extra(s), then run the **`app`** object
 | Text/JSON/XML | `md_generator.text.api.main:app` | `text`, `api`, `mcp` |
 | ZIP | `md_generator.archive.api.main:app` | `archive`, `api`, `mcp` (+ extras for nested office/PDF) |
 | URL / HTML | `md_generator.url.api.main:app` | `url`, `api`, `mcp` |
+| Playwright / SPA | `md_generator.playwright.api.main:app` | `playwright`, `api`, `mcp` |
 | Audio (Whisper) | `md_generator.media.audio.api.main:create_app` (use **`--factory`**) or `…main:app` | `audio`, `api`, `mcp` |
 | Video (Whisper) | `md_generator.media.video.api.main:create_app` (use **`--factory`**) or `…main:app` | `video`, `api`, `mcp` |
+| YouTube | `md_generator.media.youtube.api.main:create_app` (use **`--factory`**) or `…main:app` | `youtube`, `api`, `mcp` |
 
 Examples:
 
@@ -575,8 +616,10 @@ uvicorn md_generator.pdf.api.main:app --host 127.0.0.1 --port 8001
 uvicorn md_generator.word.api.main:app --host 127.0.0.1 --port 8002
 uvicorn md_generator.archive.api.main:app --host 127.0.0.1 --port 8010
 uvicorn md_generator.url.api.main:app --host 127.0.0.1 --port 8011
+uvicorn md_generator.playwright.api.main:app --host 127.0.0.1 --port 8014
 uvicorn md_generator.media.audio.api.main:create_app --factory --host 127.0.0.1 --port 8011
 uvicorn md_generator.media.video.api.main:create_app --factory --host 127.0.0.1 --port 8012
+uvicorn md_generator.media.youtube.api.main:create_app --factory --host 127.0.0.1 --port 8013
 ```
 
 ### MCP over HTTP on the same server
@@ -596,8 +639,10 @@ Prefixes differ per service (often read from a `.env` file next to the process):
 | Text | `TXT_JSON_XML_TO_MD_` | same pattern |
 | XLSX | `XLSX_TO_MD_` | `XLSX_TO_MD_TEMP_DIR`, `XLSX_TO_MD_CORS_ORIGINS`, etc. (see `md_generator.xlsx.api.app`) |
 | URL | `URL_TO_MD_` | `URL_TO_MD_MAX_SYNC_URLS`, `URL_TO_MD_MAX_SYNC_CRAWL_PAGES`, `URL_TO_MD_MAX_JOB_URLS`, `URL_TO_MD_JOB_TTL_SECONDS`, `URL_TO_MD_TEMP_DIR`, `URL_TO_MD_CORS_ORIGINS` |
+| Playwright / SPA | `PLAYWRIGHT_TO_MD_` | `PLAYWRIGHT_TO_MD_MAX_SYNC_URLS`, `PLAYWRIGHT_TO_MD_MAX_JOB_URLS`, `PLAYWRIGHT_TO_MD_JOB_TTL_SECONDS`, `PLAYWRIGHT_TO_MD_TEMP_DIR`, `PLAYWRIGHT_TO_MD_CORS_ORIGINS`, `PLAYWRIGHT_TO_MD_API_HOST`, `PLAYWRIGHT_TO_MD_API_PORT` (default **8014**) |
 | Audio API | `MD_AUDIO_` | `MD_AUDIO_MAX_UPLOAD_MB`, `MD_AUDIO_MAX_SYNC_UPLOAD_MB`, `MD_AUDIO_JOB_TTL_SECONDS`, `MD_AUDIO_TEMP_DIR`, `MD_AUDIO_CORS_ORIGINS`, `MD_AUDIO_API_HOST`, `MD_AUDIO_API_PORT` |
 | Video API | `MD_VIDEO_` | Same pattern as audio with `MD_VIDEO_*` (defaults: larger upload/sync caps, port **8012**) |
+| YouTube API | `MD_YOUTUBE_` | `MD_YOUTUBE_JOB_TTL_SECONDS`, `MD_YOUTUBE_TEMP_DIR`, `MD_YOUTUBE_CORS_ORIGINS`, `MD_YOUTUBE_API_HOST`, `MD_YOUTUBE_API_PORT` (default **8013**); optional `MD_YOUTUBE_YTDLP` path for audio fallback |
 
 Exact variable names match the `ApiSettings` / helper functions in each `api/settings` or `api/app` module.
 
@@ -621,8 +666,10 @@ Two usage patterns:
 | PPTX | `python -m md_generator.ppt.api.mcp_server` (see module docstring for flags) |
 | Image | `python -m md_generator.image.api.mcp_server` (see module for CLI) |
 | URL / HTML | `python -m md_generator.url.api.mcp_server` / `--transport sse` / `--transport streamable-http` |
+| Playwright / SPA | `md-playwright-mcp` or `python -m md_generator.playwright.api.mcp_server` / `--transport sse` / `--transport streamable-http` |
 | Audio | `md-audio-mcp` or `python -m md_generator.media.audio.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
 | Video | `md-video-mcp` or `python -m md_generator.media.video.api.mcp_server` — same transports |
+| YouTube | `md-youtube-mcp` or `python -m md_generator.media.youtube.api.mcp_server` — same transports |
 
 **Word** and **XLSX** also ship a small runner script in the repo:
 

{mdengine-0.3.0 → mdengine-0.4.0}/README.md

@@ -120,10 +120,13 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-url` | `md_generator.url.converter:main` | `md-url https://example.com/doc ./web-out --artifact-layout` |
 | `md-audio` | `md_generator.media.audio.converter:main` | `md-audio clip.mp3 transcript.md --model base` |
 | `md-video` | `md_generator.media.video.converter:main` | `md-video clip.mp4 transcript.md --model base` |
+| `md-youtube` | `md_generator.media.youtube.converter:main` | `md-youtube "https://youtu.be/…" out.md --transcript-lang en` |
 | `md-audio-api` | `md_generator.media.audio.api.run:main` | REST + MCP on port **8011** (see [Audio and video to Markdown](#audio-and-video-to-markdown)) |
 | `md-video-api` | `md_generator.media.video.api.run:main` | REST + MCP on port **8012** |
+| `md-youtube-api` | `md_generator.media.youtube.api.run:main` | REST + MCP on port **8013** (JSON `url` body; see same section) |
 | `md-audio-mcp` | `md_generator.media.audio.api.mcp_server:main` | Standalone MCP (`--transport stdio` \| `sse` \| `streamable-http`) |
 | `md-video-mcp` | `md_generator.media.video.api.mcp_server:main` | Same for video |
+| `md-youtube-mcp` | `md_generator.media.youtube.api.mcp_server:main` | Same for YouTube (`youtube_url_to_markdown`) |
 
 Every command accepts **`-h` / `--help`** for full flags (artifact layout, OCR, ZIP options, etc.).
 
@@ -337,6 +340,7 @@ Library code lives under **`md_generator.media`**: shared probing in [`document_
 ```bash
 pip install "mdengine[audio,api,mcp]"    # audio CLI + HTTP + MCP
 pip install "mdengine[video,api,mcp]"   # video CLI + HTTP + MCP (same ML stack as audio)
+pip install "mdengine[youtube,api,mcp]" # YouTube URL → Markdown (captions + metadata; optional Whisper fallback)
 ```
 
 ### Python library
@@ -365,6 +369,18 @@ md = svc.to_markdown(Path("input.mp4"), title=None)
 svc.write_markdown(Path("input.mp4"), Path("out/transcript.md"))
 ```
 
+**YouTube** — captions API + page metadata (BeautifulSoup / oEmbed); optional **yt-dlp + Whisper** when captions are missing (`mdengine[audio]` and `yt-dlp` on `PATH`, or `MD_YOUTUBE_YTDLP`):
+
+```python
+from md_generator.media.youtube import YouTubeToMarkdownService
+
+svc = YouTubeToMarkdownService(whisper_model="base")
+md = svc.to_markdown("https://www.youtube.com/watch?v=VIDEO_ID", transcript_languages=["en"])
+svc.write_markdown("https://youtu.be/VIDEO_ID", Path("out/youtube.md"))
+```
+
+For file-based pipelines, [`YouTubeConverter`](src/md_generator/media/youtube/converter.py) reads a `.url` / `.yturl` / `.youtube` file (or a `.txt` whose first non-comment line is a YouTube URL) and implements [`DocumentConverter`](src/md_generator/media/document_converter.py).
+
 Public symbols are also re-exported from [`md_generator.media`](src/md_generator/media/__init__.py) for ffprobe helpers (`ffprobe_json`, `VideoProbeResult`, …).
 
 ### REST API (FastAPI)
@@ -379,13 +395,15 @@ Each service exposes the same job pattern as other converters:
 | `GET /convert/jobs/{job_id}/download` | Markdown when `done`; workspace removed after download. |
 
 **Audio** defaults: `MD_AUDIO_MAX_UPLOAD_MB=200`, `MD_AUDIO_MAX_SYNC_UPLOAD_MB=40`, `MD_AUDIO_API_PORT=8011`.  
-**Video** defaults: `MD_VIDEO_MAX_UPLOAD_MB=500`, `MD_VIDEO_MAX_SYNC_UPLOAD_MB=80`, `MD_VIDEO_API_PORT=8012`.
+**Video** defaults: `MD_VIDEO_MAX_UPLOAD_MB=500`, `MD_VIDEO_MAX_SYNC_UPLOAD_MB=80`, `MD_VIDEO_API_PORT=8012`.  
+**YouTube** uses **JSON** (not multipart): `POST /convert/sync` and `POST /convert/jobs` accept `{"url":"https://www.youtube.com/watch?v=…","title":null,"transcript_languages":["en"],"enable_audio_fallback":true,"whisper_model":"base","language":null}`. Defaults: `MD_YOUTUBE_API_PORT=8013`, `MD_YOUTUBE_JOB_TTL_SECONDS`, `MD_YOUTUBE_CORS_ORIGINS`, `MD_YOUTUBE_TEMP_DIR`.
 
 Run with the bundled runners (each call builds the app with **`factory=True`** for a clean MCP session manager):
 
 ```bash
 md-audio-api --host 127.0.0.1 --port 8011
 md-video-api --host 127.0.0.1 --port 8012
+md-youtube-api --host 127.0.0.1 --port 8013
 ```
 
 Or with Uvicorn directly (the ASGI app is built by **`create_app()`** so each worker gets its own MCP session manager):
@@ -393,6 +411,7 @@ Or with Uvicorn directly (the ASGI app is built by **`create_app()`** so each wo
 ```bash
 uvicorn md_generator.media.audio.api.main:create_app --factory --host 127.0.0.1 --port 8011
 uvicorn md_generator.media.video.api.main:create_app --factory --host 127.0.0.1 --port 8012
+uvicorn md_generator.media.youtube.api.main:create_app --factory --host 127.0.0.1 --port 8013
 ```
 
 The module also defines **`app = create_app()`** for a single-process target: `uvicorn md_generator.media.audio.api.main:app` (no `--factory`).
@@ -401,7 +420,7 @@ Swagger is at **`/docs`** when the app is running.
 
 ### MCP (stdio, SSE, streamable HTTP)
 
-1. **With FastAPI** — start `md-audio-api` or `md-video-api`; mount Streamable HTTP MCP at **`http://<host>:<port>/mcp`** (same host as REST).
+1. **With FastAPI** — start `md-audio-api`, `md-video-api`, or `md-youtube-api`; mount Streamable HTTP MCP at **`http://<host>:<port>/mcp`** (same host as REST).
 2. **Standalone** — process speaks MCP only:
 
 ```bash
@@ -409,16 +428,18 @@ md-audio-mcp --transport stdio
 md-audio-mcp --transport sse
 md-audio-mcp --transport streamable-http
 md-video-mcp --transport stdio
+md-youtube-mcp --transport stdio
 ```
 
 **Audio MCP tools:** `transcribe_audio_path`, `transcribe_audio_base64`.  
-**Video MCP tools:** `transcribe_video_path`, `transcribe_video_base64`.
+**Video MCP tools:** `transcribe_video_path`, `transcribe_video_base64`.  
+**YouTube MCP tool:** `youtube_url_to_markdown`.
 
-Equivalent modules: `python -m md_generator.media.audio.api.mcp_server`, `python -m md_generator.media.video.api.mcp_server`.
+Equivalent modules: `python -m md_generator.media.audio.api.mcp_server`, `python -m md_generator.media.video.api.mcp_server`, `python -m md_generator.media.youtube.api.mcp_server`.
 
 ### Thin shims (repo clone)
 
-[`audio-to-md/converter.py`](audio-to-md/converter.py) and [`video-to-md/converter.py`](video-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video`. Tests and `pytest.ini` live under `audio-to-md/tests/` and `video-to-md/tests/`.
+[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`.
 
 ---
 
@@ -447,8 +468,10 @@ Install `mdengine[api]` plus the format extra(s), then run the **`app`** object
 | Text/JSON/XML | `md_generator.text.api.main:app` | `text`, `api`, `mcp` |
 | ZIP | `md_generator.archive.api.main:app` | `archive`, `api`, `mcp` (+ extras for nested office/PDF) |
 | URL / HTML | `md_generator.url.api.main:app` | `url`, `api`, `mcp` |
+| Playwright / SPA | `md_generator.playwright.api.main:app` | `playwright`, `api`, `mcp` |
 | Audio (Whisper) | `md_generator.media.audio.api.main:create_app` (use **`--factory`**) or `…main:app` | `audio`, `api`, `mcp` |
 | Video (Whisper) | `md_generator.media.video.api.main:create_app` (use **`--factory`**) or `…main:app` | `video`, `api`, `mcp` |
+| YouTube | `md_generator.media.youtube.api.main:create_app` (use **`--factory`**) or `…main:app` | `youtube`, `api`, `mcp` |
 
 Examples:
 
@@ -457,8 +480,10 @@ uvicorn md_generator.pdf.api.main:app --host 127.0.0.1 --port 8001
 uvicorn md_generator.word.api.main:app --host 127.0.0.1 --port 8002
 uvicorn md_generator.archive.api.main:app --host 127.0.0.1 --port 8010
 uvicorn md_generator.url.api.main:app --host 127.0.0.1 --port 8011
+uvicorn md_generator.playwright.api.main:app --host 127.0.0.1 --port 8014
 uvicorn md_generator.media.audio.api.main:create_app --factory --host 127.0.0.1 --port 8011
 uvicorn md_generator.media.video.api.main:create_app --factory --host 127.0.0.1 --port 8012
+uvicorn md_generator.media.youtube.api.main:create_app --factory --host 127.0.0.1 --port 8013
 ```
 
 ### MCP over HTTP on the same server
@@ -478,8 +503,10 @@ Prefixes differ per service (often read from a `.env` file next to the process):
 | Text | `TXT_JSON_XML_TO_MD_` | same pattern |
 | XLSX | `XLSX_TO_MD_` | `XLSX_TO_MD_TEMP_DIR`, `XLSX_TO_MD_CORS_ORIGINS`, etc. (see `md_generator.xlsx.api.app`) |
 | URL | `URL_TO_MD_` | `URL_TO_MD_MAX_SYNC_URLS`, `URL_TO_MD_MAX_SYNC_CRAWL_PAGES`, `URL_TO_MD_MAX_JOB_URLS`, `URL_TO_MD_JOB_TTL_SECONDS`, `URL_TO_MD_TEMP_DIR`, `URL_TO_MD_CORS_ORIGINS` |
+| Playwright / SPA | `PLAYWRIGHT_TO_MD_` | `PLAYWRIGHT_TO_MD_MAX_SYNC_URLS`, `PLAYWRIGHT_TO_MD_MAX_JOB_URLS`, `PLAYWRIGHT_TO_MD_JOB_TTL_SECONDS`, `PLAYWRIGHT_TO_MD_TEMP_DIR`, `PLAYWRIGHT_TO_MD_CORS_ORIGINS`, `PLAYWRIGHT_TO_MD_API_HOST`, `PLAYWRIGHT_TO_MD_API_PORT` (default **8014**) |
 | Audio API | `MD_AUDIO_` | `MD_AUDIO_MAX_UPLOAD_MB`, `MD_AUDIO_MAX_SYNC_UPLOAD_MB`, `MD_AUDIO_JOB_TTL_SECONDS`, `MD_AUDIO_TEMP_DIR`, `MD_AUDIO_CORS_ORIGINS`, `MD_AUDIO_API_HOST`, `MD_AUDIO_API_PORT` |
 | Video API | `MD_VIDEO_` | Same pattern as audio with `MD_VIDEO_*` (defaults: larger upload/sync caps, port **8012**) |
+| YouTube API | `MD_YOUTUBE_` | `MD_YOUTUBE_JOB_TTL_SECONDS`, `MD_YOUTUBE_TEMP_DIR`, `MD_YOUTUBE_CORS_ORIGINS`, `MD_YOUTUBE_API_HOST`, `MD_YOUTUBE_API_PORT` (default **8013**); optional `MD_YOUTUBE_YTDLP` path for audio fallback |
 
 Exact variable names match the `ApiSettings` / helper functions in each `api/settings` or `api/app` module.
 
@@ -503,8 +530,10 @@ Two usage patterns:
 | PPTX | `python -m md_generator.ppt.api.mcp_server` (see module docstring for flags) |
 | Image | `python -m md_generator.image.api.mcp_server` (see module for CLI) |
 | URL / HTML | `python -m md_generator.url.api.mcp_server` / `--transport sse` / `--transport streamable-http` |
+| Playwright / SPA | `md-playwright-mcp` or `python -m md_generator.playwright.api.mcp_server` / `--transport sse` / `--transport streamable-http` |
 | Audio | `md-audio-mcp` or `python -m md_generator.media.audio.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
 | Video | `md-video-mcp` or `python -m md_generator.media.video.api.mcp_server` — same transports |
+| YouTube | `md-youtube-mcp` or `python -m md_generator.media.youtube.api.mcp_server` — same transports |
 
 **Word** and **XLSX** also ship a small runner script in the repo:
 

{mdengine-0.3.0 → mdengine-0.4.0}/pyproject.toml

@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mdengine"
-version = "0.3.0"
+version = "0.4.0"
 description = "Convert PDF, Office, images, audio, video, text/JSON/XML, ZIP archives, and web URLs to Markdown."
 readme = "README.md"
 requires-python = ">=3.10"
 license = "MIT"
 license-files = ["LICENSE"]
 authors = [{ name = "mdengine contributors" }]
-keywords = ["markdown", "pdf", "docx", "pptx", "xlsx", "ocr", "zip", "url", "html"]
+keywords = ["markdown", "pdf", "docx", "pptx", "xlsx", "ocr", "zip", "url", "html", "youtube", "playwright"]
 classifiers = [
   "Programming Language :: Python :: 3",
   "Programming Language :: Python :: 3 :: Only",
@@ -76,6 +76,22 @@ url-full = [
 audio = ["openai-whisper>=20231117", "imageio-ffmpeg>=0.5.1"]
 # Same ML stack as audio; ships a bundled ``ffmpeg`` binary when the system has none.
 video = ["openai-whisper>=20231117", "imageio-ffmpeg>=0.5.1"]
+youtube = [
+  "youtube-transcript-api>=0.6.0",
+  "httpx>=0.27.0",
+  "beautifulsoup4>=4.12.0",
+  "lxml>=5.0.0",
+]
+# Playwright-rendered SPA → Markdown (install browsers: playwright install chromium).
+playwright = [
+  "playwright>=1.49.0",
+  "httpx>=0.27.0",
+  "beautifulsoup4>=4.12.0",
+  "lxml>=5.0.0",
+  "markdownify>=0.14.0",
+  "readability-lxml>=0.8.1",
+  "Pillow>=10.0.0",
+]
 api = [
   "fastapi>=0.115.0",
   "uvicorn[standard]>=0.32.0",
@@ -87,9 +103,13 @@ mcp = ["mcp>=1.2.0", "fastmcp>=2.3.0"]
 dev = [
   "pytest>=8.0.0",
   "httpx>=0.27.0",
+  "beautifulsoup4>=4.12.0",
+  "lxml>=5.0.0",
+  "youtube-transcript-api>=0.6.0",
   "mdengine[api]",
   "mdengine[mcp]",
   "mdengine[url]",
+  "mdengine[playwright]",
 ]
 all = [
   "pymupdf>=1.24.0",
@@ -116,6 +136,7 @@ all = [
   "beautifulsoup4>=4.12.0",
   "lxml_html_clean>=0.4.0",
   "readability-lxml>=0.8.1",
+  "youtube-transcript-api>=0.6.0",
 ]
 
 [project.urls]
@@ -138,6 +159,12 @@ md-audio-api = "md_generator.media.audio.api.run:main"
 md-video-api = "md_generator.media.video.api.run:main"
 md-audio-mcp = "md_generator.media.audio.api.mcp_server:main"
 md-video-mcp = "md_generator.media.video.api.mcp_server:main"
+md-youtube = "md_generator.media.youtube.converter:main"
+md-youtube-api = "md_generator.media.youtube.api.run:main"
+md-youtube-mcp = "md_generator.media.youtube.api.mcp_server:main"
+md-playwright = "md_generator.playwright.cli:main"
+md-playwright-api = "md_generator.playwright.api.run:main"
+md-playwright-mcp = "md_generator.playwright.api.mcp_server:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]
@@ -155,5 +182,7 @@ testpaths = [
   "url-to-md/tests",
   "audio-to-md/tests",
   "video-to-md/tests",
+  "youtube-to-md/tests",
+  "playwright-to-md/tests",
 ]
 markers = ["integration: optional dependency or slower checks"]

{mdengine-0.3.0 → mdengine-0.4.0}/src/md_generator/__init__.py

@@ -1,5 +1,5 @@
 """md_generator: PDF, Office, image, text, and ZIP to Markdown conversion."""
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 
 __all__ = ["__version__"]

mdengine-0.4.0/src/md_generator/media/youtube/__init__.py

@@ -0,0 +1,34 @@
+"""YouTube URL → Markdown (metadata, description, timestamped transcript)."""
+
+from __future__ import annotations
+
+from md_generator.media.youtube.converter import YouTubeConverter
+from md_generator.media.youtube.formatter import YouTubeMarkdownFormatter
+from md_generator.media.youtube.metadata import (
+    YouTubeMetadataError,
+    extract_video_id,
+    fetch_youtube_metadata,
+    normalize_youtube_url,
+)
+from md_generator.media.youtube.service import (
+    YouTubeConversionResult,
+    YouTubeError,
+    YouTubeToMarkdownService,
+    read_youtube_url_from_path,
+)
+from md_generator.media.youtube.transcript import YouTubeTranscriptError, fetch_transcript
+
+__all__ = [
+    "YouTubeConversionResult",
+    "YouTubeConverter",
+    "YouTubeError",
+    "YouTubeMarkdownFormatter",
+    "YouTubeMetadataError",
+    "YouTubeToMarkdownService",
+    "YouTubeTranscriptError",
+    "extract_video_id",
+    "fetch_transcript",
+    "fetch_youtube_metadata",
+    "normalize_youtube_url",
+    "read_youtube_url_from_path",
+]

mdengine-0.4.0/src/md_generator/media/youtube/api/__init__.py

@@ -0,0 +1 @@
+"""YouTube REST API and MCP entrypoints."""

mdengine-0.4.0/src/md_generator/media/youtube/api/main.py

@@ -0,0 +1,140 @@
+"""FastAPI: JSON URL conversion (sync + jobs) and Streamable HTTP MCP at ``/mcp``."""
+
+from __future__ import annotations
+
+import tempfile
+from contextlib import asynccontextmanager
+from pathlib import Path
+
+from fastapi import FastAPI, HTTPException, Request
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import FileResponse, Response
+from pydantic import BaseModel, Field
+from starlette.background import BackgroundTask
+
+from md_generator.media.job_store import MediaJobStore
+from md_generator.media.youtube.api.mcp_setup import build_mcp_stack
+from md_generator.media.youtube.api.settings import YouTubeApiSettings, cors_list
+from md_generator.media.youtube.metadata import extract_video_id
+from md_generator.media.youtube.service import YouTubeError, YouTubeToMarkdownService
+
+
+class YouTubeConvertRequest(BaseModel):
+    url: str = Field(..., min_length=1, description="YouTube watch, youtu.be, shorts, or embed URL")
+    title: str | None = None
+    transcript_languages: list[str] | None = None
+    enable_audio_fallback: bool = True
+    whisper_model: str = "base"
+    language: str | None = Field(default=None, description="Whisper language if audio fallback runs")
+
+
+def _validate_url(url: str) -> str:
+    u = url.strip()
+    if extract_video_id(u) is None:
+        raise HTTPException(status_code=400, detail="Invalid or unsupported YouTube URL")
+    return u
+
+
+def create_app() -> FastAPI:
+    mcp, mcp_http = build_mcp_stack(mount_under_fastapi=True)
+
+    @asynccontextmanager
+    async def lifespan(application: FastAPI):
+        settings = YouTubeApiSettings()
+        base = Path(settings.temp_dir) if settings.temp_dir else None
+        store = MediaJobStore(base, settings.job_ttl_seconds)
+        store.start_sweeper()
+        application.state.settings = settings
+        application.state.job_store = store
+        async with mcp.session_manager.run():
+            yield
+
+    app = FastAPI(title="youtube-to-md", lifespan=lifespan)
+    app.mount("/mcp", mcp_http)
+
+    _bootstrap = YouTubeApiSettings()
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=cors_list(_bootstrap),
+        allow_credentials="*" not in cors_list(_bootstrap),
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+    @app.post("/convert/sync")
+    async def convert_sync(body: YouTubeConvertRequest) -> Response:
+        url = _validate_url(body.url)
+        with tempfile.TemporaryDirectory(prefix="yt-sync-") as td_name:
+            out = Path(td_name) / "transcript.md"
+            svc = YouTubeToMarkdownService(whisper_model=body.whisper_model, whisper_language=body.language)
+            try:
+                svc.write_markdown(
+                    url,
+                    out,
+                    title=body.title,
+                    transcript_languages=body.transcript_languages,
+                    enable_audio_fallback=body.enable_audio_fallback,
+                )
+            except YouTubeError as e:
+                raise HTTPException(status_code=422, detail=str(e)) from e
+            md = out.read_text(encoding="utf-8")
+        return Response(
+            content=md.encode("utf-8"),
+            media_type="text/markdown; charset=utf-8",
+            headers={"Content-Disposition": 'attachment; filename="transcript.md"'},
+        )
+
+    @app.post("/convert/jobs")
+    async def convert_jobs(request: Request, body: YouTubeConvertRequest) -> dict:
+        url = _validate_url(body.url)
+        store: MediaJobStore = request.app.state.job_store
+        job = store.create_job()
+        out = job.workspace / "transcript.md"
+
+        def work() -> None:
+            svc = YouTubeToMarkdownService(whisper_model=body.whisper_model, whisper_language=body.language)
+            svc.write_markdown(
+                url,
+                out,
+                title=body.title,
+                transcript_languages=body.transcript_languages,
+                enable_audio_fallback=body.enable_audio_fallback,
+            )
+            job.result_path = out
+
+        store.run_async(job, work)
+        return {"job_id": job.job_id, "status": job.status}
+
+    @app.get("/convert/jobs/{job_id}")
+    async def job_status(request: Request, job_id: str) -> dict:
+        store: MediaJobStore = request.app.state.job_store
+        job = store.get(job_id)
+        if not job:
+            raise HTTPException(404, detail="Unknown job_id")
+        return {
+            "status": job.status,
+            "error": job.error,
+            "created_at": job.created_at,
+        }
+
+    @app.get("/convert/jobs/{job_id}/download", response_class=FileResponse)
+    async def job_download(request: Request, job_id: str) -> FileResponse:
+        store: MediaJobStore = request.app.state.job_store
+        job = store.get(job_id)
+        if not job:
+            raise HTTPException(404, detail="Unknown job_id")
+        if job.status != "done" or not job.result_path or not job.result_path.is_file():
+            raise HTTPException(400, detail="Job is not ready for download")
+        path = job.result_path
+        task = BackgroundTask(store.remove_after_download, job)
+        return FileResponse(
+            path,
+            media_type="text/markdown; charset=utf-8",
+            filename="transcript.md",
+            background=task,
+        )
+
+    return app
+
+
+app = create_app()

mdengine-0.4.0/src/md_generator/media/youtube/api/mcp_server.py

@@ -0,0 +1,31 @@
+"""Standalone MCP server: stdio, SSE, or streamable HTTP."""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="youtube-to-md MCP server")
+    parser.add_argument(
+        "--transport",
+        choices=("stdio", "sse", "streamable-http"),
+        default="stdio",
+    )
+    args = parser.parse_args()
+
+    from md_generator.media.youtube.api.mcp_setup import build_mcp_stack
+
+    mcp, _ = build_mcp_stack(mount_under_fastapi=False)
+
+    if args.transport == "stdio":
+        asyncio.run(mcp.run_stdio_async())
+    elif args.transport == "sse":
+        asyncio.run(mcp.run_sse_async())
+    else:
+        asyncio.run(mcp.run_streamable_http_async())
+
+
+if __name__ == "__main__":
+    main()