mdengine 0.1.0__tar.gz → 0.2.2__tar.gz

{mdengine-0.1.0 → mdengine-0.2.2}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: mdengine
-Version: 0.1.0
-Summary: Convert PDF, Office, images, text/JSON/XML, and ZIP archives to Markdown.
+Version: 0.2.2
+Summary: Convert PDF, Office, images, text/JSON/XML, ZIP archives, and web URLs to Markdown.
 Author: mdengine contributors
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/vishal7090/md-generator
 Project-URL: Repository, https://github.com/vishal7090/md-generator
 Project-URL: Issues, https://github.com/vishal7090/md-generator/issues
-Keywords: markdown,pdf,docx,pptx,xlsx,ocr,zip
+Keywords: markdown,pdf,docx,pptx,xlsx,ocr,zip,url,html
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: Topic :: Text Processing :: Markup :: Markdown
@@ -46,6 +46,28 @@ Provides-Extra: text
 Provides-Extra: archive
 Requires-Dist: Pillow>=10.0.0; extra == "archive"
 Requires-Dist: pytesseract>=0.3.10; extra == "archive"
+Provides-Extra: url
+Requires-Dist: httpx>=0.27.0; extra == "url"
+Requires-Dist: beautifulsoup4>=4.12.0; extra == "url"
+Requires-Dist: lxml>=5.0.0; extra == "url"
+Requires-Dist: lxml_html_clean>=0.4.0; extra == "url"
+Requires-Dist: markdownify>=0.14.0; extra == "url"
+Requires-Dist: readability-lxml>=0.8.1; extra == "url"
+Provides-Extra: url-full
+Requires-Dist: httpx>=0.27.0; extra == "url-full"
+Requires-Dist: beautifulsoup4>=4.12.0; extra == "url-full"
+Requires-Dist: lxml>=5.0.0; extra == "url-full"
+Requires-Dist: lxml_html_clean>=0.4.0; extra == "url-full"
+Requires-Dist: markdownify>=0.14.0; extra == "url-full"
+Requires-Dist: readability-lxml>=0.8.1; extra == "url-full"
+Requires-Dist: pymupdf>=1.24.0; extra == "url-full"
+Requires-Dist: pdfplumber>=0.11.0; extra == "url-full"
+Requires-Dist: mammoth>=1.8.0; extra == "url-full"
+Requires-Dist: python-pptx>=1.0.0; extra == "url-full"
+Requires-Dist: Pillow>=10.0.0; extra == "url-full"
+Requires-Dist: olefile>=0.47; extra == "url-full"
+Requires-Dist: openpyxl>=3.1.0; extra == "url-full"
+Requires-Dist: pytesseract>=0.3.10; extra == "url-full"
 Provides-Extra: api
 Requires-Dist: fastapi>=0.115.0; extra == "api"
 Requires-Dist: uvicorn[standard]>=0.32.0; extra == "api"
@@ -60,6 +82,7 @@ Requires-Dist: pytest>=8.0.0; extra == "dev"
 Requires-Dist: httpx>=0.27.0; extra == "dev"
 Requires-Dist: mdengine[api]; extra == "dev"
 Requires-Dist: mdengine[mcp]; extra == "dev"
+Requires-Dist: mdengine[url]; extra == "dev"
 Provides-Extra: all
 Requires-Dist: pymupdf>=1.24.0; extra == "all"
 Requires-Dist: pdfplumber>=0.11.0; extra == "all"
@@ -82,6 +105,9 @@ Requires-Dist: httpx>=0.27.0; extra == "all"
 Requires-Dist: pydantic-settings>=2.0.0; extra == "all"
 Requires-Dist: mcp>=1.2.0; extra == "all"
 Requires-Dist: fastmcp>=2.3.0; extra == "all"
+Requires-Dist: beautifulsoup4>=4.12.0; extra == "all"
+Requires-Dist: lxml_html_clean>=0.4.0; extra == "all"
+Requires-Dist: readability-lxml>=0.8.1; extra == "all"
 Dynamic: license-file
 
 # mdengine
@@ -158,6 +184,8 @@ pip install "mdengine[all]"
 | `image-ocr` | Heavy OCR backends (pytesseract, paddle, easyocr, …) |
 | `text` | TXT / JSON / XML converter (stdlib-oriented; marker extra) |
 | `archive` | ZIP → Markdown layout (Pillow; optional tesseract for inline image OCR) |
+| `url` | HTTP(S) HTML → Markdown (httpx, readability-lxml, markdownify, BeautifulSoup, lxml) |
+| `url-full` | `url` plus PDF/Word/PPTX/XLSX/archive stack for **post-converting** downloaded linked files to Markdown |
 | `api` | FastAPI, uvicorn, httpx, pydantic-settings |
 | `mcp` | MCP servers (`mcp`, `fastmcp` where used) |
 | `dev` | pytest + API/MCP test helpers |
@@ -199,6 +227,7 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-image` | `md_generator.image.converter:main` | `md-image ./scans page.md` |
 | `md-text` | `md_generator.text.converter:main` | `md-text config.xml out.md` |
 | `md-zip` | `md_generator.archive.converter:main` | `md-zip bundle.zip ./zip-out` |
+| `md-url` | `md_generator.url.converter:main` | `md-url https://example.com/doc ./web-out --artifact-layout` |
 
 Every command accepts **`-h` / `--help`** for full flags (artifact layout, OCR, ZIP options, etc.).
 
@@ -215,6 +244,7 @@ md-xlsx -i export.csv -o ./csv-out
 md-image ./photos ocr.md --engines tess --strategy best
 md-text data.json data.md
 md-zip archive.zip ./unzipped-md
+md-url https://example.com/page ./page-bundle --artifact-layout
 ```
 
 **Windows PowerShell** (same commands; use backslashes for paths if you prefer)
@@ -222,6 +252,7 @@ md-zip archive.zip ./unzipped-md
 ```powershell
 md-pdf .\manual.pdf .\out\doc.md
 md-zip .\archive.zip .\zip-out
+md-url https://example.com/page .\page-bundle --artifact-layout
 ```
 
 **Windows CMD**
@@ -229,11 +260,12 @@ md-zip .\archive.zip .\zip-out
 ```cmd
 md-pdf manual.pdf out\doc.md
 md-zip archive.zip zip-out
+md-url https://example.com/page page-bundle --artifact-layout
 ```
 
 ### 5. Run without `pip install` (repo clone + `PYTHONPATH`)
 
-The folders `pdf-to-md/`, `word-to-md/`, … contain a thin `converter.py` that calls the same code as `md-pdf`, `md-word`, etc. From the **repository root**, point Python at `src` so `md_generator` imports, then run the shim:
+The folders `pdf-to-md/`, `word-to-md/`, `url-to-md/`, … contain a thin `converter.py` that calls the same code as `md-pdf`, `md-word`, etc. From the **repository root**, point Python at `src` so `md_generator` imports, then run the shim:
 
 **PowerShell**
 
@@ -394,12 +426,12 @@ convert_zip(
 
 All format APIs follow a similar pattern:
 
-- **`POST /convert/sync`** — upload a file; response is often a **ZIP** (artifact bundle) for larger formats.
+- **`POST /convert/sync`** — upload a file (most converters) **or** send JSON (`url-to-md`); response is often a **ZIP** (artifact bundle) for larger formats.
 - **`POST /convert/jobs`** — async job; returns `job_id`.
 - **`GET /convert/jobs/{job_id}`** — status.
 - **`GET /convert/jobs/{job_id}/download`** — download result when ready.
 
-Upload field name is **`file`** (multipart form). Use `httpx` or `curl -F "file=@path/to/file"`.
+Upload field name is **`file`** (multipart form) for file-based converters. Use `httpx` or `curl -F "file=@path/to/file"`. **URL** conversion uses a **JSON** body (`url` or `urls`); see [url-to-md/README.md](url-to-md/README.md).
 
 ### Run with Uvicorn
 
@@ -414,6 +446,7 @@ Install `mdengine[api]` plus the format extra(s), then run the **`app`** object
 | Image | `md_generator.image.api.main:app` | `image`, `api`, `mcp` |
 | Text/JSON/XML | `md_generator.text.api.main:app` | `text`, `api`, `mcp` |
 | ZIP | `md_generator.archive.api.main:app` | `archive`, `api`, `mcp` (+ extras for nested office/PDF) |
+| URL / HTML | `md_generator.url.api.main:app` | `url`, `api`, `mcp` |
 
 Examples:
 
@@ -421,6 +454,7 @@ Examples:
 uvicorn md_generator.pdf.api.main:app --host 127.0.0.1 --port 8001
 uvicorn md_generator.word.api.main:app --host 127.0.0.1 --port 8002
 uvicorn md_generator.archive.api.main:app --host 127.0.0.1 --port 8010
+uvicorn md_generator.url.api.main:app --host 127.0.0.1 --port 8011
 ```
 
 ### MCP over HTTP on the same server
@@ -439,6 +473,7 @@ Prefixes differ per service (often read from a `.env` file next to the process):
 | PPTX | `PPT_TO_MD_` | `PPT_TO_MD_MAX_UPLOAD_MB`, … |
 | Text | `TXT_JSON_XML_TO_MD_` | same pattern |
 | XLSX | `XLSX_TO_MD_` | `XLSX_TO_MD_TEMP_DIR`, `XLSX_TO_MD_CORS_ORIGINS`, etc. (see `md_generator.xlsx.api.app`) |
+| URL | `URL_TO_MD_` | `URL_TO_MD_MAX_SYNC_URLS`, `URL_TO_MD_MAX_SYNC_CRAWL_PAGES`, `URL_TO_MD_MAX_JOB_URLS`, `URL_TO_MD_JOB_TTL_SECONDS`, `URL_TO_MD_TEMP_DIR`, `URL_TO_MD_CORS_ORIGINS` |
 
 Exact variable names match the `ApiSettings` / helper functions in each `api/settings` or `api/app` module.
 
@@ -461,6 +496,7 @@ Two usage patterns:
 | PDF (FastMCP) | `python -m md_generator.pdf.api.mcp_server` / `--transport stdio` / `sse` / `streamable-http` |
 | PPTX | `python -m md_generator.ppt.api.mcp_server` (see module docstring for flags) |
 | Image | `python -m md_generator.image.api.mcp_server` (see module for CLI) |
+| URL / HTML | `python -m md_generator.url.api.mcp_server` / `--transport sse` / `--transport streamable-http` |
 
 **Word** and **XLSX** also ship a small runner script in the repo:
 

{mdengine-0.1.0 → mdengine-0.2.2}/README.md

@@ -72,6 +72,8 @@ pip install "mdengine[all]"
 | `image-ocr` | Heavy OCR backends (pytesseract, paddle, easyocr, …) |
 | `text` | TXT / JSON / XML converter (stdlib-oriented; marker extra) |
 | `archive` | ZIP → Markdown layout (Pillow; optional tesseract for inline image OCR) |
+| `url` | HTTP(S) HTML → Markdown (httpx, readability-lxml, markdownify, BeautifulSoup, lxml) |
+| `url-full` | `url` plus PDF/Word/PPTX/XLSX/archive stack for **post-converting** downloaded linked files to Markdown |
 | `api` | FastAPI, uvicorn, httpx, pydantic-settings |
 | `mcp` | MCP servers (`mcp`, `fastmcp` where used) |
 | `dev` | pytest + API/MCP test helpers |
@@ -113,6 +115,7 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-image` | `md_generator.image.converter:main` | `md-image ./scans page.md` |
 | `md-text` | `md_generator.text.converter:main` | `md-text config.xml out.md` |
 | `md-zip` | `md_generator.archive.converter:main` | `md-zip bundle.zip ./zip-out` |
+| `md-url` | `md_generator.url.converter:main` | `md-url https://example.com/doc ./web-out --artifact-layout` |
 
 Every command accepts **`-h` / `--help`** for full flags (artifact layout, OCR, ZIP options, etc.).
 
@@ -129,6 +132,7 @@ md-xlsx -i export.csv -o ./csv-out
 md-image ./photos ocr.md --engines tess --strategy best
 md-text data.json data.md
 md-zip archive.zip ./unzipped-md
+md-url https://example.com/page ./page-bundle --artifact-layout
 ```
 
 **Windows PowerShell** (same commands; use backslashes for paths if you prefer)
@@ -136,6 +140,7 @@ md-zip archive.zip ./unzipped-md
 ```powershell
 md-pdf .\manual.pdf .\out\doc.md
 md-zip .\archive.zip .\zip-out
+md-url https://example.com/page .\page-bundle --artifact-layout
 ```
 
 **Windows CMD**
@@ -143,11 +148,12 @@ md-zip .\archive.zip .\zip-out
 ```cmd
 md-pdf manual.pdf out\doc.md
 md-zip archive.zip zip-out
+md-url https://example.com/page page-bundle --artifact-layout
 ```
 
 ### 5. Run without `pip install` (repo clone + `PYTHONPATH`)
 
-The folders `pdf-to-md/`, `word-to-md/`, … contain a thin `converter.py` that calls the same code as `md-pdf`, `md-word`, etc. From the **repository root**, point Python at `src` so `md_generator` imports, then run the shim:
+The folders `pdf-to-md/`, `word-to-md/`, `url-to-md/`, … contain a thin `converter.py` that calls the same code as `md-pdf`, `md-word`, etc. From the **repository root**, point Python at `src` so `md_generator` imports, then run the shim:
 
 **PowerShell**
 
@@ -308,12 +314,12 @@ convert_zip(
 
 All format APIs follow a similar pattern:
 
-- **`POST /convert/sync`** — upload a file; response is often a **ZIP** (artifact bundle) for larger formats.
+- **`POST /convert/sync`** — upload a file (most converters) **or** send JSON (`url-to-md`); response is often a **ZIP** (artifact bundle) for larger formats.
 - **`POST /convert/jobs`** — async job; returns `job_id`.
 - **`GET /convert/jobs/{job_id}`** — status.
 - **`GET /convert/jobs/{job_id}/download`** — download result when ready.
 
-Upload field name is **`file`** (multipart form). Use `httpx` or `curl -F "file=@path/to/file"`.
+Upload field name is **`file`** (multipart form) for file-based converters. Use `httpx` or `curl -F "file=@path/to/file"`. **URL** conversion uses a **JSON** body (`url` or `urls`); see [url-to-md/README.md](url-to-md/README.md).
 
 ### Run with Uvicorn
 
@@ -328,6 +334,7 @@ Install `mdengine[api]` plus the format extra(s), then run the **`app`** object
 | Image | `md_generator.image.api.main:app` | `image`, `api`, `mcp` |
 | Text/JSON/XML | `md_generator.text.api.main:app` | `text`, `api`, `mcp` |
 | ZIP | `md_generator.archive.api.main:app` | `archive`, `api`, `mcp` (+ extras for nested office/PDF) |
+| URL / HTML | `md_generator.url.api.main:app` | `url`, `api`, `mcp` |
 
 Examples:
 
@@ -335,6 +342,7 @@ Examples:
 uvicorn md_generator.pdf.api.main:app --host 127.0.0.1 --port 8001
 uvicorn md_generator.word.api.main:app --host 127.0.0.1 --port 8002
 uvicorn md_generator.archive.api.main:app --host 127.0.0.1 --port 8010
+uvicorn md_generator.url.api.main:app --host 127.0.0.1 --port 8011
 ```
 
 ### MCP over HTTP on the same server
@@ -353,6 +361,7 @@ Prefixes differ per service (often read from a `.env` file next to the process):
 | PPTX | `PPT_TO_MD_` | `PPT_TO_MD_MAX_UPLOAD_MB`, … |
 | Text | `TXT_JSON_XML_TO_MD_` | same pattern |
 | XLSX | `XLSX_TO_MD_` | `XLSX_TO_MD_TEMP_DIR`, `XLSX_TO_MD_CORS_ORIGINS`, etc. (see `md_generator.xlsx.api.app`) |
+| URL | `URL_TO_MD_` | `URL_TO_MD_MAX_SYNC_URLS`, `URL_TO_MD_MAX_SYNC_CRAWL_PAGES`, `URL_TO_MD_MAX_JOB_URLS`, `URL_TO_MD_JOB_TTL_SECONDS`, `URL_TO_MD_TEMP_DIR`, `URL_TO_MD_CORS_ORIGINS` |
 
 Exact variable names match the `ApiSettings` / helper functions in each `api/settings` or `api/app` module.
 
@@ -375,6 +384,7 @@ Two usage patterns:
 | PDF (FastMCP) | `python -m md_generator.pdf.api.mcp_server` / `--transport stdio` / `sse` / `streamable-http` |
 | PPTX | `python -m md_generator.ppt.api.mcp_server` (see module docstring for flags) |
 | Image | `python -m md_generator.image.api.mcp_server` (see module for CLI) |
+| URL / HTML | `python -m md_generator.url.api.mcp_server` / `--transport sse` / `--transport streamable-http` |
 
 **Word** and **XLSX** also ship a small runner script in the repo:
 

{mdengine-0.1.0 → mdengine-0.2.2}/pyproject.toml

@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mdengine"
-version = "0.1.0"
-description = "Convert PDF, Office, images, text/JSON/XML, and ZIP archives to Markdown."
+version = "0.2.2"
+description = "Convert PDF, Office, images, text/JSON/XML, ZIP archives, and web URLs to Markdown."
 readme = "README.md"
 requires-python = ">=3.10"
 license = "MIT"
 license-files = ["LICENSE"]
 authors = [{ name = "mdengine contributors" }]
-keywords = ["markdown", "pdf", "docx", "pptx", "xlsx", "ocr", "zip"]
+keywords = ["markdown", "pdf", "docx", "pptx", "xlsx", "ocr", "zip", "url", "html"]
 classifiers = [
   "Programming Language :: Python :: 3",
   "Programming Language :: Python :: 3 :: Only",
@@ -47,6 +47,31 @@ image-ocr = [
 ]
 text = []
 archive = ["Pillow>=10.0.0", "pytesseract>=0.3.10"]
+url = [
+  "httpx>=0.27.0",
+  "beautifulsoup4>=4.12.0",
+  "lxml>=5.0.0",
+  "lxml_html_clean>=0.4.0",
+  "markdownify>=0.14.0",
+  "readability-lxml>=0.8.1",
+]
+# Convenience extra for URL fetch + post-convert of downloaded office/PDF/ZIP/etc.
+url-full = [
+  "httpx>=0.27.0",
+  "beautifulsoup4>=4.12.0",
+  "lxml>=5.0.0",
+  "lxml_html_clean>=0.4.0",
+  "markdownify>=0.14.0",
+  "readability-lxml>=0.8.1",
+  "pymupdf>=1.24.0",
+  "pdfplumber>=0.11.0",
+  "mammoth>=1.8.0",
+  "python-pptx>=1.0.0",
+  "Pillow>=10.0.0",
+  "olefile>=0.47",
+  "openpyxl>=3.1.0",
+  "pytesseract>=0.3.10",
+]
 api = [
   "fastapi>=0.115.0",
   "uvicorn[standard]>=0.32.0",
@@ -55,7 +80,13 @@ api = [
   "pydantic-settings>=2.0.0",
 ]
 mcp = ["mcp>=1.2.0", "fastmcp>=2.3.0"]
-dev = ["pytest>=8.0.0", "httpx>=0.27.0", "mdengine[api]", "mdengine[mcp]"]
+dev = [
+  "pytest>=8.0.0",
+  "httpx>=0.27.0",
+  "mdengine[api]",
+  "mdengine[mcp]",
+  "mdengine[url]",
+]
 all = [
   "pymupdf>=1.24.0",
   "pdfplumber>=0.11.0",
@@ -78,6 +109,9 @@ all = [
   "pydantic-settings>=2.0.0",
   "mcp>=1.2.0",
   "fastmcp>=2.3.0",
+  "beautifulsoup4>=4.12.0",
+  "lxml_html_clean>=0.4.0",
+  "readability-lxml>=0.8.1",
 ]
 
 [project.urls]
@@ -93,6 +127,7 @@ md-xlsx = "md_generator.xlsx.converter:main"
 md-image = "md_generator.image.converter:main"
 md-text = "md_generator.text.converter:main"
 md-zip = "md_generator.archive.converter:main"
+md-url = "md_generator.url.converter:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]
@@ -107,5 +142,6 @@ testpaths = [
   "image-to-md/tests",
   "txt-json-xml-to-md/tests",
   "zip-to-md/tests",
+  "url-to-md/tests",
 ]
 markers = ["integration: optional dependency or slower checks"]

{mdengine-0.1.0 → mdengine-0.2.2}/src/md_generator/__init__.py

@@ -1,5 +1,5 @@
 """md_generator: PDF, Office, image, text, and ZIP to Markdown conversion."""
 
-__version__ = "0.1.0"
+__version__ = "0.2.2"
 
 __all__ = ["__version__"]

mdengine-0.2.2/src/md_generator/url/__init__.py

@@ -0,0 +1,24 @@
+"""Convert public HTTP(S) HTML pages to Markdown with downloaded assets."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from md_generator.url.options import ConvertOptions
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+__all__ = ["ConvertOptions", "convert_url", "convert_urls_from_list"]
+
+
+def __getattr__(name: str) -> object:
+    if name == "convert_url":
+        from md_generator.url.convert_impl import convert_url as v
+
+        return v
+    if name == "convert_urls_from_list":
+        from md_generator.url.convert_impl import convert_urls_from_list as v
+
+        return v
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

mdengine-0.2.2/src/md_generator/url/api/__init__.py

@@ -0,0 +1 @@
+# FastAPI + MCP for url-to-md

mdengine-0.2.2/src/md_generator/url/api/convert_runner.py

@@ -0,0 +1,34 @@
+from __future__ import annotations
+
+import io
+import zipfile
+from pathlib import Path
+
+from md_generator.url.convert_impl import convert_url_job
+from md_generator.url.options import ConvertOptions
+
+
+def zip_directory(root: Path) -> bytes:
+    buf = io.BytesIO()
+    with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
+        for p in sorted(root.rglob("*")):
+            if p.is_file():
+                arc = p.relative_to(root)
+                zf.write(p, arc.as_posix())
+    return buf.getvalue()
+
+
+def build_artifact_zip_bytes(
+    *,
+    url: str | None,
+    urls: list[str] | None,
+    options: ConvertOptions,
+) -> bytes:
+    """Run conversion into a temp dir; return ZIP bytes (document.md + assets/ + pages/ when applicable)."""
+    import tempfile
+
+    with tempfile.TemporaryDirectory() as td:
+        out = Path(td) / "artifact"
+        out.mkdir(parents=True, exist_ok=True)
+        convert_url_job(url, urls, out, options)
+        return zip_directory(out)

mdengine-0.2.2/src/md_generator/url/api/jobs.py

@@ -0,0 +1,92 @@
+from __future__ import annotations
+
+import shutil
+import threading
+import time
+import uuid
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Callable
+
+
+@dataclass
+class Job:
+    job_id: str
+    workspace: Path
+    status: str = "queued"
+    error: str | None = None
+    created_at: float = field(default_factory=time.time)
+    zip_path: Path | None = None
+
+
+class JobStore:
+    def __init__(self, base_temp: Path | None, ttl_seconds: int) -> None:
+        self._base = base_temp
+        self._ttl = ttl_seconds
+        self._jobs: dict[str, Job] = {}
+        self._lock = threading.Lock()
+        self._sweeper_started = False
+
+    def _root(self) -> Path:
+        import tempfile
+
+        if self._base:
+            p = Path(self._base)
+            p.mkdir(parents=True, exist_ok=True)
+            return p
+        return Path(tempfile.gettempdir()) / "url-to-md-jobs"
+
+    def start_sweeper(self) -> None:
+        if self._sweeper_started:
+            return
+        self._sweeper_started = True
+
+        def loop() -> None:
+            while True:
+                time.sleep(60)
+                self.sweep()
+
+        threading.Thread(target=loop, daemon=True).start()
+
+    def sweep(self) -> None:
+        now = time.time()
+        with self._lock:
+            dead: list[str] = []
+            for jid, job in self._jobs.items():
+                if job.status in ("done", "failed") and now - job.created_at > self._ttl:
+                    dead.append(jid)
+            for jid in dead:
+                job = self._jobs.pop(jid, None)
+                if job and job.workspace.exists():
+                    shutil.rmtree(job.workspace, ignore_errors=True)
+
+    def create_job(self) -> Job:
+        jid = str(uuid.uuid4())
+        ws = self._root() / jid
+        ws.mkdir(parents=True, exist_ok=True)
+        job = Job(job_id=jid, workspace=ws)
+        with self._lock:
+            self._jobs[jid] = job
+        return job
+
+    def get(self, job_id: str) -> Job | None:
+        with self._lock:
+            return self._jobs.get(job_id)
+
+    def run_async(self, job: Job, fn: Callable[[], None]) -> None:
+        def target() -> None:
+            try:
+                job.status = "running"
+                fn()
+                job.status = "done"
+            except Exception as e:
+                job.status = "failed"
+                job.error = str(e)
+
+        threading.Thread(target=target, daemon=True).start()
+
+    def remove_after_download(self, job: Job) -> None:
+        with self._lock:
+            self._jobs.pop(job.job_id, None)
+        if job.workspace.exists():
+            shutil.rmtree(job.workspace, ignore_errors=True)