mdengine 0.4.0__tar.gz → 0.5.0__tar.gz

{mdengine-0.4.0 → mdengine-0.5.0}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: mdengine
-Version: 0.4.0
+Version: 0.5.0
 Summary: Convert PDF, Office, images, audio, video, text/JSON/XML, ZIP archives, and web URLs to Markdown.
 Author: mdengine contributors
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/vishal7090/md-generator
 Project-URL: Repository, https://github.com/vishal7090/md-generator
 Project-URL: Issues, https://github.com/vishal7090/md-generator/issues
-Keywords: markdown,pdf,docx,pptx,xlsx,ocr,zip,url,html,youtube,playwright
+Keywords: markdown,pdf,docx,pptx,xlsx,ocr,zip,url,html,youtube,playwright,database
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: Topic :: Text Processing :: Markup :: Markdown
@@ -87,6 +87,18 @@ Requires-Dist: lxml>=5.0.0; extra == "playwright"
 Requires-Dist: markdownify>=0.14.0; extra == "playwright"
 Requires-Dist: readability-lxml>=0.8.1; extra == "playwright"
 Requires-Dist: Pillow>=10.0.0; extra == "playwright"
+Provides-Extra: db
+Requires-Dist: sqlalchemy>=2.0.0; extra == "db"
+Requires-Dist: pyyaml>=6.0.0; extra == "db"
+Requires-Dist: psycopg2-binary>=2.9.9; extra == "db"
+Requires-Dist: pymysql>=1.1.0; extra == "db"
+Requires-Dist: oracledb>=2.0.0; extra == "db"
+Requires-Dist: pymongo>=4.6.0; extra == "db"
+Requires-Dist: mermaid-py>=0.8.0; extra == "db"
+Provides-Extra: graph
+Requires-Dist: networkx>=3.2.0; extra == "graph"
+Requires-Dist: neo4j>=5.14.0; extra == "graph"
+Requires-Dist: pyyaml>=6.0.0; extra == "graph"
 Provides-Extra: api
 Requires-Dist: fastapi>=0.115.0; extra == "api"
 Requires-Dist: uvicorn[standard]>=0.32.0; extra == "api"
@@ -106,6 +118,8 @@ Requires-Dist: mdengine[api]; extra == "dev"
 Requires-Dist: mdengine[mcp]; extra == "dev"
 Requires-Dist: mdengine[url]; extra == "dev"
 Requires-Dist: mdengine[playwright]; extra == "dev"
+Requires-Dist: mdengine[db]; extra == "dev"
+Requires-Dist: mdengine[graph]; extra == "dev"
 Provides-Extra: all
 Requires-Dist: pymupdf>=1.24.0; extra == "all"
 Requires-Dist: pdfplumber>=0.11.0; extra == "all"
@@ -136,7 +150,7 @@ Dynamic: license-file
 
 # mdengine
 
-Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, and **audio / video** (Whisper transcription → Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
+Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), and **graphs** (Neo4j / NetworkX → Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
 
 - **PyPI name:** `mdengine` (import package: `md_generator`)
 - **Source:** [github.com/vishal7090/md-generator](https://github.com/vishal7090/md-generator)
@@ -214,6 +228,7 @@ pip install "mdengine[all]"
 | `video` | **Video → Markdown** (ffmpeg extracts mono 16 kHz WAV, then same Whisper stack as `audio`) |
 | `api` | FastAPI, uvicorn, httpx, pydantic-settings |
 | `mcp` | MCP servers (`mcp`, `fastmcp` where used) |
+| `graph` | **Graph → Markdown** (Neo4j Bolt + NetworkX GraphML/GML): `networkx`, `neo4j`, `pyyaml` |
 | `dev` | pytest + API/MCP test helpers |
 | `all` | Large superset of dependencies (use only if you need everything) |
 
@@ -228,7 +243,7 @@ All converters can be run from a terminal after you install the package (with th
 ### 1. Install (once)
 
 ```bash
-pip install "mdengine[pdf,word]"          # adjust extras: ppt, xlsx, image, archive, text, …
+pip install "mdengine[pdf,word]"          # adjust extras: ppt, xlsx, image, archive, text, db, graph, …
 # or from a clone:
 pip install -e ".[pdf,word,archive]"
 ```
@@ -263,6 +278,19 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-audio-mcp` | `md_generator.media.audio.api.mcp_server:main` | Standalone MCP (`--transport stdio` \| `sse` \| `streamable-http`) |
 | `md-video-mcp` | `md_generator.media.video.api.mcp_server:main` | Same for video |
 | `md-youtube-mcp` | `md_generator.media.youtube.api.mcp_server:main` | Same for YouTube (`youtube_url_to_markdown`) |
+| `md-db` | `md_generator.db.cli.main:main` | `pip install "mdengine[db]"` then `md-db --config db.yaml` (or `mdengine db-to-md …`) |
+| `md-db-api` | `md_generator.db.api.run:main` | FastAPI on port **8010** (`DB_TO_MD_PORT`): `POST /db-to-md/run`, `/db-to-md/job`, SSE `/db-to-md/job/{id}/events` |
+| `md-db-mcp` | `md_generator.db.api.mcp_server:main` | Standalone MCP for metadata export tools |
+| `md-graph` | `md_generator.graph.cli.main:main` | `pip install "mdengine[graph]"` then `md-graph --source neo4j --uri bolt://…` (or `mdengine graph-to-md …`) |
+| `md-graph-api` | `md_generator.graph.api.run:main` | FastAPI on port **8012** (`GRAPH_TO_MD_PORT`): `POST /graph-to-md/run`, `/graph-to-md/job`, SSE `/graph-to-md/job/{id}/events` |
+| `md-graph-mcp` | `md_generator.graph.api.mcp_server:main` | Standalone MCP for graph export tools |
+| `mdengine` | `md_generator.engine_cli:main` | `mdengine db-to-md …` / `mdengine graph-to-md …` (delegates to `md-db` / `md-graph`) |
+
+**db-to-md ER diagrams:** add **`erd`** to the export feature list (YAML `features.include`, API body, or CLI `--include …,erd`). **Preferred:** **Graphviz** (`dot` on `PATH`, or **`GRAPHVIZ_DOT`**) produces `erd/*.dot`, `erd/*.png`, and `erd/*.svg`. **If Graphviz is missing,** the exporter falls back to **Mermaid** (`erDiagram`): it writes `erd/*.mermaid` plus a fenced **`erd/*.md`** for GitHub-style preview. With **`mermaid-py`** (included in `mdengine[db]`), it also requests **PNG/SVG** via mermaid.ink (requires network unless you self-host mermaid.ink and set **`MERMAID_INK_SERVER`** per [mermaid-py](https://pypi.org/project/mermaid-py/)). Tune **`erd.max_tables`** (default 100) and **`erd.scope`** (`full` \| `per_schema` \| `per_table`) under `erd:` in YAML; CLI: `--erd-max-tables`, `--erd-scope`. Async job SSE uses `progress_update` with `current` starting with `erd:`.
+
+**db-to-md split exports and README merge:** with **`output.split_files: true`**, set **`output.write_combined_feature_markdown: true`** to also write root-level combined Markdown (for example `tables.md`, `functions.md`, `indexes.md`, and feature-specific paths such as `oracle/packages.md` or `mongodb/collections.md` when those features run). Set **`output.readme_feature_merge`** to **`inline`** (append full bundle bodies into `README.md`) or **`toc`** (append a linked list to those files). If merge is not **`none`** and split files are on, combined bundle writes are turned on automatically when loading config. CLI: `--write-combined-feature-markdown`, `--readme-feature-merge none|inline|toc`.
+
+**graph-to-md (Neo4j + NetworkX):** library lives under [`md_generator/graph/`](src/md_generator/graph/). **Sources:** **`networkx`** (GraphML/GML via `graph.graph_file` or `--graph-file`) or **`neo4j`** (`graph.uri`, `graph.user`, `graph.password`, optional `graph.database` for `session(database=…)`). **Output layout:** by default **`output.combine_markdown: true`** writes **`nodes.md`**, **`relationship.md`**, and **`graph_summary.md`** (summary plus embedded nodes and relationships). Set **`combine_markdown: false`** or CLI **`--individual` / `--markdown-layout individual`** for per-entity files under **`nodes/`** and **`relationships/`**. **Diagrams:** **`viz.mermaid: true`** (default) writes **`graph/graph.mmd`** and embeds a fenced Mermaid block in the export **`README.md`** (no Graphviz required). **`viz.enabled: true`** or CLI **`--viz`** also writes **`graph/graph.dot`** and runs **`dot`** for PNG/SVG/PDF when Graphviz is on `PATH` (or **`GRAPHVIZ_DOT`**). CLI **`--no-mermaid`** disables Mermaid; **`--depth`**, **`--start-node`**, **`--max-nodes`**, **`--max-edges`** bound traversal. Packaged defaults: [`src/md_generator/graph/config/default.yaml`](src/md_generator/graph/config/default.yaml). Tests: **`graph-to-md/tests/`**; API image: [`graph-to-md/Dockerfile.api`](graph-to-md/Dockerfile.api).
 
 Every command accepts **`-h` / `--help`** for full flags (artifact layout, OCR, ZIP options, etc.).
 
@@ -282,6 +310,7 @@ md-zip archive.zip ./unzipped-md
 md-url https://example.com/page ./page-bundle --artifact-layout
 md-audio ./voice.mp3 ./voice.md --model tiny
 md-video ./screen.mp4 ./screen.md --model base
+pip install "mdengine[graph]" && md-graph --source neo4j --uri neo4j://localhost:7687 --user neo4j --password secret --database neo4j --output ./graph-out --viz
 ```
 
 **Windows PowerShell** (same commands; use backslashes for paths if you prefer)
@@ -292,6 +321,8 @@ md-zip .\archive.zip .\zip-out
 md-url https://example.com/page .\page-bundle --artifact-layout
 md-audio .\voice.mp3 .\voice.md --model tiny
 md-video .\screen.mp4 .\screen.md --model base
+pip install "mdengine[graph]"
+md-graph --source neo4j --uri neo4j://localhost:7687 --user neo4j --password secret --database neo4j --output .\graph-out --viz
 ```
 
 **Windows CMD**
@@ -575,7 +606,7 @@ Equivalent modules: `python -m md_generator.media.audio.api.mcp_server`, `python
 
 ### Thin shims (repo clone)
 
-[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`.
+[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`. [`db-to-md/converter.py`](db-to-md/converter.py) delegates to `md-db`; tests live under `db-to-md/tests/`. **graph-to-md** uses `md-graph` / `mdengine graph-to-md` directly (no thin `converter.py` shim); tests live under [`graph-to-md/tests/`](graph-to-md/tests/).
 
 ---
 
@@ -605,6 +636,8 @@ Install `mdengine[api]` plus the format extra(s), then run the **`app`** object
 | ZIP | `md_generator.archive.api.main:app` | `archive`, `api`, `mcp` (+ extras for nested office/PDF) |
 | URL / HTML | `md_generator.url.api.main:app` | `url`, `api`, `mcp` |
 | Playwright / SPA | `md_generator.playwright.api.main:app` | `playwright`, `api`, `mcp` |
+| Database metadata | `md_generator.db.api.main:app` | `db`, `api`, `mcp` |
+| Graph metadata (Neo4j / NetworkX) | `md_generator.graph.api.main:app` | `graph`, `api`, `mcp` |
 | Audio (Whisper) | `md_generator.media.audio.api.main:create_app` (use **`--factory`**) or `…main:app` | `audio`, `api`, `mcp` |
 | Video (Whisper) | `md_generator.media.video.api.main:create_app` (use **`--factory`**) or `…main:app` | `video`, `api`, `mcp` |
 | YouTube | `md_generator.media.youtube.api.main:create_app` (use **`--factory`**) or `…main:app` | `youtube`, `api`, `mcp` |
@@ -617,11 +650,14 @@ uvicorn md_generator.word.api.main:app --host 127.0.0.1 --port 8002
 uvicorn md_generator.archive.api.main:app --host 127.0.0.1 --port 8010
 uvicorn md_generator.url.api.main:app --host 127.0.0.1 --port 8011
 uvicorn md_generator.playwright.api.main:app --host 127.0.0.1 --port 8014
+uvicorn md_generator.graph.api.main:app --host 127.0.0.1 --port 8012
 uvicorn md_generator.media.audio.api.main:create_app --factory --host 127.0.0.1 --port 8011
 uvicorn md_generator.media.video.api.main:create_app --factory --host 127.0.0.1 --port 8012
 uvicorn md_generator.media.youtube.api.main:create_app --factory --host 127.0.0.1 --port 8013
 ```
 
+**Port note:** **`md-graph-api`** and **`md-video-api`** both default to **8012**; set **`GRAPH_TO_MD_PORT`** or **`MD_VIDEO_API_PORT`** when you need both on one machine.
+
 ### MCP over HTTP on the same server
 
 These apps mount an MCP HTTP app at **`/mcp`** (Streamable HTTP / framework-specific). Start the API as above, then point an MCP client at `http://<host>:<port>/mcp` where supported.
@@ -640,6 +676,8 @@ Prefixes differ per service (often read from a `.env` file next to the process):
 | XLSX | `XLSX_TO_MD_` | `XLSX_TO_MD_TEMP_DIR`, `XLSX_TO_MD_CORS_ORIGINS`, etc. (see `md_generator.xlsx.api.app`) |
 | URL | `URL_TO_MD_` | `URL_TO_MD_MAX_SYNC_URLS`, `URL_TO_MD_MAX_SYNC_CRAWL_PAGES`, `URL_TO_MD_MAX_JOB_URLS`, `URL_TO_MD_JOB_TTL_SECONDS`, `URL_TO_MD_TEMP_DIR`, `URL_TO_MD_CORS_ORIGINS` |
 | Playwright / SPA | `PLAYWRIGHT_TO_MD_` | `PLAYWRIGHT_TO_MD_MAX_SYNC_URLS`, `PLAYWRIGHT_TO_MD_MAX_JOB_URLS`, `PLAYWRIGHT_TO_MD_JOB_TTL_SECONDS`, `PLAYWRIGHT_TO_MD_TEMP_DIR`, `PLAYWRIGHT_TO_MD_CORS_ORIGINS`, `PLAYWRIGHT_TO_MD_API_HOST`, `PLAYWRIGHT_TO_MD_API_PORT` (default **8014**) |
+| Database metadata | `DB_TO_MD_` | `DB_TO_MD_JOB_SQLITE_PATH`, `DB_TO_MD_JOB_WORKSPACE_ROOT`, `DB_TO_MD_CORS_ORIGINS`, `DB_TO_MD_MAX_SYNC_ZIP_MB`, `DB_TO_MD_HOST`, `DB_TO_MD_PORT` (default **8010**) |
+| Graph metadata | `GRAPH_TO_MD_` | `GRAPH_TO_MD_JOB_SQLITE_PATH`, `GRAPH_TO_MD_JOB_WORKSPACE_ROOT`, `GRAPH_TO_MD_CORS_ORIGINS`, `GRAPH_TO_MD_MAX_SYNC_ZIP_MB`, `GRAPH_TO_MD_HOST`, `GRAPH_TO_MD_PORT` (default **8012**) |
 | Audio API | `MD_AUDIO_` | `MD_AUDIO_MAX_UPLOAD_MB`, `MD_AUDIO_MAX_SYNC_UPLOAD_MB`, `MD_AUDIO_JOB_TTL_SECONDS`, `MD_AUDIO_TEMP_DIR`, `MD_AUDIO_CORS_ORIGINS`, `MD_AUDIO_API_HOST`, `MD_AUDIO_API_PORT` |
 | Video API | `MD_VIDEO_` | Same pattern as audio with `MD_VIDEO_*` (defaults: larger upload/sync caps, port **8012**) |
 | YouTube API | `MD_YOUTUBE_` | `MD_YOUTUBE_JOB_TTL_SECONDS`, `MD_YOUTUBE_TEMP_DIR`, `MD_YOUTUBE_CORS_ORIGINS`, `MD_YOUTUBE_API_HOST`, `MD_YOUTUBE_API_PORT` (default **8013**); optional `MD_YOUTUBE_YTDLP` path for audio fallback |
@@ -670,6 +708,8 @@ Two usage patterns:
 | Audio | `md-audio-mcp` or `python -m md_generator.media.audio.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
 | Video | `md-video-mcp` or `python -m md_generator.media.video.api.mcp_server` — same transports |
 | YouTube | `md-youtube-mcp` or `python -m md_generator.media.youtube.api.mcp_server` — same transports |
+| Database metadata | `md-db-mcp` or `python -m md_generator.db.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
+| Graph (Neo4j / NetworkX) | `md-graph-mcp` or `python -m md_generator.graph.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
 
 **Word** and **XLSX** also ship a small runner script in the repo:
 
@@ -694,7 +734,7 @@ pip install -e ".[dev,all]"   # or a smaller subset of extras
 python -m pytest
 ```
 
-Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`); `pyproject.toml` configures `pythonpath = ["src"]` so `md_generator` resolves without a separate `PYTHONPATH`.
+Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`) and **`graph-to-md/tests/`**; `pyproject.toml` configures `pythonpath = ["src"]` so `md_generator` resolves without a separate `PYTHONPATH`.
 
 ---
 
@@ -704,7 +744,7 @@ Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/test
 |------|------|
 | `LICENSE` | MIT license text |
 | `CODE_OF_CONDUCT.md` | [Contributor Covenant](https://www.contributor-covenant.org/) 2.1 |
-| `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/audio/`](src/md_generator/media/audio/) and [`media/video/`](src/md_generator/media/video/) |
+| `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/audio/`](src/md_generator/media/audio/) and [`media/video/`](src/md_generator/media/video/); **graph-to-md** under [`graph/`](src/md_generator/graph/) |
 | `pyproject.toml` | Packaging, extras, CLI entry points, pytest |
 | `*-to-md/` | **Docs, tests, fixtures**, thin `converter.py` shims, some `run.py` helpers |
 | `README.md` | This document |

{mdengine-0.4.0 → mdengine-0.5.0}/README.md

@@ -1,6 +1,6 @@
 # mdengine
 
-Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, and **audio / video** (Whisper transcription → Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
+Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), and **graphs** (Neo4j / NetworkX → Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
 
 - **PyPI name:** `mdengine` (import package: `md_generator`)
 - **Source:** [github.com/vishal7090/md-generator](https://github.com/vishal7090/md-generator)
@@ -78,6 +78,7 @@ pip install "mdengine[all]"
 | `video` | **Video → Markdown** (ffmpeg extracts mono 16 kHz WAV, then same Whisper stack as `audio`) |
 | `api` | FastAPI, uvicorn, httpx, pydantic-settings |
 | `mcp` | MCP servers (`mcp`, `fastmcp` where used) |
+| `graph` | **Graph → Markdown** (Neo4j Bolt + NetworkX GraphML/GML): `networkx`, `neo4j`, `pyyaml` |
 | `dev` | pytest + API/MCP test helpers |
 | `all` | Large superset of dependencies (use only if you need everything) |
 
@@ -92,7 +93,7 @@ All converters can be run from a terminal after you install the package (with th
 ### 1. Install (once)
 
 ```bash
-pip install "mdengine[pdf,word]"          # adjust extras: ppt, xlsx, image, archive, text, …
+pip install "mdengine[pdf,word]"          # adjust extras: ppt, xlsx, image, archive, text, db, graph, …
 # or from a clone:
 pip install -e ".[pdf,word,archive]"
 ```
@@ -127,6 +128,19 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-audio-mcp` | `md_generator.media.audio.api.mcp_server:main` | Standalone MCP (`--transport stdio` \| `sse` \| `streamable-http`) |
 | `md-video-mcp` | `md_generator.media.video.api.mcp_server:main` | Same for video |
 | `md-youtube-mcp` | `md_generator.media.youtube.api.mcp_server:main` | Same for YouTube (`youtube_url_to_markdown`) |
+| `md-db` | `md_generator.db.cli.main:main` | `pip install "mdengine[db]"` then `md-db --config db.yaml` (or `mdengine db-to-md …`) |
+| `md-db-api` | `md_generator.db.api.run:main` | FastAPI on port **8010** (`DB_TO_MD_PORT`): `POST /db-to-md/run`, `/db-to-md/job`, SSE `/db-to-md/job/{id}/events` |
+| `md-db-mcp` | `md_generator.db.api.mcp_server:main` | Standalone MCP for metadata export tools |
+| `md-graph` | `md_generator.graph.cli.main:main` | `pip install "mdengine[graph]"` then `md-graph --source neo4j --uri bolt://…` (or `mdengine graph-to-md …`) |
+| `md-graph-api` | `md_generator.graph.api.run:main` | FastAPI on port **8012** (`GRAPH_TO_MD_PORT`): `POST /graph-to-md/run`, `/graph-to-md/job`, SSE `/graph-to-md/job/{id}/events` |
+| `md-graph-mcp` | `md_generator.graph.api.mcp_server:main` | Standalone MCP for graph export tools |
+| `mdengine` | `md_generator.engine_cli:main` | `mdengine db-to-md …` / `mdengine graph-to-md …` (delegates to `md-db` / `md-graph`) |
+
+**db-to-md ER diagrams:** add **`erd`** to the export feature list (YAML `features.include`, API body, or CLI `--include …,erd`). **Preferred:** **Graphviz** (`dot` on `PATH`, or **`GRAPHVIZ_DOT`**) produces `erd/*.dot`, `erd/*.png`, and `erd/*.svg`. **If Graphviz is missing,** the exporter falls back to **Mermaid** (`erDiagram`): it writes `erd/*.mermaid` plus a fenced **`erd/*.md`** for GitHub-style preview. With **`mermaid-py`** (included in `mdengine[db]`), it also requests **PNG/SVG** via mermaid.ink (requires network unless you self-host mermaid.ink and set **`MERMAID_INK_SERVER`** per [mermaid-py](https://pypi.org/project/mermaid-py/)). Tune **`erd.max_tables`** (default 100) and **`erd.scope`** (`full` \| `per_schema` \| `per_table`) under `erd:` in YAML; CLI: `--erd-max-tables`, `--erd-scope`. Async job SSE uses `progress_update` with `current` starting with `erd:`.
+
+**db-to-md split exports and README merge:** with **`output.split_files: true`**, set **`output.write_combined_feature_markdown: true`** to also write root-level combined Markdown (for example `tables.md`, `functions.md`, `indexes.md`, and feature-specific paths such as `oracle/packages.md` or `mongodb/collections.md` when those features run). Set **`output.readme_feature_merge`** to **`inline`** (append full bundle bodies into `README.md`) or **`toc`** (append a linked list to those files). If merge is not **`none`** and split files are on, combined bundle writes are turned on automatically when loading config. CLI: `--write-combined-feature-markdown`, `--readme-feature-merge none|inline|toc`.
+
+**graph-to-md (Neo4j + NetworkX):** library lives under [`md_generator/graph/`](src/md_generator/graph/). **Sources:** **`networkx`** (GraphML/GML via `graph.graph_file` or `--graph-file`) or **`neo4j`** (`graph.uri`, `graph.user`, `graph.password`, optional `graph.database` for `session(database=…)`). **Output layout:** by default **`output.combine_markdown: true`** writes **`nodes.md`**, **`relationship.md`**, and **`graph_summary.md`** (summary plus embedded nodes and relationships). Set **`combine_markdown: false`** or CLI **`--individual` / `--markdown-layout individual`** for per-entity files under **`nodes/`** and **`relationships/`**. **Diagrams:** **`viz.mermaid: true`** (default) writes **`graph/graph.mmd`** and embeds a fenced Mermaid block in the export **`README.md`** (no Graphviz required). **`viz.enabled: true`** or CLI **`--viz`** also writes **`graph/graph.dot`** and runs **`dot`** for PNG/SVG/PDF when Graphviz is on `PATH` (or **`GRAPHVIZ_DOT`**). CLI **`--no-mermaid`** disables Mermaid; **`--depth`**, **`--start-node`**, **`--max-nodes`**, **`--max-edges`** bound traversal. Packaged defaults: [`src/md_generator/graph/config/default.yaml`](src/md_generator/graph/config/default.yaml). Tests: **`graph-to-md/tests/`**; API image: [`graph-to-md/Dockerfile.api`](graph-to-md/Dockerfile.api).
 
 Every command accepts **`-h` / `--help`** for full flags (artifact layout, OCR, ZIP options, etc.).
 
@@ -146,6 +160,7 @@ md-zip archive.zip ./unzipped-md
 md-url https://example.com/page ./page-bundle --artifact-layout
 md-audio ./voice.mp3 ./voice.md --model tiny
 md-video ./screen.mp4 ./screen.md --model base
+pip install "mdengine[graph]" && md-graph --source neo4j --uri neo4j://localhost:7687 --user neo4j --password secret --database neo4j --output ./graph-out --viz
 ```
 
 **Windows PowerShell** (same commands; use backslashes for paths if you prefer)
@@ -156,6 +171,8 @@ md-zip .\archive.zip .\zip-out
 md-url https://example.com/page .\page-bundle --artifact-layout
 md-audio .\voice.mp3 .\voice.md --model tiny
 md-video .\screen.mp4 .\screen.md --model base
+pip install "mdengine[graph]"
+md-graph --source neo4j --uri neo4j://localhost:7687 --user neo4j --password secret --database neo4j --output .\graph-out --viz
 ```
 
 **Windows CMD**
@@ -439,7 +456,7 @@ Equivalent modules: `python -m md_generator.media.audio.api.mcp_server`, `python
 
 ### Thin shims (repo clone)
 
-[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`.
+[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`. [`db-to-md/converter.py`](db-to-md/converter.py) delegates to `md-db`; tests live under `db-to-md/tests/`. **graph-to-md** uses `md-graph` / `mdengine graph-to-md` directly (no thin `converter.py` shim); tests live under [`graph-to-md/tests/`](graph-to-md/tests/).
 
 ---
 
@@ -469,6 +486,8 @@ Install `mdengine[api]` plus the format extra(s), then run the **`app`** object
 | ZIP | `md_generator.archive.api.main:app` | `archive`, `api`, `mcp` (+ extras for nested office/PDF) |
 | URL / HTML | `md_generator.url.api.main:app` | `url`, `api`, `mcp` |
 | Playwright / SPA | `md_generator.playwright.api.main:app` | `playwright`, `api`, `mcp` |
+| Database metadata | `md_generator.db.api.main:app` | `db`, `api`, `mcp` |
+| Graph metadata (Neo4j / NetworkX) | `md_generator.graph.api.main:app` | `graph`, `api`, `mcp` |
 | Audio (Whisper) | `md_generator.media.audio.api.main:create_app` (use **`--factory`**) or `…main:app` | `audio`, `api`, `mcp` |
 | Video (Whisper) | `md_generator.media.video.api.main:create_app` (use **`--factory`**) or `…main:app` | `video`, `api`, `mcp` |
 | YouTube | `md_generator.media.youtube.api.main:create_app` (use **`--factory`**) or `…main:app` | `youtube`, `api`, `mcp` |
@@ -481,11 +500,14 @@ uvicorn md_generator.word.api.main:app --host 127.0.0.1 --port 8002
 uvicorn md_generator.archive.api.main:app --host 127.0.0.1 --port 8010
 uvicorn md_generator.url.api.main:app --host 127.0.0.1 --port 8011
 uvicorn md_generator.playwright.api.main:app --host 127.0.0.1 --port 8014
+uvicorn md_generator.graph.api.main:app --host 127.0.0.1 --port 8012
 uvicorn md_generator.media.audio.api.main:create_app --factory --host 127.0.0.1 --port 8011
 uvicorn md_generator.media.video.api.main:create_app --factory --host 127.0.0.1 --port 8012
 uvicorn md_generator.media.youtube.api.main:create_app --factory --host 127.0.0.1 --port 8013
 ```
 
+**Port note:** **`md-graph-api`** and **`md-video-api`** both default to **8012**; set **`GRAPH_TO_MD_PORT`** or **`MD_VIDEO_API_PORT`** when you need both on one machine.
+
 ### MCP over HTTP on the same server
 
 These apps mount an MCP HTTP app at **`/mcp`** (Streamable HTTP / framework-specific). Start the API as above, then point an MCP client at `http://<host>:<port>/mcp` where supported.
@@ -504,6 +526,8 @@ Prefixes differ per service (often read from a `.env` file next to the process):
 | XLSX | `XLSX_TO_MD_` | `XLSX_TO_MD_TEMP_DIR`, `XLSX_TO_MD_CORS_ORIGINS`, etc. (see `md_generator.xlsx.api.app`) |
 | URL | `URL_TO_MD_` | `URL_TO_MD_MAX_SYNC_URLS`, `URL_TO_MD_MAX_SYNC_CRAWL_PAGES`, `URL_TO_MD_MAX_JOB_URLS`, `URL_TO_MD_JOB_TTL_SECONDS`, `URL_TO_MD_TEMP_DIR`, `URL_TO_MD_CORS_ORIGINS` |
 | Playwright / SPA | `PLAYWRIGHT_TO_MD_` | `PLAYWRIGHT_TO_MD_MAX_SYNC_URLS`, `PLAYWRIGHT_TO_MD_MAX_JOB_URLS`, `PLAYWRIGHT_TO_MD_JOB_TTL_SECONDS`, `PLAYWRIGHT_TO_MD_TEMP_DIR`, `PLAYWRIGHT_TO_MD_CORS_ORIGINS`, `PLAYWRIGHT_TO_MD_API_HOST`, `PLAYWRIGHT_TO_MD_API_PORT` (default **8014**) |
+| Database metadata | `DB_TO_MD_` | `DB_TO_MD_JOB_SQLITE_PATH`, `DB_TO_MD_JOB_WORKSPACE_ROOT`, `DB_TO_MD_CORS_ORIGINS`, `DB_TO_MD_MAX_SYNC_ZIP_MB`, `DB_TO_MD_HOST`, `DB_TO_MD_PORT` (default **8010**) |
+| Graph metadata | `GRAPH_TO_MD_` | `GRAPH_TO_MD_JOB_SQLITE_PATH`, `GRAPH_TO_MD_JOB_WORKSPACE_ROOT`, `GRAPH_TO_MD_CORS_ORIGINS`, `GRAPH_TO_MD_MAX_SYNC_ZIP_MB`, `GRAPH_TO_MD_HOST`, `GRAPH_TO_MD_PORT` (default **8012**) |
 | Audio API | `MD_AUDIO_` | `MD_AUDIO_MAX_UPLOAD_MB`, `MD_AUDIO_MAX_SYNC_UPLOAD_MB`, `MD_AUDIO_JOB_TTL_SECONDS`, `MD_AUDIO_TEMP_DIR`, `MD_AUDIO_CORS_ORIGINS`, `MD_AUDIO_API_HOST`, `MD_AUDIO_API_PORT` |
 | Video API | `MD_VIDEO_` | Same pattern as audio with `MD_VIDEO_*` (defaults: larger upload/sync caps, port **8012**) |
 | YouTube API | `MD_YOUTUBE_` | `MD_YOUTUBE_JOB_TTL_SECONDS`, `MD_YOUTUBE_TEMP_DIR`, `MD_YOUTUBE_CORS_ORIGINS`, `MD_YOUTUBE_API_HOST`, `MD_YOUTUBE_API_PORT` (default **8013**); optional `MD_YOUTUBE_YTDLP` path for audio fallback |
@@ -534,6 +558,8 @@ Two usage patterns:
 | Audio | `md-audio-mcp` or `python -m md_generator.media.audio.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
 | Video | `md-video-mcp` or `python -m md_generator.media.video.api.mcp_server` — same transports |
 | YouTube | `md-youtube-mcp` or `python -m md_generator.media.youtube.api.mcp_server` — same transports |
+| Database metadata | `md-db-mcp` or `python -m md_generator.db.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
+| Graph (Neo4j / NetworkX) | `md-graph-mcp` or `python -m md_generator.graph.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
 
 **Word** and **XLSX** also ship a small runner script in the repo:
 
@@ -558,7 +584,7 @@ pip install -e ".[dev,all]"   # or a smaller subset of extras
 python -m pytest
 ```
 
-Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`); `pyproject.toml` configures `pythonpath = ["src"]` so `md_generator` resolves without a separate `PYTHONPATH`.
+Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`) and **`graph-to-md/tests/`**; `pyproject.toml` configures `pythonpath = ["src"]` so `md_generator` resolves without a separate `PYTHONPATH`.
 
 ---
 
@@ -568,7 +594,7 @@ Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/test
 |------|------|
 | `LICENSE` | MIT license text |
 | `CODE_OF_CONDUCT.md` | [Contributor Covenant](https://www.contributor-covenant.org/) 2.1 |
-| `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/audio/`](src/md_generator/media/audio/) and [`media/video/`](src/md_generator/media/video/) |
+| `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/audio/`](src/md_generator/media/audio/) and [`media/video/`](src/md_generator/media/video/); **graph-to-md** under [`graph/`](src/md_generator/graph/) |
 | `pyproject.toml` | Packaging, extras, CLI entry points, pytest |
 | `*-to-md/` | **Docs, tests, fixtures**, thin `converter.py` shims, some `run.py` helpers |
 | `README.md` | This document |

{mdengine-0.4.0 → mdengine-0.5.0}/pyproject.toml

@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mdengine"
-version = "0.4.0"
+version = "0.5.0"
 description = "Convert PDF, Office, images, audio, video, text/JSON/XML, ZIP archives, and web URLs to Markdown."
 readme = "README.md"
 requires-python = ">=3.10"
 license = "MIT"
 license-files = ["LICENSE"]
 authors = [{ name = "mdengine contributors" }]
-keywords = ["markdown", "pdf", "docx", "pptx", "xlsx", "ocr", "zip", "url", "html", "youtube", "playwright"]
+keywords = ["markdown", "pdf", "docx", "pptx", "xlsx", "ocr", "zip", "url", "html", "youtube", "playwright", "database"]
 classifiers = [
   "Programming Language :: Python :: 3",
   "Programming Language :: Python :: 3 :: Only",
@@ -92,6 +92,20 @@ playwright = [
   "readability-lxml>=0.8.1",
   "Pillow>=10.0.0",
 ]
+db = [
+  "sqlalchemy>=2.0.0",
+  "pyyaml>=6.0.0",
+  "psycopg2-binary>=2.9.9",
+  "pymysql>=1.1.0",
+  "oracledb>=2.0.0",
+  "pymongo>=4.6.0",
+  "mermaid-py>=0.8.0",
+]
+graph = [
+  "networkx>=3.2.0",
+  "neo4j>=5.14.0",
+  "pyyaml>=6.0.0",
+]
 api = [
   "fastapi>=0.115.0",
   "uvicorn[standard]>=0.32.0",
@@ -110,6 +124,8 @@ dev = [
   "mdengine[mcp]",
   "mdengine[url]",
   "mdengine[playwright]",
+  "mdengine[db]",
+  "mdengine[graph]",
 ]
 all = [
   "pymupdf>=1.24.0",
@@ -165,10 +181,21 @@ md-youtube-mcp = "md_generator.media.youtube.api.mcp_server:main"
 md-playwright = "md_generator.playwright.cli:main"
 md-playwright-api = "md_generator.playwright.api.run:main"
 md-playwright-mcp = "md_generator.playwright.api.mcp_server:main"
+md-db = "md_generator.db.cli.main:main"
+md-db-api = "md_generator.db.api.run:main"
+md-db-mcp = "md_generator.db.api.mcp_server:main"
+md-graph = "md_generator.graph.cli.main:main"
+md-graph-api = "md_generator.graph.api.run:main"
+md-graph-mcp = "md_generator.graph.api.mcp_server:main"
+mdengine = "md_generator.engine_cli:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]
 
+[tool.setuptools.package-data]
+"md_generator.db.config" = ["*.yaml"]
+"md_generator.graph.config" = ["*.yaml"]
+
 [tool.pytest.ini_options]
 pythonpath = ["src"]
 testpaths = [
@@ -184,5 +211,7 @@ testpaths = [
   "video-to-md/tests",
   "youtube-to-md/tests",
   "playwright-to-md/tests",
+  "db-to-md/tests",
+  "graph-to-md/tests",
 ]
 markers = ["integration: optional dependency or slower checks"]

{mdengine-0.4.0 → mdengine-0.5.0}/src/md_generator/__init__.py

@@ -1,5 +1,5 @@
 """md_generator: PDF, Office, image, text, and ZIP to Markdown conversion."""
 
-__version__ = "0.4.0"
+__version__ = "0.5.0"
 
 __all__ = ["__version__"]

mdengine-0.5.0/src/md_generator/db/__init__.py

@@ -0,0 +1,5 @@
+"""Database metadata to Markdown (LLM-ready, deterministic)."""
+
+from md_generator.db.core.models import RunMetadata
+
+__all__ = ["RunMetadata"]

mdengine-0.5.0/src/md_generator/db/adapters/__init__.py

@@ -0,0 +1,3 @@
+from md_generator.db.adapters.factory import create_adapter
+
+__all__ = ["create_adapter"]

mdengine-0.5.0/src/md_generator/db/adapters/factory.py

@@ -0,0 +1,40 @@
+from __future__ import annotations
+
+from typing import Any
+
+from sqlalchemy.engine.url import make_url
+
+from md_generator.db.adapters.mongo_adapter import MongoAdapter
+from md_generator.db.adapters.mysql_adapter import MysqlAdapter
+from md_generator.db.adapters.oracle_adapter import OracleAdapter
+from md_generator.db.adapters.postgres_adapter import PostgresAdapter
+from md_generator.db.core.base_adapter import BaseAdapter
+
+
+def create_adapter(
+    db_type: str,
+    uri: str,
+    *,
+    schema: str | None = None,
+    database: str | None = None,
+    limits: dict[str, Any] | None = None,
+) -> BaseAdapter:
+    lim = dict(limits or {})
+    t = db_type.lower().strip()
+    if t in ("postgres", "postgresql"):
+        sch = schema or "public"
+        return PostgresAdapter(uri, sch, lim)
+    if t in ("mysql", "mariadb"):
+        sch = schema or make_url(uri).database
+        if not sch:
+            raise ValueError("MySQL requires database in URI path or --schema")
+        return MysqlAdapter(uri, sch, lim)
+    if t in ("oracle",):
+        if not schema:
+            raise ValueError("Oracle requires schema (owner) in config or --schema")
+        return OracleAdapter(uri, schema, lim)
+    if t in ("mongo", "mongodb"):
+        if not database:
+            raise ValueError("MongoDB requires database name in config or --database")
+        return MongoAdapter(uri, database, lim)
+    raise ValueError(f"Unsupported database type: {db_type!r}")

mdengine-0.5.0/src/md_generator/db/adapters/mongo_adapter.py

@@ -0,0 +1,142 @@
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any
+
+from md_generator.db.core.base_adapter import BaseAdapter
+from md_generator.db.core.models import MongoCollectionInfo, MongoIndexInfo
+
+
+def _bson_type_name(v: Any) -> str:
+    if v is None:
+        return "null"
+    t = type(v).__name__
+    if t == "ObjectId":
+        return "ObjectId"
+    if t in ("int", "float"):
+        return "number"
+    if t == "bool":
+        return "boolean"
+    if t == "str":
+        return "string"
+    if t == "bytes":
+        return "binary"
+    if t == "datetime":
+        return "datetime"
+    if t == "list":
+        return "array"
+    if t == "dict":
+        return "object"
+    return t
+
+
+def _merge_schema(a: Any, b: Any) -> Any:
+    """Deterministic merge of inferred type trees."""
+    if a is None:
+        return b
+    if b is None:
+        return a
+    if isinstance(a, str) and isinstance(b, str):
+        if a == b:
+            return a
+        return "|".join(sorted({a, b}))
+    if isinstance(a, dict) and isinstance(b, dict):
+        keys = sorted(set(a) | set(b))
+        return {k: _merge_schema(a.get(k), b.get(k)) for k in keys}
+    if isinstance(a, list) and isinstance(b, list):
+        merged: Any = None
+        for x in a + b:
+            merged = _merge_schema(merged, x) if merged is not None else x
+        return [merged] if merged is not None else []
+    if isinstance(a, list):
+        return _merge_schema(a[0] if a else None, b)
+    if isinstance(b, list):
+        return _merge_schema(a, b[0] if b else None)
+    return "|".join(sorted({_bson_type_name(a), _bson_type_name(b)}))
+
+
+def _infer_from_doc(doc: Mapping[str, Any]) -> Any:
+    out: dict[str, Any] = {}
+    for k in sorted(doc.keys()):
+        v = doc[k]
+        if isinstance(v, dict):
+            out[k] = _infer_from_doc(v)
+        elif isinstance(v, list):
+            elem: Any = None
+            for item in v:
+                if isinstance(item, dict):
+                    elem = _merge_schema(elem, _infer_from_doc(item))
+                else:
+                    elem = _merge_schema(elem, _bson_type_name(item))
+            out[k] = [elem] if elem is not None else ["null"]
+        else:
+            out[k] = _bson_type_name(v)
+    return out
+
+
+class MongoAdapter(BaseAdapter):
+    db_type = "mongo"
+
+    def __init__(self, uri: str, database: str, limits: dict[str, Any]) -> None:
+        from pymongo import MongoClient
+
+        self._client: Any = MongoClient(uri, serverSelectionTimeoutMS=8000)
+        self._db = self._client[database]
+        self._database_name = database
+        self._limits = limits
+
+    def limits(self) -> dict[str, Any]:
+        return dict(self._limits)
+
+    def validate_connection(self) -> None:
+        self._client.admin.command("ping")
+
+    def close(self) -> None:
+        self._client.close()
+
+    def get_collections(self) -> list[MongoCollectionInfo]:
+        max_c = int(self._limits.get("max_collections", 500))
+        sample_n = int(self._limits.get("sample_size", 50))
+        names = sorted(n for n in self._db.list_collection_names() if not n.startswith("system."))[:max_c]
+        out: list[MongoCollectionInfo] = []
+        for name in names:
+            coll = self._db[name]
+            schema: Any = None
+            count = 0
+            try:
+                for doc in coll.aggregate([{"$sample": {"size": sample_n}}]):
+                    if not isinstance(doc, dict):
+                        continue
+                    count += 1
+                    schema = _merge_schema(schema, _infer_from_doc(doc))
+            except Exception:
+                try:
+                    for doc in coll.find().limit(sample_n):
+                        if isinstance(doc, dict):
+                            count += 1
+                            schema = _merge_schema(schema, _infer_from_doc(doc))
+                except Exception:
+                    schema = {}
+            if schema is None:
+                schema = {}
+            idxes: list[MongoIndexInfo] = []
+            try:
+                for spec in coll.list_indexes():
+                    raw = dict(spec)
+                    iname = str(raw.get("name") or "")
+                    unique = bool(raw.get("unique", False))
+                    keys_raw = raw.get("key") or {}
+                    keys = {str(k): v for k, v in dict(keys_raw).items()}
+                    idxes.append(MongoIndexInfo(name=iname, keys=keys, unique=unique))
+            except Exception:
+                pass
+            idxes.sort(key=lambda x: x.name)
+            out.append(
+                MongoCollectionInfo(
+                    name=name,
+                    inferred_schema=schema if isinstance(schema, dict) else {"_inferred": schema},
+                    indexes=tuple(idxes),
+                    sample_size=count,
+                )
+            )
+        return out