mdengine 0.11.0__tar.gz → 0.12.0__tar.gz

{mdengine-0.11.0 → mdengine-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mdengine
-Version: 0.11.0
+Version: 0.12.0
 Summary: Convert PDF, Office, images, audio, video, text/JSON/XML, ZIP archives, web URLs, databases, graphs, and OpenAPI specs to Markdown.
 Author: mdengine contributors
 License-Expression: MIT
@@ -101,6 +101,7 @@ Requires-Dist: pymysql>=1.1.0; extra == "db"
 Requires-Dist: pyodbc>=5.0.0; extra == "db"
 Requires-Dist: oracledb>=2.0.0; extra == "db"
 Requires-Dist: pymongo>=4.6.0; extra == "db"
+Requires-Dist: elasticsearch>=8.0.0; extra == "db"
 Requires-Dist: mermaid-py>=0.8.0; extra == "db"
 Provides-Extra: graph
 Requires-Dist: networkx>=3.2.0; extra == "graph"
@@ -126,6 +127,18 @@ Requires-Dist: tree-sitter>=0.21.0; extra == "codeflow-treesitter"
 Requires-Dist: tree-sitter-javascript>=0.21.0; extra == "codeflow-treesitter"
 Requires-Dist: tree-sitter-typescript>=0.23.0; extra == "codeflow-treesitter"
 Requires-Dist: tree-sitter-cpp>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-java>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-python>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-go>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-php>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-rust>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-kotlin>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-c-sharp>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-swift>=0.7.2; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-ruby>=0.23.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-lua>=0.2.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-scala>=0.6.0; extra == "codeflow-treesitter"
+Requires-Dist: tree-sitter-zig>=0.2.0; extra == "codeflow-treesitter"
 Provides-Extra: codeflow-clang
 Requires-Dist: clang>=16.0.0; extra == "codeflow-clang"
 Provides-Extra: codeflow-semantic
@@ -223,7 +236,7 @@ Dynamic: license-file
 
 # mdengine
 
-Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), **graphs** (Neo4j / NetworkX → Markdown), **OpenAPI** specs, **Playwright**-captured web pages (including SPAs), and **source code** (codeflow → architecture Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
+Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), **graphs** (Neo4j / NetworkX → Markdown), **OpenAPI** specs, **Playwright**-captured web pages (including SPAs), **application logs** (plain, JSON, CSV, ZIP bundles → Markdown with optional clustering and semantic grouping), **OpenTelemetry** traces (**OTLP** JSON or protobuf → Markdown trace summaries; optional correlation with log exports via `otel_path`), **SAP artifacts** (ABAP, CDS, DDIC, OData, BAPI, IDoc, transport → AI-ready Markdown knowledge packs), and **source code** (codeflow → architecture Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
 
 - **PyPI name:** `mdengine` (import package: `md_generator`)
 - **Source:** [github.com/vishal7090/md-generator](https://github.com/vishal7090/md-generator)
@@ -305,13 +318,15 @@ pip install "mdengine[all]"
 | `openapi` | **OpenAPI / Swagger → Markdown** (`prance`, `openapi-spec-validator`, PyYAML) |
 | `codeflow` | **Static codeflow / call graphs → Markdown** (`networkx`, `javalang`; core scanners) |
 | `codeflow-worker` | Optional **Celery + Redis** workers for the codeflow API async path |
-| `codeflow-treesitter` | **Tree-sitter** parsers (JS/TS/TSX, C++) for richer codeflow slices |
+| `codeflow-treesitter` | **Tree-sitter** parsers (JS/TS/TSX, C++, Java, Python, Go, PHP); use `--parser-mode treesitter` — see [codeflow-to-md/docs/parser-backends.md](codeflow-to-md/docs/parser-backends.md) |
 | `codeflow-clang` | **libclang** bindings for C/C++ parsing when used by the codeflow pipeline |
 | `codeflow-semantic` | **Semantic clustering** (SentenceTransformers, scikit-learn, numpy; large install) |
 | `log` | **Log files → Markdown** (`pandas`, PyYAML, date parsing) |
 | `log-cluster` | Log pipeline **clustering** (scikit-learn) |
 | `log-semantic` | Log pipeline **semantic** helpers (SentenceTransformers + scikit-learn; large) |
 | `log-pretty` | Optional **loguru** for pretty console diagnostics during log conversion |
+| `log-otel-proto` | **OTLP protobuf** ingest for **`md-otel`** and log/OTEL tooling (`opentelemetry-proto`, `protobuf`) |
+| `sap` | **SAP artifacts → Markdown** (ABAP, CDS, DDIC, OData, BAPI, IDoc, transport): PyYAML, NetworkX, Pydantic |
 | `api` | FastAPI, uvicorn, httpx, pydantic-settings |
 | `mcp` | MCP servers (`mcp`, `fastmcp` where used) |
 | `graph` | **Graph → Markdown** (Neo4j Bolt + NetworkX GraphML/GML): `networkx`, `neo4j`, `pyyaml` |
@@ -386,7 +401,11 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-log` | `md_generator.log.cli.main:main` | `pip install "mdengine[log]"` then `md-log --config log.yaml` (or `mdengine log-to-md …`) |
 | `md-log-api` | `md_generator.log.api.run:main` | FastAPI on port **8012** (`LOG_TO_MD_PORT`); set a different port if **`md-graph-api`** or **`md-video-api`** already uses **8012** |
 | `md-log-mcp` | `md_generator.log.api.mcp_server:main` | Standalone MCP (`stdio` \| `sse` \| `streamable-http`) |
-| `mdengine` | `md_generator.engine_cli:main` | `mdengine ai assist …`, `mdengine ai export …`, `mdengine skill build …`, `mdengine db-to-md …`, `mdengine log-to-md …`, `mdengine graph-to-md …`, `mdengine openapi-to-md generate …`, `mdengine codeflow-to-md scan …` |
+| `md-otel` | `md_generator.otel.cli.main:main` | `pip install "mdengine[log-otel-proto]"` then `md-otel --input traces.json --output ./otel-docs` (or `mdengine otel-to-md …`; add **`--protobuf`** for OTLP binary) |
+| `md-sap` | `md_generator.sap.cli.main:main` | `pip install "mdengine[sap]"` then `md-sap ./sap-source --output ./sap-out --graph --chunk` (or `mdengine sap-to-md …`) |
+| `md-sap-api` | `md_generator.sap.api.run:main` | FastAPI on port **8020** (`SAP_TO_MD_PORT`): `POST /sap-to-md/run`, `POST /sap-to-md/job`, `GET /sap-to-md/job/{id}/download` |
+| `md-sap-mcp` | `md_generator.sap.api.mcp_server:main` | Runs the same FastAPI app as **`md-sap-api`** (MCP mount when enabled in app) |
+| `mdengine` | `md_generator.engine_cli:main` | `mdengine ai assist …`, `mdengine ai export …`, `mdengine skill build …`, `mdengine db-to-md …`, `mdengine log-to-md …`, `mdengine otel-to-md …`, `mdengine sap-to-md …`, `mdengine graph-to-md …`, `mdengine openapi-to-md generate …`, `mdengine codeflow-to-md scan …` |
 
 **openapi-to-md (`md-openapi`):** OpenAPI **3.x** is parsed directly; **Swagger 2.0** (`swagger: "2.0"`) documents are **converted in-process** to OpenAPI 3.0.3 (deterministic, in-repo converter) before `$ref` resolution. Edge-heavy specs (unusual OAuth2 flows, vendor extensions) may still need fixes after conversion.
 
@@ -406,6 +425,14 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 
 **db-to-md API SQLite upload:** **`POST /db-to-md/run/sqlite`** — `multipart/form-data` with field **`file`** (the `.sqlite` / `.db` bytes; must start with the standard `SQLite format 3` header) and optional **`config`** (JSON string with the same shape as a normal run body minus `database`: `schema`, `output`, `features`, `execution`, `limits`, `erd`). Returns the metadata ZIP immediately. **`POST /db-to-md/job/sqlite`** — same form fields; saves the file under the job workspace and runs the existing job pipeline (`GET /db-to-md/job/{id}/download` when complete). Upload size cap: env **`DB_TO_MD_MAX_SQLITE_UPLOAD_MB`** (default **256**). Large ZIPs may still require the async job path if they exceed **`DB_TO_MD_MAX_SYNC_ZIP_MB`** on the sync upload route.
 
+**db-to-md Elasticsearch / OpenSearch:** set **`database.type: elasticsearch`** and **`database.uri`** to the cluster URL (for example `https://localhost:9200`). Install **`mdengine[db]`** (includes the **`elasticsearch`** Python client). Feature flags: **`elasticsearch_indices`**, **`elasticsearch_data_streams`**, **`elasticsearch_component_templates`**, **`elasticsearch_index_templates`**, **`elasticsearch_ingest_pipelines`**, **`elasticsearch_ilm_policies`**, **`elasticsearch_slm_policies`**, **`elasticsearch_field_caps`**, **`elasticsearch_snapshot_repositories`**, **`elasticsearch_search_templates`**, **`elasticsearch_search_architecture`**, **`elasticsearch_search_dependency_graph`**. Output under **`elasticsearch/`** (indices, data streams, templates, pipelines, ILM, SLM, snapshots, search templates, **`alias_graph.md`**, **`search_architecture.md`**, **`search_dependency_graph.md`**). Index docs add **`## Mapping Summary`** (field counts, nested/vector/semantic totals) and **`## Operational Notes`** (health, replicas, vector-field signals). Search templates add **`## Query type`** using a vector-aware taxonomy (full text, aggregation, suggest, vector, semantic, hybrid). **`search_architecture.md`** narrates the cluster lifecycle (ingestion → indexing → retention → search → backup → alias routing) with aggregate mapping metrics, query-type counts, vector/AI readiness, and an **OpenSearch Compatibility** matrix when **`limits.opensearch: true`** or export probes detect API gaps. **`search_dependency_graph.md`** links templates to indices, aliases, and pipelines (heuristic); enable with **`elasticsearch_search_dependency_graph`** or **`limits.emit_dependency_graph: true`** (requires **`elasticsearch_search_architecture`**). Snapshot limits: **`limits.snapshot_repository_types`** (optional type filter), **`limits.max_snapshot_repositories`**. Search-template limits: **`limits.search_templates_langs`** (default **`mustache`**), **`limits.max_template_source_chars`** (markdown truncation). Presentation: **`output.elasticsearch_mapping_mode`** (`flattened` \| `summarized` \| `raw`), **`output.elasticsearch_include_raw_json`**, **`output.elasticsearch_analyzer_format`** (`chain` \| `json` \| `both`), **`output.max_json_block_chars`** (default **32000**; **`0`** = unlimited JSON blocks). Truncation and other export issues appear in per-doc **`## Export Warnings`** and **`export_manifest.json`** **`warnings`** when present. OpenSearch: set **`limits.opensearch: true`** to use ISM instead of ILM. Example config: [`example/db/elasticsearch.example.yaml`](example/db/elasticsearch.example.yaml). CLI: **`--type elasticsearch`**, **`--index-pattern`**, **`--list-indices`**. Security flags **`elasticsearch_security_roles`**, **`elasticsearch_security_users`**, **`elasticsearch_security_api_keys`** are **reserved**: when included they write deterministic placeholder docs under **`elasticsearch/security/`** only (no live `_security` API calls, no secrets). Optional **`security.redact_sensitive_values: true`** redacts matching keys in pipelines, templates, snapshots, and settings (pattern names only in manifest **`redaction`** audit block).
+
+**db-to-md API Elasticsearch upload:** **`POST /db-to-md/run/elasticsearch`** — `multipart/form-data` with **`file`** (ZIP of JSON metadata: **`mappings/`**, **`settings/`**, optional **`pipelines/`**, **`templates/`**, **`ilm/`**, etc.) and optional **`config`** JSON (`output`, `features`, `limits`). **`POST /db-to-md/job/elasticsearch`** for async jobs. Cap: **`DB_TO_MD_MAX_ELASTICSEARCH_UPLOAD_MB`** (default **64**).
+
+**sap-to-md (`md-sap`):** parses **ABAP**, **CDS**, **DDIC** exports, **OData** metadata, **BAPI**, **IDoc**, and transport artifacts into chunked Markdown knowledge packs. Optional **`--include-lineage`**, **`--include-governance`**, **`--graph`**, **`--chunk`**. Library: [`md_generator/sap/`](src/md_generator/sap/). Deeper design: [`sap-to-md/README.md`](sap-to-md/README.md).
+
+**otel-to-md (`md-otel`):** reads **OTLP** trace exports (**JSON** or **protobuf** with **`log-otel-proto`**) and writes **`trace.md`** (span list). No separate HTTP API entry point; use **`md-otel`** or correlate traces with log exports via **`input.otel_path`** in log YAML. Library: [`md_generator/otel/`](src/md_generator/otel/).
+
 **graph-to-md (Neo4j + NetworkX):** library lives under [`md_generator/graph/`](src/md_generator/graph/). **Sources:** **`networkx`** (GraphML/GML via `graph.graph_file` or `--graph-file`) or **`neo4j`** (`graph.uri`, `graph.user`, `graph.password`, optional `graph.database` for `session(database=…)`). **Output layout:** by default **`output.combine_markdown: true`** writes **`nodes.md`**, **`relationship.md`**, and **`graph_summary.md`** (summary plus embedded nodes and relationships). Set **`combine_markdown: false`** or CLI **`--individual` / `--markdown-layout individual`** for per-entity files under **`nodes/`** and **`relationships/`**. **Diagrams:** **`viz.mermaid: true`** (default) writes **`graph/graph.mmd`** and embeds a fenced Mermaid block in the export **`README.md`** (no Graphviz required). **`viz.enabled: true`** or CLI **`--viz`** also writes **`graph/graph.dot`** and runs **`dot`** for PNG/SVG/PDF when Graphviz is on `PATH` (or **`GRAPHVIZ_DOT`**). CLI **`--no-mermaid`** disables Mermaid; **`--depth`**, **`--start-node`**, **`--max-nodes`**, **`--max-edges`** bound traversal. Packaged defaults: [`src/md_generator/graph/config/default.yaml`](src/md_generator/graph/config/default.yaml). Tests: **`graph-to-md/tests/`**; API image: [`graph-to-md/Dockerfile.api`](graph-to-md/Dockerfile.api).
 
 Every command accepts **`-h` / `--help`** for full flags (artifact layout, OCR, ZIP options, etc.).
@@ -430,6 +457,8 @@ pip install "mdengine[graph]" && md-graph --source neo4j --uri neo4j://localhost
 pip install "mdengine[playwright]" && playwright install chromium && md-playwright https://example.com/app ./spa-out
 pip install "mdengine[openapi]" && md-openapi generate --file openapi.yaml --output ./openapi-md
 pip install "mdengine[log]" && md-log --config log.yaml
+pip install "mdengine[log-otel-proto]" && md-otel --input otlp-traces.json --output ./otel-docs
+pip install "mdengine[sap]" && md-sap ./sap-source --output ./sap-out --graph --chunk
 md-codeflow scan ./src --output ./cf-out --lang python
 ```
 
@@ -706,6 +735,44 @@ cfg = load_log_run_config(Path("log-export.yaml"))
 extract_to_markdown(cfg)
 ```
 
+Set **`input.otel_path`** in the YAML (or API body) to a directory of OTLP JSON exports so the log pipeline can **correlate** log lines with spans. Install **`mdengine[log-otel-proto]`** when ingesting protobuf OTLP via **`md-otel`**.
+
+### OpenTelemetry (otel-to-md)
+
+```python
+from pathlib import Path
+
+from md_generator.otel.otel_parser import load_otlp_json
+from md_generator.otel.otel_spans import parse_otlp_spans
+
+doc = load_otlp_json(Path("otlp-traces.json"))
+spans = parse_otlp_spans(doc)
+```
+
+CLI (writes **`trace.md`** under **`--output`**):
+
+```bash
+pip install "mdengine[log-otel-proto]"
+md-otel --input otlp-traces.json --output ./otel-docs
+md-otel --input export.pb --output ./otel-docs --protobuf
+```
+
+Shim: [`otel-to-md/converter.py`](otel-to-md/converter.py) delegates to **`md-otel`**.
+
+### SAP artifacts (sap-to-md)
+
+```python
+from pathlib import Path
+
+from md_generator.sap.core.extractor import extract_to_markdown
+from md_generator.sap.core.run_config import load_run_config
+
+cfg = load_run_config(Path("sap-export.yaml"))
+extract_to_markdown(cfg)
+```
+
+Or pass paths on the CLI: **`md-sap ./abap ./cds --output ./sap-out --include-lineage --include-governance`**. See [`sap-to-md/README.md`](sap-to-md/README.md) for API routes and architecture notes.
+
 ### Codeflow (static call graphs)
 
 ```python
@@ -836,7 +903,7 @@ Equivalent modules: `python -m md_generator.media.audio.api.mcp_server`, `python
 
 ### Thin shims (repo clone)
 
-[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`. [`db-to-md/converter.py`](db-to-md/converter.py) delegates to `md-db`; tests live under `db-to-md/tests/`. **graph-to-md** uses `md-graph` / `mdengine graph-to-md` directly (no thin `converter.py` shim); tests live under [`graph-to-md/tests/`](graph-to-md/tests/). [`openapi-to-md/converter.py`](openapi-to-md/converter.py) delegates to `md-openapi`; tests live under [`openapi-to-md/tests/`](openapi-to-md/tests/); example OpenAPI and output notes: [`openapi-to-md/examples/`](openapi-to-md/examples/). [`log-to-md/converter.py`](log-to-md/converter.py) delegates to `md-log`; tests under [`log-to-md/tests/`](log-to-md/tests/). [`codeflow-to-md/converter.py`](codeflow-to-md/converter.py) invokes the same CLI entrypoint as `md-codeflow` / `codeflow`; tests under [`codeflow-to-md/tests/`](codeflow-to-md/tests/). [`url-to-md/converter.py`](url-to-md/converter.py) matches `md-url`; tests under [`url-to-md/tests/`](url-to-md/tests/). [`playwright-to-md/tests/`](playwright-to-md/tests/) cover the Playwright pipeline (no root `converter.py` shim required).
+[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`. [`db-to-md/converter.py`](db-to-md/converter.py) delegates to `md-db`; tests live under `db-to-md/tests/`. **graph-to-md** uses `md-graph` / `mdengine graph-to-md` directly (no thin `converter.py` shim); tests live under [`graph-to-md/tests/`](graph-to-md/tests/). [`openapi-to-md/converter.py`](openapi-to-md/converter.py) delegates to `md-openapi`; tests live under [`openapi-to-md/tests/`](openapi-to-md/tests/); example OpenAPI and output notes: [`openapi-to-md/examples/`](openapi-to-md/examples/). [`log-to-md/converter.py`](log-to-md/converter.py) delegates to `md-log`; tests under [`log-to-md/tests/`](log-to-md/tests/) (including OTLP fixtures under `log-to-md/tests/fixtures/otel/`). [`otel-to-md/converter.py`](otel-to-md/converter.py) delegates to `md-otel`. [`sap-to-md/converter.py`](sap-to-md/converter.py) delegates to `md-sap`; tests under [`sap-to-md/tests/`](sap-to-md/tests/); module README: [`sap-to-md/README.md`](sap-to-md/README.md). [`codeflow-to-md/converter.py`](codeflow-to-md/converter.py) invokes the same CLI entrypoint as `md-codeflow` / `codeflow`; tests under [`codeflow-to-md/tests/`](codeflow-to-md/tests/). [`url-to-md/converter.py`](url-to-md/converter.py) matches `md-url`; tests under [`url-to-md/tests/`](url-to-md/tests/). [`playwright-to-md/tests/`](playwright-to-md/tests/) cover the Playwright pipeline (no root `converter.py` shim required).
 
 ---
 
@@ -874,6 +941,7 @@ Install `mdengine[api]` plus the format extra(s), then run the **`app`** object
 | YouTube | `md_generator.media.youtube.api.main:create_app` (use **`--factory`**) or `…main:app` | `youtube`, `api`, `mcp` |
 | Codeflow (workspace scans) | `md_generator.codeflow.api.main:app` | `codeflow`, `api`, `mcp` |
 | Log files → Markdown | `md_generator.log.api.main:app` | `log`, `api`, `mcp` |
+| SAP artifacts | `md_generator.sap.api.main:app` | `sap`, `api` |
 
 Examples (ports are **illustrative** so several services can run together; each `md-*-api` / `run` module has its own **default** — set `*_PORT` / `*_API_PORT` env vars to match your layout):
 
@@ -885,6 +953,7 @@ uvicorn md_generator.url.api.main:app --host 127.0.0.1 --port 8017
 uvicorn md_generator.db.api.main:app --host 127.0.0.1 --port 8010
 uvicorn md_generator.playwright.api.main:app --host 127.0.0.1 --port 8014
 uvicorn md_generator.graph.api.main:app --host 127.0.0.1 --port 8020
+uvicorn md_generator.sap.api.main:app --host 127.0.0.1 --port 8021
 uvicorn md_generator.openapi.api.main:app --host 127.0.0.1 --port 8015
 uvicorn md_generator.codeflow.api.main:app --host 127.0.0.1 --port 8016
 uvicorn md_generator.log.api.main:app --host 127.0.0.1 --port 8018
@@ -893,7 +962,7 @@ uvicorn md_generator.media.video.api.main:create_app --factory --host 127.0.0.1
 uvicorn md_generator.media.youtube.api.main:create_app --factory --host 127.0.0.1 --port 8013
 ```
 
-**Port note:** **`md-graph-api`**, **`md-video-api`**, and **`md-log-api`** all default to **8012** in their respective `run.py` modules. Run only one on that port, or set **`GRAPH_TO_MD_PORT`**, **`MD_VIDEO_API_PORT`**, and **`LOG_TO_MD_PORT`** so each process listens on a unique port. **`md-db-api`** defaults to **8010** (not shown colliding with the archive example above, which uses **8008** on purpose). **`md-openapi-api`** defaults to **8015** (`OPENAPI_TO_MD_PORT`). **`md-youtube-api`** defaults to **8013**; **`md-playwright-api`** / **`PLAYWRIGHT_TO_MD_API_PORT`** default to **8014**. **`md-codeflow-api`** defaults to **8016** (`CODEFLOW_TO_MD_PORT`).
+**Port note:** **`md-graph-api`**, **`md-video-api`**, and **`md-log-api`** all default to **8012** in their respective `run.py` modules. Run only one on that port, or set **`GRAPH_TO_MD_PORT`**, **`MD_VIDEO_API_PORT`**, and **`LOG_TO_MD_PORT`** so each process listens on a unique port. **`md-db-api`** defaults to **8010**. **`md-sap-api`** defaults to **8020** (`SAP_TO_MD_PORT`) — the graph Uvicorn example above uses **8020** on purpose as an illustration; use **8021** (or another port) when running **both** graph and SAP APIs locally. **`md-openapi-api`** defaults to **8015** (`OPENAPI_TO_MD_PORT`). **`md-youtube-api`** defaults to **8013**; **`md-playwright-api`** / **`PLAYWRIGHT_TO_MD_API_PORT`** default to **8014**. **`md-codeflow-api`** defaults to **8016** (`CODEFLOW_TO_MD_PORT`). **`md-otel`** has **no** bundled HTTP server (CLI only).
 
 ### MCP over HTTP on the same server
 
@@ -918,6 +987,7 @@ Prefixes differ per service (often read from a `.env` file next to the process):
 | OpenAPI → Markdown | `OPENAPI_TO_MD_` | `OPENAPI_TO_MD_CORS_ORIGINS`, `OPENAPI_TO_MD_MAX_SYNC_ZIP_MB`, `OPENAPI_TO_MD_HOST`, `OPENAPI_TO_MD_PORT` (default **8015**) |
 | Codeflow API | `CODEFLOW_TO_MD_` / `CODEFLOW_` | `CODEFLOW_TO_MD_HOST`, `CODEFLOW_TO_MD_PORT` (default **8016**); in-app limits: `CODEFLOW_MAX_UPLOAD_ZIP_MB`, `CODEFLOW_MAX_SYNC_ZIP_MB`, `CODEFLOW_JOB_WORKSPACE_ROOT`, `CODEFLOW_SQLITE_PATH`, CORS via `CODEFLOW_CORS` |
 | Log API | `LOG_TO_MD_` | `LOG_TO_MD_HOST`, `LOG_TO_MD_PORT` (default **8012** — change if **graph** or **video** API uses that port), `LOG_TO_MD_CORS_ORIGINS`, `LOG_TO_MD_MAX_SYNC_ZIP_MB`, `LOG_TO_MD_MAX_LOG_UPLOAD_MB`, workspace / SQLite paths |
+| SAP API | `SAP_TO_MD_` | `SAP_TO_MD_HOST`, `SAP_TO_MD_PORT` (default **8020**), `SAP_TO_MD_CORS_ORIGINS`, `SAP_TO_MD_MAX_SYNC_ZIP_MB`, `SAP_TO_MD_JOB_SQLITE_PATH`, `SAP_TO_MD_JOB_WORKSPACE_ROOT` |
 | Audio API | `MD_AUDIO_` | `MD_AUDIO_MAX_UPLOAD_MB`, `MD_AUDIO_MAX_SYNC_UPLOAD_MB`, `MD_AUDIO_JOB_TTL_SECONDS`, `MD_AUDIO_TEMP_DIR`, `MD_AUDIO_CORS_ORIGINS`, `MD_AUDIO_API_HOST`, `MD_AUDIO_API_PORT` |
 | Video API | `MD_VIDEO_` | Same pattern as audio with `MD_VIDEO_*` (defaults: larger upload/sync caps, port **8012**) |
 | YouTube API | `MD_YOUTUBE_` | `MD_YOUTUBE_JOB_TTL_SECONDS`, `MD_YOUTUBE_TEMP_DIR`, `MD_YOUTUBE_CORS_ORIGINS`, `MD_YOUTUBE_API_HOST`, `MD_YOUTUBE_API_PORT` (default **8013**); optional `MD_YOUTUBE_YTDLP` path for audio fallback |
@@ -953,6 +1023,7 @@ Two usage patterns:
 | OpenAPI → Markdown | `md-openapi-mcp` or `python -m md_generator.openapi.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
 | Codeflow | `md-codeflow-mcp` or `python -m md_generator.codeflow.api.mcp_server` — **stdio** MCP (see module for details) |
 | Log files | `md-log-mcp` or `python -m md_generator.log.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
+| SAP | `md-sap-mcp` or `md-sap-api` — FastAPI app (same as HTTP API; dedicated MCP tools when mounted in app) |
 
 **Word** and **XLSX** also ship a small runner script in the repo:
 
@@ -1081,7 +1152,7 @@ pip install -e ".[dev,all]"   # or a smaller subset of extras
 python -m pytest
 ```
 
-Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`), plus **`url-to-md/tests/`**, **`playwright-to-md/tests/`**, **`youtube-to-md/tests/`**, **`graph-to-md/tests/`**, **`openapi-to-md/tests/`**, **`codeflow-to-md/tests/`**, **`log-to-md/tests/`**, and **[`tool-assistant/tests/`](tool-assistant/tests/)** for the skill SDK; `pyproject.toml` sets `pythonpath = ["src"]` so **`md_generator`** (including **`md_generator.tools.assistant`**) resolves without a manual `PYTHONPATH` when you use `pytest` from the config.
+Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`), plus **`url-to-md/tests/`**, **`playwright-to-md/tests/`**, **`youtube-to-md/tests/`**, **`graph-to-md/tests/`**, **`openapi-to-md/tests/`**, **`codeflow-to-md/tests/`**, **`log-to-md/tests/`**, **`sap-to-md/tests/`**, and **[`tool-assistant/tests/`](tool-assistant/tests/)** for the skill SDK; `pyproject.toml` sets `pythonpath = ["src"]` so **`md_generator`** (including **`md_generator.tools.assistant`**) resolves without a manual `PYTHONPATH` when you use `pytest` from the config.
 
 **Docs site (maintainers):** install **`mdengine[docs]`**, then build or serve the MkDocs project from the repo root (`mkdocs.yml`). The published site URL is listed under **Documentation** in `[project.urls]` on PyPI ([GitHub Pages build](https://vishal7090.github.io/md-generator/)).
 
@@ -1118,7 +1189,7 @@ python -m md_generator.codeflow.cli.main scan path/to/src --output ./codeflow-ou
 | `docs/` | **MkDocs** source (module guides, `index.md`); build with **`mdengine[docs]`** |
 | `mkdocs.yml` | MkDocs configuration |
 | `CODE_OF_CONDUCT.md` | [Contributor Covenant](https://www.contributor-covenant.org/) 2.1 |
-| `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/audio/`](src/md_generator/media/audio/) and [`media/video/`](src/md_generator/media/video/); **graph-to-md** under [`graph/`](src/md_generator/graph/) |
+| `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/`](src/md_generator/media/); **logs** [`log/`](src/md_generator/log/), **OTEL** [`otel/`](src/md_generator/otel/), **SAP** [`sap/`](src/md_generator/sap/), **graphs** [`graph/`](src/md_generator/graph/) |
 | `pyproject.toml` | Packaging, extras, CLI entry points, pytest |
 | `ai/` | **Distributable AI skills** (`SKILL.md` trees, `registry.json`, `dependency-graph.json`); regenerate with `PYTHONPATH=src python -m md_generator.tools.skill_builder` or **`mdengine skill build`** (after `pip install -e .`) |
 | `src/md_generator/tools/skill_builder/` | **Skill generator** — scans `src/md_generator` and `pyproject.toml` |

{mdengine-0.11.0 → mdengine-0.12.0}/README.md

@@ -1,6 +1,6 @@
 # mdengine
 
-Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), **graphs** (Neo4j / NetworkX → Markdown), **OpenAPI** specs, **Playwright**-captured web pages (including SPAs), and **source code** (codeflow → architecture Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
+Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), **graphs** (Neo4j / NetworkX → Markdown), **OpenAPI** specs, **Playwright**-captured web pages (including SPAs), **application logs** (plain, JSON, CSV, ZIP bundles → Markdown with optional clustering and semantic grouping), **OpenTelemetry** traces (**OTLP** JSON or protobuf → Markdown trace summaries; optional correlation with log exports via `otel_path`), **SAP artifacts** (ABAP, CDS, DDIC, OData, BAPI, IDoc, transport → AI-ready Markdown knowledge packs), and **source code** (codeflow → architecture Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
 
 - **PyPI name:** `mdengine` (import package: `md_generator`)
 - **Source:** [github.com/vishal7090/md-generator](https://github.com/vishal7090/md-generator)
@@ -82,13 +82,15 @@ pip install "mdengine[all]"
 | `openapi` | **OpenAPI / Swagger → Markdown** (`prance`, `openapi-spec-validator`, PyYAML) |
 | `codeflow` | **Static codeflow / call graphs → Markdown** (`networkx`, `javalang`; core scanners) |
 | `codeflow-worker` | Optional **Celery + Redis** workers for the codeflow API async path |
-| `codeflow-treesitter` | **Tree-sitter** parsers (JS/TS/TSX, C++) for richer codeflow slices |
+| `codeflow-treesitter` | **Tree-sitter** parsers (JS/TS/TSX, C++, Java, Python, Go, PHP); use `--parser-mode treesitter` — see [codeflow-to-md/docs/parser-backends.md](codeflow-to-md/docs/parser-backends.md) |
 | `codeflow-clang` | **libclang** bindings for C/C++ parsing when used by the codeflow pipeline |
 | `codeflow-semantic` | **Semantic clustering** (SentenceTransformers, scikit-learn, numpy; large install) |
 | `log` | **Log files → Markdown** (`pandas`, PyYAML, date parsing) |
 | `log-cluster` | Log pipeline **clustering** (scikit-learn) |
 | `log-semantic` | Log pipeline **semantic** helpers (SentenceTransformers + scikit-learn; large) |
 | `log-pretty` | Optional **loguru** for pretty console diagnostics during log conversion |
+| `log-otel-proto` | **OTLP protobuf** ingest for **`md-otel`** and log/OTEL tooling (`opentelemetry-proto`, `protobuf`) |
+| `sap` | **SAP artifacts → Markdown** (ABAP, CDS, DDIC, OData, BAPI, IDoc, transport): PyYAML, NetworkX, Pydantic |
 | `api` | FastAPI, uvicorn, httpx, pydantic-settings |
 | `mcp` | MCP servers (`mcp`, `fastmcp` where used) |
 | `graph` | **Graph → Markdown** (Neo4j Bolt + NetworkX GraphML/GML): `networkx`, `neo4j`, `pyyaml` |
@@ -163,7 +165,11 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-log` | `md_generator.log.cli.main:main` | `pip install "mdengine[log]"` then `md-log --config log.yaml` (or `mdengine log-to-md …`) |
 | `md-log-api` | `md_generator.log.api.run:main` | FastAPI on port **8012** (`LOG_TO_MD_PORT`); set a different port if **`md-graph-api`** or **`md-video-api`** already uses **8012** |
 | `md-log-mcp` | `md_generator.log.api.mcp_server:main` | Standalone MCP (`stdio` \| `sse` \| `streamable-http`) |
-| `mdengine` | `md_generator.engine_cli:main` | `mdengine ai assist …`, `mdengine ai export …`, `mdengine skill build …`, `mdengine db-to-md …`, `mdengine log-to-md …`, `mdengine graph-to-md …`, `mdengine openapi-to-md generate …`, `mdengine codeflow-to-md scan …` |
+| `md-otel` | `md_generator.otel.cli.main:main` | `pip install "mdengine[log-otel-proto]"` then `md-otel --input traces.json --output ./otel-docs` (or `mdengine otel-to-md …`; add **`--protobuf`** for OTLP binary) |
+| `md-sap` | `md_generator.sap.cli.main:main` | `pip install "mdengine[sap]"` then `md-sap ./sap-source --output ./sap-out --graph --chunk` (or `mdengine sap-to-md …`) |
+| `md-sap-api` | `md_generator.sap.api.run:main` | FastAPI on port **8020** (`SAP_TO_MD_PORT`): `POST /sap-to-md/run`, `POST /sap-to-md/job`, `GET /sap-to-md/job/{id}/download` |
+| `md-sap-mcp` | `md_generator.sap.api.mcp_server:main` | Runs the same FastAPI app as **`md-sap-api`** (MCP mount when enabled in app) |
+| `mdengine` | `md_generator.engine_cli:main` | `mdengine ai assist …`, `mdengine ai export …`, `mdengine skill build …`, `mdengine db-to-md …`, `mdengine log-to-md …`, `mdengine otel-to-md …`, `mdengine sap-to-md …`, `mdengine graph-to-md …`, `mdengine openapi-to-md generate …`, `mdengine codeflow-to-md scan …` |
 
 **openapi-to-md (`md-openapi`):** OpenAPI **3.x** is parsed directly; **Swagger 2.0** (`swagger: "2.0"`) documents are **converted in-process** to OpenAPI 3.0.3 (deterministic, in-repo converter) before `$ref` resolution. Edge-heavy specs (unusual OAuth2 flows, vendor extensions) may still need fixes after conversion.
 
@@ -183,6 +189,14 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 
 **db-to-md API SQLite upload:** **`POST /db-to-md/run/sqlite`** — `multipart/form-data` with field **`file`** (the `.sqlite` / `.db` bytes; must start with the standard `SQLite format 3` header) and optional **`config`** (JSON string with the same shape as a normal run body minus `database`: `schema`, `output`, `features`, `execution`, `limits`, `erd`). Returns the metadata ZIP immediately. **`POST /db-to-md/job/sqlite`** — same form fields; saves the file under the job workspace and runs the existing job pipeline (`GET /db-to-md/job/{id}/download` when complete). Upload size cap: env **`DB_TO_MD_MAX_SQLITE_UPLOAD_MB`** (default **256**). Large ZIPs may still require the async job path if they exceed **`DB_TO_MD_MAX_SYNC_ZIP_MB`** on the sync upload route.
 
+**db-to-md Elasticsearch / OpenSearch:** set **`database.type: elasticsearch`** and **`database.uri`** to the cluster URL (for example `https://localhost:9200`). Install **`mdengine[db]`** (includes the **`elasticsearch`** Python client). Feature flags: **`elasticsearch_indices`**, **`elasticsearch_data_streams`**, **`elasticsearch_component_templates`**, **`elasticsearch_index_templates`**, **`elasticsearch_ingest_pipelines`**, **`elasticsearch_ilm_policies`**, **`elasticsearch_slm_policies`**, **`elasticsearch_field_caps`**, **`elasticsearch_snapshot_repositories`**, **`elasticsearch_search_templates`**, **`elasticsearch_search_architecture`**, **`elasticsearch_search_dependency_graph`**. Output under **`elasticsearch/`** (indices, data streams, templates, pipelines, ILM, SLM, snapshots, search templates, **`alias_graph.md`**, **`search_architecture.md`**, **`search_dependency_graph.md`**). Index docs add **`## Mapping Summary`** (field counts, nested/vector/semantic totals) and **`## Operational Notes`** (health, replicas, vector-field signals). Search templates add **`## Query type`** using a vector-aware taxonomy (full text, aggregation, suggest, vector, semantic, hybrid). **`search_architecture.md`** narrates the cluster lifecycle (ingestion → indexing → retention → search → backup → alias routing) with aggregate mapping metrics, query-type counts, vector/AI readiness, and an **OpenSearch Compatibility** matrix when **`limits.opensearch: true`** or export probes detect API gaps. **`search_dependency_graph.md`** links templates to indices, aliases, and pipelines (heuristic); enable with **`elasticsearch_search_dependency_graph`** or **`limits.emit_dependency_graph: true`** (requires **`elasticsearch_search_architecture`**). Snapshot limits: **`limits.snapshot_repository_types`** (optional type filter), **`limits.max_snapshot_repositories`**. Search-template limits: **`limits.search_templates_langs`** (default **`mustache`**), **`limits.max_template_source_chars`** (markdown truncation). Presentation: **`output.elasticsearch_mapping_mode`** (`flattened` \| `summarized` \| `raw`), **`output.elasticsearch_include_raw_json`**, **`output.elasticsearch_analyzer_format`** (`chain` \| `json` \| `both`), **`output.max_json_block_chars`** (default **32000**; **`0`** = unlimited JSON blocks). Truncation and other export issues appear in per-doc **`## Export Warnings`** and **`export_manifest.json`** **`warnings`** when present. OpenSearch: set **`limits.opensearch: true`** to use ISM instead of ILM. Example config: [`example/db/elasticsearch.example.yaml`](example/db/elasticsearch.example.yaml). CLI: **`--type elasticsearch`**, **`--index-pattern`**, **`--list-indices`**. Security flags **`elasticsearch_security_roles`**, **`elasticsearch_security_users`**, **`elasticsearch_security_api_keys`** are **reserved**: when included they write deterministic placeholder docs under **`elasticsearch/security/`** only (no live `_security` API calls, no secrets). Optional **`security.redact_sensitive_values: true`** redacts matching keys in pipelines, templates, snapshots, and settings (pattern names only in manifest **`redaction`** audit block).
+
+**db-to-md API Elasticsearch upload:** **`POST /db-to-md/run/elasticsearch`** — `multipart/form-data` with **`file`** (ZIP of JSON metadata: **`mappings/`**, **`settings/`**, optional **`pipelines/`**, **`templates/`**, **`ilm/`**, etc.) and optional **`config`** JSON (`output`, `features`, `limits`). **`POST /db-to-md/job/elasticsearch`** for async jobs. Cap: **`DB_TO_MD_MAX_ELASTICSEARCH_UPLOAD_MB`** (default **64**).
+
+**sap-to-md (`md-sap`):** parses **ABAP**, **CDS**, **DDIC** exports, **OData** metadata, **BAPI**, **IDoc**, and transport artifacts into chunked Markdown knowledge packs. Optional **`--include-lineage`**, **`--include-governance`**, **`--graph`**, **`--chunk`**. Library: [`md_generator/sap/`](src/md_generator/sap/). Deeper design: [`sap-to-md/README.md`](sap-to-md/README.md).
+
+**otel-to-md (`md-otel`):** reads **OTLP** trace exports (**JSON** or **protobuf** with **`log-otel-proto`**) and writes **`trace.md`** (span list). No separate HTTP API entry point; use **`md-otel`** or correlate traces with log exports via **`input.otel_path`** in log YAML. Library: [`md_generator/otel/`](src/md_generator/otel/).
+
 **graph-to-md (Neo4j + NetworkX):** library lives under [`md_generator/graph/`](src/md_generator/graph/). **Sources:** **`networkx`** (GraphML/GML via `graph.graph_file` or `--graph-file`) or **`neo4j`** (`graph.uri`, `graph.user`, `graph.password`, optional `graph.database` for `session(database=…)`). **Output layout:** by default **`output.combine_markdown: true`** writes **`nodes.md`**, **`relationship.md`**, and **`graph_summary.md`** (summary plus embedded nodes and relationships). Set **`combine_markdown: false`** or CLI **`--individual` / `--markdown-layout individual`** for per-entity files under **`nodes/`** and **`relationships/`**. **Diagrams:** **`viz.mermaid: true`** (default) writes **`graph/graph.mmd`** and embeds a fenced Mermaid block in the export **`README.md`** (no Graphviz required). **`viz.enabled: true`** or CLI **`--viz`** also writes **`graph/graph.dot`** and runs **`dot`** for PNG/SVG/PDF when Graphviz is on `PATH` (or **`GRAPHVIZ_DOT`**). CLI **`--no-mermaid`** disables Mermaid; **`--depth`**, **`--start-node`**, **`--max-nodes`**, **`--max-edges`** bound traversal. Packaged defaults: [`src/md_generator/graph/config/default.yaml`](src/md_generator/graph/config/default.yaml). Tests: **`graph-to-md/tests/`**; API image: [`graph-to-md/Dockerfile.api`](graph-to-md/Dockerfile.api).
 
 Every command accepts **`-h` / `--help`** for full flags (artifact layout, OCR, ZIP options, etc.).
@@ -207,6 +221,8 @@ pip install "mdengine[graph]" && md-graph --source neo4j --uri neo4j://localhost
 pip install "mdengine[playwright]" && playwright install chromium && md-playwright https://example.com/app ./spa-out
 pip install "mdengine[openapi]" && md-openapi generate --file openapi.yaml --output ./openapi-md
 pip install "mdengine[log]" && md-log --config log.yaml
+pip install "mdengine[log-otel-proto]" && md-otel --input otlp-traces.json --output ./otel-docs
+pip install "mdengine[sap]" && md-sap ./sap-source --output ./sap-out --graph --chunk
 md-codeflow scan ./src --output ./cf-out --lang python
 ```
 
@@ -483,6 +499,44 @@ cfg = load_log_run_config(Path("log-export.yaml"))
 extract_to_markdown(cfg)
 ```
 
+Set **`input.otel_path`** in the YAML (or API body) to a directory of OTLP JSON exports so the log pipeline can **correlate** log lines with spans. Install **`mdengine[log-otel-proto]`** when ingesting protobuf OTLP via **`md-otel`**.
+
+### OpenTelemetry (otel-to-md)
+
+```python
+from pathlib import Path
+
+from md_generator.otel.otel_parser import load_otlp_json
+from md_generator.otel.otel_spans import parse_otlp_spans
+
+doc = load_otlp_json(Path("otlp-traces.json"))
+spans = parse_otlp_spans(doc)
+```
+
+CLI (writes **`trace.md`** under **`--output`**):
+
+```bash
+pip install "mdengine[log-otel-proto]"
+md-otel --input otlp-traces.json --output ./otel-docs
+md-otel --input export.pb --output ./otel-docs --protobuf
+```
+
+Shim: [`otel-to-md/converter.py`](otel-to-md/converter.py) delegates to **`md-otel`**.
+
+### SAP artifacts (sap-to-md)
+
+```python
+from pathlib import Path
+
+from md_generator.sap.core.extractor import extract_to_markdown
+from md_generator.sap.core.run_config import load_run_config
+
+cfg = load_run_config(Path("sap-export.yaml"))
+extract_to_markdown(cfg)
+```
+
+Or pass paths on the CLI: **`md-sap ./abap ./cds --output ./sap-out --include-lineage --include-governance`**. See [`sap-to-md/README.md`](sap-to-md/README.md) for API routes and architecture notes.
+
 ### Codeflow (static call graphs)
 
 ```python
@@ -613,7 +667,7 @@ Equivalent modules: `python -m md_generator.media.audio.api.mcp_server`, `python
 
 ### Thin shims (repo clone)
 
-[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`. [`db-to-md/converter.py`](db-to-md/converter.py) delegates to `md-db`; tests live under `db-to-md/tests/`. **graph-to-md** uses `md-graph` / `mdengine graph-to-md` directly (no thin `converter.py` shim); tests live under [`graph-to-md/tests/`](graph-to-md/tests/). [`openapi-to-md/converter.py`](openapi-to-md/converter.py) delegates to `md-openapi`; tests live under [`openapi-to-md/tests/`](openapi-to-md/tests/); example OpenAPI and output notes: [`openapi-to-md/examples/`](openapi-to-md/examples/). [`log-to-md/converter.py`](log-to-md/converter.py) delegates to `md-log`; tests under [`log-to-md/tests/`](log-to-md/tests/). [`codeflow-to-md/converter.py`](codeflow-to-md/converter.py) invokes the same CLI entrypoint as `md-codeflow` / `codeflow`; tests under [`codeflow-to-md/tests/`](codeflow-to-md/tests/). [`url-to-md/converter.py`](url-to-md/converter.py) matches `md-url`; tests under [`url-to-md/tests/`](url-to-md/tests/). [`playwright-to-md/tests/`](playwright-to-md/tests/) cover the Playwright pipeline (no root `converter.py` shim required).
+[`audio-to-md/converter.py`](audio-to-md/converter.py), [`video-to-md/converter.py`](video-to-md/converter.py), and [`youtube-to-md/converter.py`](youtube-to-md/converter.py) delegate to the same `main` as `md-audio` / `md-video` / `md-youtube`. Tests and `pytest.ini` live under `audio-to-md/tests/`, `video-to-md/tests/`, and `youtube-to-md/tests/`. [`db-to-md/converter.py`](db-to-md/converter.py) delegates to `md-db`; tests live under `db-to-md/tests/`. **graph-to-md** uses `md-graph` / `mdengine graph-to-md` directly (no thin `converter.py` shim); tests live under [`graph-to-md/tests/`](graph-to-md/tests/). [`openapi-to-md/converter.py`](openapi-to-md/converter.py) delegates to `md-openapi`; tests live under [`openapi-to-md/tests/`](openapi-to-md/tests/); example OpenAPI and output notes: [`openapi-to-md/examples/`](openapi-to-md/examples/). [`log-to-md/converter.py`](log-to-md/converter.py) delegates to `md-log`; tests under [`log-to-md/tests/`](log-to-md/tests/) (including OTLP fixtures under `log-to-md/tests/fixtures/otel/`). [`otel-to-md/converter.py`](otel-to-md/converter.py) delegates to `md-otel`. [`sap-to-md/converter.py`](sap-to-md/converter.py) delegates to `md-sap`; tests under [`sap-to-md/tests/`](sap-to-md/tests/); module README: [`sap-to-md/README.md`](sap-to-md/README.md). [`codeflow-to-md/converter.py`](codeflow-to-md/converter.py) invokes the same CLI entrypoint as `md-codeflow` / `codeflow`; tests under [`codeflow-to-md/tests/`](codeflow-to-md/tests/). [`url-to-md/converter.py`](url-to-md/converter.py) matches `md-url`; tests under [`url-to-md/tests/`](url-to-md/tests/). [`playwright-to-md/tests/`](playwright-to-md/tests/) cover the Playwright pipeline (no root `converter.py` shim required).
 
 ---
 
@@ -651,6 +705,7 @@ Install `mdengine[api]` plus the format extra(s), then run the **`app`** object
 | YouTube | `md_generator.media.youtube.api.main:create_app` (use **`--factory`**) or `…main:app` | `youtube`, `api`, `mcp` |
 | Codeflow (workspace scans) | `md_generator.codeflow.api.main:app` | `codeflow`, `api`, `mcp` |
 | Log files → Markdown | `md_generator.log.api.main:app` | `log`, `api`, `mcp` |
+| SAP artifacts | `md_generator.sap.api.main:app` | `sap`, `api` |
 
 Examples (ports are **illustrative** so several services can run together; each `md-*-api` / `run` module has its own **default** — set `*_PORT` / `*_API_PORT` env vars to match your layout):
 
@@ -662,6 +717,7 @@ uvicorn md_generator.url.api.main:app --host 127.0.0.1 --port 8017
 uvicorn md_generator.db.api.main:app --host 127.0.0.1 --port 8010
 uvicorn md_generator.playwright.api.main:app --host 127.0.0.1 --port 8014
 uvicorn md_generator.graph.api.main:app --host 127.0.0.1 --port 8020
+uvicorn md_generator.sap.api.main:app --host 127.0.0.1 --port 8021
 uvicorn md_generator.openapi.api.main:app --host 127.0.0.1 --port 8015
 uvicorn md_generator.codeflow.api.main:app --host 127.0.0.1 --port 8016
 uvicorn md_generator.log.api.main:app --host 127.0.0.1 --port 8018
@@ -670,7 +726,7 @@ uvicorn md_generator.media.video.api.main:create_app --factory --host 127.0.0.1
 uvicorn md_generator.media.youtube.api.main:create_app --factory --host 127.0.0.1 --port 8013
 ```
 
-**Port note:** **`md-graph-api`**, **`md-video-api`**, and **`md-log-api`** all default to **8012** in their respective `run.py` modules. Run only one on that port, or set **`GRAPH_TO_MD_PORT`**, **`MD_VIDEO_API_PORT`**, and **`LOG_TO_MD_PORT`** so each process listens on a unique port. **`md-db-api`** defaults to **8010** (not shown colliding with the archive example above, which uses **8008** on purpose). **`md-openapi-api`** defaults to **8015** (`OPENAPI_TO_MD_PORT`). **`md-youtube-api`** defaults to **8013**; **`md-playwright-api`** / **`PLAYWRIGHT_TO_MD_API_PORT`** default to **8014**. **`md-codeflow-api`** defaults to **8016** (`CODEFLOW_TO_MD_PORT`).
+**Port note:** **`md-graph-api`**, **`md-video-api`**, and **`md-log-api`** all default to **8012** in their respective `run.py` modules. Run only one on that port, or set **`GRAPH_TO_MD_PORT`**, **`MD_VIDEO_API_PORT`**, and **`LOG_TO_MD_PORT`** so each process listens on a unique port. **`md-db-api`** defaults to **8010**. **`md-sap-api`** defaults to **8020** (`SAP_TO_MD_PORT`) — the graph Uvicorn example above uses **8020** on purpose as an illustration; use **8021** (or another port) when running **both** graph and SAP APIs locally. **`md-openapi-api`** defaults to **8015** (`OPENAPI_TO_MD_PORT`). **`md-youtube-api`** defaults to **8013**; **`md-playwright-api`** / **`PLAYWRIGHT_TO_MD_API_PORT`** default to **8014**. **`md-codeflow-api`** defaults to **8016** (`CODEFLOW_TO_MD_PORT`). **`md-otel`** has **no** bundled HTTP server (CLI only).
 
 ### MCP over HTTP on the same server
 
@@ -695,6 +751,7 @@ Prefixes differ per service (often read from a `.env` file next to the process):
 | OpenAPI → Markdown | `OPENAPI_TO_MD_` | `OPENAPI_TO_MD_CORS_ORIGINS`, `OPENAPI_TO_MD_MAX_SYNC_ZIP_MB`, `OPENAPI_TO_MD_HOST`, `OPENAPI_TO_MD_PORT` (default **8015**) |
 | Codeflow API | `CODEFLOW_TO_MD_` / `CODEFLOW_` | `CODEFLOW_TO_MD_HOST`, `CODEFLOW_TO_MD_PORT` (default **8016**); in-app limits: `CODEFLOW_MAX_UPLOAD_ZIP_MB`, `CODEFLOW_MAX_SYNC_ZIP_MB`, `CODEFLOW_JOB_WORKSPACE_ROOT`, `CODEFLOW_SQLITE_PATH`, CORS via `CODEFLOW_CORS` |
 | Log API | `LOG_TO_MD_` | `LOG_TO_MD_HOST`, `LOG_TO_MD_PORT` (default **8012** — change if **graph** or **video** API uses that port), `LOG_TO_MD_CORS_ORIGINS`, `LOG_TO_MD_MAX_SYNC_ZIP_MB`, `LOG_TO_MD_MAX_LOG_UPLOAD_MB`, workspace / SQLite paths |
+| SAP API | `SAP_TO_MD_` | `SAP_TO_MD_HOST`, `SAP_TO_MD_PORT` (default **8020**), `SAP_TO_MD_CORS_ORIGINS`, `SAP_TO_MD_MAX_SYNC_ZIP_MB`, `SAP_TO_MD_JOB_SQLITE_PATH`, `SAP_TO_MD_JOB_WORKSPACE_ROOT` |
 | Audio API | `MD_AUDIO_` | `MD_AUDIO_MAX_UPLOAD_MB`, `MD_AUDIO_MAX_SYNC_UPLOAD_MB`, `MD_AUDIO_JOB_TTL_SECONDS`, `MD_AUDIO_TEMP_DIR`, `MD_AUDIO_CORS_ORIGINS`, `MD_AUDIO_API_HOST`, `MD_AUDIO_API_PORT` |
 | Video API | `MD_VIDEO_` | Same pattern as audio with `MD_VIDEO_*` (defaults: larger upload/sync caps, port **8012**) |
 | YouTube API | `MD_YOUTUBE_` | `MD_YOUTUBE_JOB_TTL_SECONDS`, `MD_YOUTUBE_TEMP_DIR`, `MD_YOUTUBE_CORS_ORIGINS`, `MD_YOUTUBE_API_HOST`, `MD_YOUTUBE_API_PORT` (default **8013**); optional `MD_YOUTUBE_YTDLP` path for audio fallback |
@@ -730,6 +787,7 @@ Two usage patterns:
 | OpenAPI → Markdown | `md-openapi-mcp` or `python -m md_generator.openapi.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
 | Codeflow | `md-codeflow-mcp` or `python -m md_generator.codeflow.api.mcp_server` — **stdio** MCP (see module for details) |
 | Log files | `md-log-mcp` or `python -m md_generator.log.api.mcp_server` — `--transport stdio` (default), `sse`, `streamable-http` |
+| SAP | `md-sap-mcp` or `md-sap-api` — FastAPI app (same as HTTP API; dedicated MCP tools when mounted in app) |
 
 **Word** and **XLSX** also ship a small runner script in the repo:
 
@@ -858,7 +916,7 @@ pip install -e ".[dev,all]"   # or a smaller subset of extras
 python -m pytest
 ```
 
-Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`), plus **`url-to-md/tests/`**, **`playwright-to-md/tests/`**, **`youtube-to-md/tests/`**, **`graph-to-md/tests/`**, **`openapi-to-md/tests/`**, **`codeflow-to-md/tests/`**, **`log-to-md/tests/`**, and **[`tool-assistant/tests/`](tool-assistant/tests/)** for the skill SDK; `pyproject.toml` sets `pythonpath = ["src"]` so **`md_generator`** (including **`md_generator.tools.assistant`**) resolves without a manual `PYTHONPATH` when you use `pytest` from the config.
+Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`), plus **`url-to-md/tests/`**, **`playwright-to-md/tests/`**, **`youtube-to-md/tests/`**, **`graph-to-md/tests/`**, **`openapi-to-md/tests/`**, **`codeflow-to-md/tests/`**, **`log-to-md/tests/`**, **`sap-to-md/tests/`**, and **[`tool-assistant/tests/`](tool-assistant/tests/)** for the skill SDK; `pyproject.toml` sets `pythonpath = ["src"]` so **`md_generator`** (including **`md_generator.tools.assistant`**) resolves without a manual `PYTHONPATH` when you use `pytest` from the config.
 
 **Docs site (maintainers):** install **`mdengine[docs]`**, then build or serve the MkDocs project from the repo root (`mkdocs.yml`). The published site URL is listed under **Documentation** in `[project.urls]` on PyPI ([GitHub Pages build](https://vishal7090.github.io/md-generator/)).
 
@@ -895,7 +953,7 @@ python -m md_generator.codeflow.cli.main scan path/to/src --output ./codeflow-ou
 | `docs/` | **MkDocs** source (module guides, `index.md`); build with **`mdengine[docs]`** |
 | `mkdocs.yml` | MkDocs configuration |
 | `CODE_OF_CONDUCT.md` | [Contributor Covenant](https://www.contributor-covenant.org/) 2.1 |
-| `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/audio/`](src/md_generator/media/audio/) and [`media/video/`](src/md_generator/media/video/); **graph-to-md** under [`graph/`](src/md_generator/graph/) |
+| `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/`](src/md_generator/media/); **logs** [`log/`](src/md_generator/log/), **OTEL** [`otel/`](src/md_generator/otel/), **SAP** [`sap/`](src/md_generator/sap/), **graphs** [`graph/`](src/md_generator/graph/) |
 | `pyproject.toml` | Packaging, extras, CLI entry points, pytest |
 | `ai/` | **Distributable AI skills** (`SKILL.md` trees, `registry.json`, `dependency-graph.json`); regenerate with `PYTHONPATH=src python -m md_generator.tools.skill_builder` or **`mdengine skill build`** (after `pip install -e .`) |
 | `src/md_generator/tools/skill_builder/` | **Skill generator** — scans `src/md_generator` and `pyproject.toml` |

{mdengine-0.11.0 → mdengine-0.12.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mdengine"
-version = "0.11.0"
+version = "0.12.0"
 description = "Convert PDF, Office, images, audio, video, text/JSON/XML, ZIP archives, web URLs, databases, graphs, and OpenAPI specs to Markdown."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -101,6 +101,7 @@ db = [
   "pyodbc>=5.0.0",
   "oracledb>=2.0.0",
   "pymongo>=4.6.0",
+  "elasticsearch>=8.0.0",
   "mermaid-py>=0.8.0",
 ]
 graph = [
@@ -133,6 +134,18 @@ codeflow-treesitter = [
   "tree-sitter-javascript>=0.21.0",
   "tree-sitter-typescript>=0.23.0",
   "tree-sitter-cpp>=0.23.0",
+  "tree-sitter-java>=0.23.0",
+  "tree-sitter-python>=0.23.0",
+  "tree-sitter-go>=0.23.0",
+  "tree-sitter-php>=0.23.0",
+  "tree-sitter-rust>=0.23.0",
+  "tree-sitter-kotlin>=0.23.0",
+  "tree-sitter-c-sharp>=0.23.0",
+  "tree-sitter-swift>=0.7.2",
+  "tree-sitter-ruby>=0.23.0",
+  "tree-sitter-lua>=0.2.0",
+  "tree-sitter-scala>=0.6.0",
+  "tree-sitter-zig>=0.2.0",
 ]
 codeflow-clang = [
   "clang>=16.0.0",

{mdengine-0.11.0 → mdengine-0.12.0}/src/md_generator/__init__.py

@@ -1,5 +1,5 @@
 """md_generator: PDF, Office, image, text, and ZIP to Markdown conversion."""
 
-__version__ = "0.11.0"
+__version__ = "0.12.0"
 
 __all__ = ["__version__"]

{mdengine-0.11.0 → mdengine-0.12.0}/src/md_generator/codeflow/cli/main.py

@@ -76,6 +76,7 @@ def build_parser() -> argparse.ArgumentParser:
         default="mixed",
         help=(
             "mixed | python | java | javascript | typescript | tsx | cpp | go | php | "
+            "rust | kotlin | csharp | swift | ruby | lua | scala | zig | "
             "comma-separated (e.g. python,javascript). Aliases: js→javascript, ts→typescript."
         ),
     )
@@ -377,7 +378,8 @@ def build_parser() -> argparse.ArgumentParser:
         "--parser-mode",
         choices=("auto", "treesitter", "external"),
         default="auto",
-        help="C++: treesitter=Tree-sitter only; external=clang only; auto=current fallback",
+        help="auto=native parsers; treesitter=Tree-sitter for java/python/go/php/cpp (+ JS/TS when installed); "
+        "external=clang only for C++",
     )
     scan.add_argument(
         "--ui-cfg-max-methods",

{mdengine-0.11.0 → mdengine-0.12.0}/src/md_generator/codeflow/core/extractor.py

@@ -172,6 +172,22 @@ def _resolve_scan_entry_ids(cfg: ScanConfig, g: CodeflowGraph, all_entries: list
     return entry_ids, warnings
 
 
+def _parse_backend_summary_lines(parse_results: list[FileParseResult]) -> list[str]:
+    from collections import Counter
+
+    counts: Counter[tuple[str, str, str | None]] = Counter()
+    for fr in parse_results:
+        counts[(fr.language, fr.parse_backend, fr.grammar_package)] += 1
+    if not counts:
+        return []
+    lines = ["## Parse backends", ""]
+    for (lang, backend, gram), n in sorted(counts.items()):
+        gpart = f" (`{gram}`)" if gram else ""
+        lines.append(f"- **{lang}** / {backend}{gpart}: {n} file(s)")
+    lines.append("")
+    return lines
+
+
 def _write_scan_summary(
     path: Path,
     *,
@@ -183,6 +199,7 @@ def _write_scan_summary(
     warnings: list[str],
     cfg: ScanConfig,
     pr_impact: dict[str, Any] | None = None,
+    parse_results: list[FileParseResult] | None = None,
 ) -> None:
     lines = [
         "# Scan summary",
@@ -218,6 +235,10 @@ def _write_scan_summary(
         f"- **enable_dependency_graph:** {cfg.enable_dependency_graph}",
         f"- **structural_graph_enabled:** {cfg.structural_graph_enabled()}",
         f"- **parser_mode:** `{cfg.parser_mode}`",
+    ]
+    if parse_results:
+        lines.extend(_parse_backend_summary_lines(parse_results))
+    lines += [
         f"- **ui_cfg_max_methods:** {cfg.ui_cfg_max_methods}",
         f"- **intelligence_transitive_callers:** {cfg.intelligence_transitive_callers}",
         f"- **emit_system_graph_stats:** {cfg.emit_system_graph_stats}",
@@ -448,6 +469,12 @@ def run_scan(cfg: ScanConfig, *, workspace: LoadedWorkspace | None = None) -> Pa
     all_entries = [e for pr in parse_results for e in pr.entries]
     entry_ids, scan_warnings = _resolve_scan_entry_ids(cfg, g, all_entries)
     scan_warnings = scan_warnings_pre + scan_warnings
+    from md_generator.codeflow.parsers.capability_registry import capability_warnings
+
+    for lang in sorted({fr.language for fr in parse_results}):
+        scan_warnings.extend(
+            capability_warnings(lang, parser_mode=cfg.parser_mode, emit_cfg=cfg.emit_cfg),
+        )
     scan_warnings.extend(scan_semantic_warnings)
 
     fmts = {x.strip().lower() for x in cfg.formats}
@@ -920,6 +947,7 @@ def run_scan(cfg: ScanConfig, *, workspace: LoadedWorkspace | None = None) -> Pa
             warnings=scan_warnings,
             cfg=cfg,
             pr_impact=pr_impact_payload,
+            parse_results=parse_results,
         )
 
     return out