mdengine 0.7.0__tar.gz → 0.8.1__tar.gz

{mdengine-0.7.0/src/mdengine.egg-info → mdengine-0.8.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mdengine
-Version: 0.7.0
+Version: 0.8.1
 Summary: Convert PDF, Office, images, audio, video, text/JSON/XML, ZIP archives, web URLs, databases, graphs, and OpenAPI specs to Markdown.
 Author: mdengine contributors
 License-Expression: MIT
@@ -106,6 +106,9 @@ Requires-Dist: openapi-spec-validator>=0.7.0; extra == "openapi"
 Provides-Extra: codeflow
 Requires-Dist: networkx>=3.2.0; extra == "codeflow"
 Requires-Dist: javalang>=0.13.0; extra == "codeflow"
+Provides-Extra: codeflow-worker
+Requires-Dist: celery>=5.3.0; extra == "codeflow-worker"
+Requires-Dist: redis>=5.0.0; extra == "codeflow-worker"
 Provides-Extra: codeflow-treesitter
 Requires-Dist: tree-sitter>=0.21.0; extra == "codeflow-treesitter"
 Requires-Dist: tree-sitter-javascript>=0.21.0; extra == "codeflow-treesitter"
@@ -113,6 +116,10 @@ Requires-Dist: tree-sitter-typescript>=0.23.0; extra == "codeflow-treesitter"
 Requires-Dist: tree-sitter-cpp>=0.23.0; extra == "codeflow-treesitter"
 Provides-Extra: codeflow-clang
 Requires-Dist: clang>=16.0.0; extra == "codeflow-clang"
+Provides-Extra: codeflow-semantic
+Requires-Dist: numpy>=1.24.0; extra == "codeflow-semantic"
+Requires-Dist: scikit-learn>=1.3.0; extra == "codeflow-semantic"
+Requires-Dist: sentence-transformers>=3.0.0; extra == "codeflow-semantic"
 Provides-Extra: api
 Requires-Dist: fastapi>=0.115.0; extra == "api"
 Requires-Dist: uvicorn[standard]>=0.32.0; extra == "api"
@@ -122,6 +129,10 @@ Requires-Dist: pydantic-settings>=2.0.0; extra == "api"
 Provides-Extra: mcp
 Requires-Dist: mcp>=1.2.0; extra == "mcp"
 Requires-Dist: fastmcp>=2.3.0; extra == "mcp"
+Provides-Extra: skill-openai
+Requires-Dist: openai>=1.30.0; extra == "skill-openai"
+Provides-Extra: skill-rag-chroma
+Requires-Dist: chromadb>=0.4.22; extra == "skill-rag-chroma"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0.0; extra == "dev"
 Requires-Dist: httpx>=0.27.0; extra == "dev"
@@ -167,14 +178,14 @@ Dynamic: license-file
 
 # mdengine
 
-Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), and **graphs** (Neo4j / NetworkX → Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
+Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), **graphs** (Neo4j / NetworkX → Markdown), **OpenAPI** specs, **Playwright**-captured web pages (including SPAs), and **source code** (codeflow → architecture Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
 
 - **PyPI name:** `mdengine` (import package: `md_generator`)
 - **Source:** [github.com/vishal7090/md-generator](https://github.com/vishal7090/md-generator)
 - **Python:** 3.10+
 - **License:** [MIT](LICENSE)
 
-**Quick links:** [On a new computer](#on-a-new-computer) · [Command-line execution](#command-line-execution) · [Python library](#python-library) · [Audio and video](#audio-and-video-to-markdown) · [HTTP API](#http-api-fastapi) · [MCP](#mcp-model-context-protocol) · [Development](#development) · [Code of Conduct](CODE_OF_CONDUCT.md)
+**Quick links:** [On a new computer](#on-a-new-computer) · [Command-line execution](#command-line-execution) · [Python library](#python-library) · [Audio and video](#audio-and-video-to-markdown) · [HTTP API](#http-api-fastapi) · [MCP](#mcp-model-context-protocol) · [AI assistant CLI](#ai-assistant-cli) · [Development](#development) · [Code of Conduct](CODE_OF_CONDUCT.md)
 
 ---
 
@@ -248,6 +259,8 @@ pip install "mdengine[all]"
 | `graph` | **Graph → Markdown** (Neo4j Bolt + NetworkX GraphML/GML): `networkx`, `neo4j`, `pyyaml` |
 | `dev` | pytest + API/MCP test helpers |
 | `all` | Large superset of dependencies (use only if you need everything) |
+| `skill-openai` | Optional helpers for OpenAI-style calls with assembled skill context (`md_generator.tools.assistant`) |
+| `skill-rag-chroma` | Optional Chroma-based RAG over skill markdown (`MasterAgent.ask(..., use_rag=True)`) |
 
 Nested ZIP and office files inside archives require the corresponding extras (e.g. `archive` plus `pdf` for PDFs inside a ZIP).
 
@@ -304,7 +317,7 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-openapi` | `md_generator.openapi.cli.main:main` | `pip install "mdengine[openapi]"` then `md-openapi generate --file openapi.yaml --output ./docs` (or `mdengine openapi-to-md generate …`) |
 | `md-openapi-api` | `md_generator.openapi.api.run:main` | FastAPI on port **8015** (`OPENAPI_TO_MD_PORT`): `POST /openapi-to-md/generate` (OpenAPI upload → ZIP), `/health`, MCP at **`/mcp`** |
 | `md-openapi-mcp` | `md_generator.openapi.api.mcp_server:main` | Standalone MCP: `api_validate_openapi_yaml`, `api_generate_readme_markdown`, `api_run_sync_zip_base64` |
-| `mdengine` | `md_generator.engine_cli:main` | `mdengine db-to-md …` / `mdengine graph-to-md …` / `mdengine openapi-to-md generate …` |
+| `mdengine` | `md_generator.engine_cli:main` | `mdengine ai assist …`, `mdengine ai export …`, `mdengine skill build …`, `mdengine db-to-md …`, `mdengine graph-to-md …`, `mdengine openapi-to-md generate …` |
 
 **openapi-to-md (`md-openapi`):** OpenAPI **3.x** is parsed directly; **Swagger 2.0** (`swagger: "2.0"`) documents are **converted in-process** to OpenAPI 3.0.3 (deterministic, in-repo converter) before `$ref` resolution. Edge-heavy specs (unusual OAuth2 flows, vendor extensions) may still need fixes after conversion.
 
@@ -757,6 +770,110 @@ Install **`mdengine[mcp]`** (and usually **`[api]`** when using HTTP) for MCP-re
 
 ---
 
+## AI assistant CLI
+
+The repo ships **distributable AI skills** under [`ai/`](ai/) (Markdown skills, [`ai/registry.json`](ai/registry.json), routing, and [`ai/dependency-graph.json`](ai/dependency-graph.json)). A small SDK in [`src/md_generator/tools/assistant/`](src/md_generator/tools/assistant/) assembles **grounded context** from a natural-language query for Cursor, Claude, OpenAI-style hosts, or your own LLM pipeline.
+
+**Further detail:** [ai/README.md](ai/README.md) (layout, registry schema, master agent).
+
+**PyPI installs:** `mdengine ai assist` and `mdengine ai export` read the **bundled** skill data shipped with **`md_generator.tools.assistant`** unless **`MDENGINE_SKILL_AI_ROOT`** points at another directory (for example the repo’s [`ai/`](ai/) while developing). You do **not** need **`mdengine skill build`** unless you maintain this repository and want to regenerate that bundle.
+
+Script index (all `project.scripts`): [`ai/skills/mdengine-reference/references/entrypoints.md`](ai/skills/mdengine-reference/references/entrypoints.md).
+
+### Regenerate skills, graph, and bundled data
+
+**Maintainers / repo clones.** From the **repository root**, refresh generated files under [`ai/`](ai/) and copy the bundle into **`src/md_generator/tools/assistant/data/`** for packaging:
+
+```bash
+# Windows PowerShell
+$env:PYTHONPATH = "src"
+python -m md_generator.tools.skill_builder
+```
+
+```bash
+# macOS / Linux
+PYTHONPATH=src python -m md_generator.tools.skill_builder
+```
+
+After `pip install -e .`, you can run **`mdengine skill build`** from the repo root without setting `PYTHONPATH` (same as `python -m md_generator.tools.skill_builder` when `PYTHONPATH=src`).
+
+- Refreshes `ai/skills/**/SKILL.md`, `ai/skills/global-skill.md`, `ai/skills/modules/*.md`, `ai/registry.json` (routing), and `ai/dependency-graph.json`.
+- Copies the same bundle into **`src/md_generator/tools/assistant/data/`** for the installed wheel.
+
+**Incremental** (only area skills touched under `src/md_generator/` since a git ref; global/graph/registry still refresh):
+
+```bash
+mdengine skill build --since HEAD~1
+```
+
+```bash
+PYTHONPATH=src python -m md_generator.tools.skill_builder --since HEAD~1
+```
+
+**Custom repo root** (CI or non-default cwd):
+
+```bash
+mdengine skill build --root /path/to/md-generator
+```
+
+### Use the live `ai/` tree from a clone (optional)
+
+When developing skills, point the SDK at the repo’s `ai/` directory instead of the copy under `src/md_generator/tools/assistant/data/`:
+
+```powershell
+# PowerShell
+$env:MDENGINE_SKILL_AI_ROOT = (Resolve-Path ".\ai").Path
+```
+
+```bash
+export MDENGINE_SKILL_AI_ROOT="$(pwd)/ai"
+```
+
+### Console: `mdengine ai assist` and `mdengine ai export`
+
+After `pip install mdengine` (or `pip install -e .` from a clone), use the **`mdengine`** entry point (no separate AI script is installed).
+
+- **`mdengine ai assist`** — prints a single **Markdown** context (selected skills + global architecture) to **stdout**, meant to paste into an LLM or pipe to a file.
+- **`mdengine ai export`** — prints **host-shaped** output to stdout, or writes with **`-o` / `--output`**: OpenAI **`messages`** JSON (`--format openai`), a Claude-style project prompt (`claude`), or Cursor-style rules Markdown (`cursor`).
+
+**Flags (both subcommands):** **`--rag`** — use optional Chroma RAG when **`mdengine[skill-rag-chroma]`** is installed (falls back to full skills if Chroma is unavailable). **`--ai-root PATH`** — same effect as **`MDENGINE_SKILL_AI_ROOT`** for that run only.
+
+**Discover flags:** `mdengine ai assist --help` · `mdengine ai export --help`
+
+**`ai export` arguments:** **`--format`** `openai` \| `claude` \| `cursor` (required) · **`--query`** `…` (required) · **`-o` / `--output`** `file` (optional; default stdout).
+
+```bash
+mdengine ai assist "How do I run md-pdf on a file?"
+mdengine ai assist "openapi mcp" --rag
+mdengine ai export --format openai --query "md-db ERD" -o bundle.json
+mdengine ai export --format claude --query "playwright"
+mdengine ai export --format cursor --query "xlsx excel"
+```
+
+One-off override without env var: `mdengine ai assist "…" --ai-root /path/to/ai`.
+
+### Python API
+
+```python
+from md_generator.tools.assistant import MasterAgent, Registry
+
+reg = Registry.load_default()
+agent = MasterAgent(reg)
+result = agent.ask("openapi mcp", use_rag=False)
+print(result.resolved_skill_ids)
+print(result.context_markdown[:2000])
+```
+
+Optional RAG: install **`mdengine[skill-rag-chroma]`**, then `agent.ask(..., use_rag=True)` (falls back to full skills if Chroma is unavailable). The same optional stack is used by **`mdengine ai assist --rag`** and **`mdengine ai export … --rag`**.
+
+### Tests (skill SDK)
+
+```bash
+PYTHONPATH=src python -m pytest tool-assistant/tests -q
+```
+
+---
+
 ## Development
 
 ```bash
@@ -764,7 +881,30 @@ pip install -e ".[dev,all]"   # or a smaller subset of extras
 python -m pytest
 ```
 
-Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`), **`graph-to-md/tests/`**, and **`openapi-to-md/tests/`**; `pyproject.toml` configures `pythonpath = ["src"]` so `md_generator` resolves without a separate `PYTHONPATH`.
+Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`), **`graph-to-md/tests/`**, **`openapi-to-md/tests/`**, and **[`tool-assistant/tests/`](tool-assistant/tests/)** for the skill SDK; `pyproject.toml` sets `pythonpath = ["src"]` so **`md_generator`** (including **`md_generator.tools.assistant`**) resolves without a manual `PYTHONPATH` when you use `pytest` from the config.
+
+### Codeflow (`md_generator.codeflow`)
+
+Static call graphs and per-entry Markdown/Mermaid/JSON for **Python**, **Java** (Spring REST, Kafka listeners, **Liferay / JSR-286 portlets**), and other parsers. Output is **static only** (`graph-full.json` can include unresolved `unknown::*` callees).
+
+More detail: **[`codeflow-to-md/docs/graph-and-outputs.md`](codeflow-to-md/docs/graph-and-outputs.md)** (node/edge model, `graph-schema.json`, `methods/` layout).
+
+- **Formats:** pass **`--formats md,mermaid,json,html`** (comma-separated). **`--output`** is the **output directory**, not the format list.
+- **Graph schema export:** optional **`--emit-graph-schema`** (with `json` in formats) writes **`graph-schema.json`** — stable Node/Edge view with derived File/Class nodes and `CONTAINS` edges. **`graph-full.json`** stays unchanged for existing consumers.
+- **Markdown intelligence:** flow and entry Markdown include **Called By** and **Impact** lists (capped with **`--intelligence-list-cap`**, default 80).
+- **IR / CFG (optional):** **`--emit-cfg`** builds a language-agnostic IR and writes **`cfg.json`**, **`cfg.mmd`**, and appends a CFG Mermaid block to **`flow.md`** (when `md` is in formats). Cap with **`--cfg-max-nodes`** (default 500).
+- **Entries:** use `--entry` (comma-separated symbol ids), **`--entries-file`** (one id per line, `#` comments allowed), or rely on detectors. **`--emit-entry-per-method`** emits one output directory per method/entry symbol under **`methods/<slug>/`**; cap volume with **`--emit-entry-max`** (default cap **10000** when the flag is set and max is omitted). **`--entry-fallback`** `none` \| `roots` \| `first_n` controls behavior when nothing is detected (default **`roots`**, in-degree 0 symbols, limited by **`--entry-fallback-max`**).
+- **Java flow:** edges can record **branch context** from enclosing `if`, `switch`, or ternary expressions (`condition` / `labels` on edges). **Business rules** Markdown picks up validation-style **`@NotNull`** / **`@Size`** / **`@Valid`** (name heuristics) and throws such as **`IllegalArgumentException`**.
+- **JavaScript / TypeScript / TSX:** Tree-sitter adds **`condition_label`** from enclosing control flow, **`fr.branches`** for `if` / `switch`, and **`fr.rules`** for **`throw`** and common **`assert` / `console.assert`** calls (slice-scoped in the collector).
+- **C++ (tree-sitter path):** **`condition_label`** on calls, **`fr.branches`** for `if` / `switch`, plus **`rules/cpp_rules.py`** for **`throw`** / **`assert`** when methods appear in the slice. (Clang-only parses still get throw/assert via the same extractor when tree-sitter is installed.)
+- **Go:** the **`codeflow_go_dump`** helper emits **`condition`** on calls inside `if` / `switch` / `for` / `range`, and **`rules`** for **`panic`** / **`log.Fatal` / `Fatalf`**; **`fr.rules`** are merged slice-scoped.
+- **PHP:** the **`codeflow_php_dump`** helper adds **`condition`** on calls (from enclosing **`if` / `case`**), **`rules`** for **`throw`**, and **`branches`** for documentation. Install PHP + Composer deps under `codeflow-to-md/examples/codeflow_php_dump` (or mirror under `tools/codeflow_php_dump`) for PHP scans.
+- **Kinds / filters:** `--include` accepts aliases such as **`portlet`** / **`liferay`** for portlet entries.
+- **Operators:** each run writes **`scan-summary.md`** at the output root (counts, warnings, fallback mode) unless **`--no-scan-summary`** is set.
+
+```bash
+python -m md_generator.codeflow.cli.main scan path/to/src --output ./codeflow-out --lang java
+```
 
 ---
 
@@ -776,6 +916,10 @@ Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/test
 | `CODE_OF_CONDUCT.md` | [Contributor Covenant](https://www.contributor-covenant.org/) 2.1 |
 | `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/audio/`](src/md_generator/media/audio/) and [`media/video/`](src/md_generator/media/video/); **graph-to-md** under [`graph/`](src/md_generator/graph/) |
 | `pyproject.toml` | Packaging, extras, CLI entry points, pytest |
+| `ai/` | **Distributable AI skills** (`SKILL.md` trees, `registry.json`, `dependency-graph.json`); regenerate with `PYTHONPATH=src python -m md_generator.tools.skill_builder` or **`mdengine skill build`** (after `pip install -e .`) |
+| `src/md_generator/tools/skill_builder/` | **Skill generator** — scans `src/md_generator` and `pyproject.toml` |
+| `src/md_generator/tools/assistant/` | **Skill SDK** (`md_generator.tools.assistant`): `MasterAgent`, `Registry`, bundled `data/` copy of `ai/` |
+| `tool-assistant/tests/` | **Pytests** for the assistant SDK (`md_generator.tools.assistant`) |
 | `*-to-md/` | **Docs, tests, fixtures**, thin `converter.py` shims, some `run.py` helpers |
 | `README.md` | This document |
 

{mdengine-0.7.0 → mdengine-0.8.1}/README.md

@@ -1,13 +1,13 @@
 # mdengine
 
-Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), and **graphs** (Neo4j / NetworkX → Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
+Single Python distribution for converting **PDF**, **Word (.docx)**, **PowerPoint (.pptx)**, **Excel (.xlsx/.xlsm)**, **images** (OCR), **plain text / JSON / XML**, **ZIP archives**, **audio / video** (Whisper transcription → Markdown), **database metadata** (SQL + Mongo), **graphs** (Neo4j / NetworkX → Markdown), **OpenAPI** specs, **Playwright**-captured web pages (including SPAs), and **source code** (codeflow → architecture Markdown) into **Markdown** (and related assets). Install only the extras you need; everything imports under the **`md_generator`** package.
 
 - **PyPI name:** `mdengine` (import package: `md_generator`)
 - **Source:** [github.com/vishal7090/md-generator](https://github.com/vishal7090/md-generator)
 - **Python:** 3.10+
 - **License:** [MIT](LICENSE)
 
-**Quick links:** [On a new computer](#on-a-new-computer) · [Command-line execution](#command-line-execution) · [Python library](#python-library) · [Audio and video](#audio-and-video-to-markdown) · [HTTP API](#http-api-fastapi) · [MCP](#mcp-model-context-protocol) · [Development](#development) · [Code of Conduct](CODE_OF_CONDUCT.md)
+**Quick links:** [On a new computer](#on-a-new-computer) · [Command-line execution](#command-line-execution) · [Python library](#python-library) · [Audio and video](#audio-and-video-to-markdown) · [HTTP API](#http-api-fastapi) · [MCP](#mcp-model-context-protocol) · [AI assistant CLI](#ai-assistant-cli) · [Development](#development) · [Code of Conduct](CODE_OF_CONDUCT.md)
 
 ---
 
@@ -81,6 +81,8 @@ pip install "mdengine[all]"
 | `graph` | **Graph → Markdown** (Neo4j Bolt + NetworkX GraphML/GML): `networkx`, `neo4j`, `pyyaml` |
 | `dev` | pytest + API/MCP test helpers |
 | `all` | Large superset of dependencies (use only if you need everything) |
+| `skill-openai` | Optional helpers for OpenAI-style calls with assembled skill context (`md_generator.tools.assistant`) |
+| `skill-rag-chroma` | Optional Chroma-based RAG over skill markdown (`MasterAgent.ask(..., use_rag=True)`) |
 
 Nested ZIP and office files inside archives require the corresponding extras (e.g. `archive` plus `pdf` for PDFs inside a ZIP).
 
@@ -137,7 +139,7 @@ If the shell reports “command not found”, ensure the Python **Scripts** dire
 | `md-openapi` | `md_generator.openapi.cli.main:main` | `pip install "mdengine[openapi]"` then `md-openapi generate --file openapi.yaml --output ./docs` (or `mdengine openapi-to-md generate …`) |
 | `md-openapi-api` | `md_generator.openapi.api.run:main` | FastAPI on port **8015** (`OPENAPI_TO_MD_PORT`): `POST /openapi-to-md/generate` (OpenAPI upload → ZIP), `/health`, MCP at **`/mcp`** |
 | `md-openapi-mcp` | `md_generator.openapi.api.mcp_server:main` | Standalone MCP: `api_validate_openapi_yaml`, `api_generate_readme_markdown`, `api_run_sync_zip_base64` |
-| `mdengine` | `md_generator.engine_cli:main` | `mdengine db-to-md …` / `mdengine graph-to-md …` / `mdengine openapi-to-md generate …` |
+| `mdengine` | `md_generator.engine_cli:main` | `mdengine ai assist …`, `mdengine ai export …`, `mdengine skill build …`, `mdengine db-to-md …`, `mdengine graph-to-md …`, `mdengine openapi-to-md generate …` |
 
 **openapi-to-md (`md-openapi`):** OpenAPI **3.x** is parsed directly; **Swagger 2.0** (`swagger: "2.0"`) documents are **converted in-process** to OpenAPI 3.0.3 (deterministic, in-repo converter) before `$ref` resolution. Edge-heavy specs (unusual OAuth2 flows, vendor extensions) may still need fixes after conversion.
 
@@ -590,6 +592,110 @@ Install **`mdengine[mcp]`** (and usually **`[api]`** when using HTTP) for MCP-re
 
 ---
 
+## AI assistant CLI
+
+The repo ships **distributable AI skills** under [`ai/`](ai/) (Markdown skills, [`ai/registry.json`](ai/registry.json), routing, and [`ai/dependency-graph.json`](ai/dependency-graph.json)). A small SDK in [`src/md_generator/tools/assistant/`](src/md_generator/tools/assistant/) assembles **grounded context** from a natural-language query for Cursor, Claude, OpenAI-style hosts, or your own LLM pipeline.
+
+**Further detail:** [ai/README.md](ai/README.md) (layout, registry schema, master agent).
+
+**PyPI installs:** `mdengine ai assist` and `mdengine ai export` read the **bundled** skill data shipped with **`md_generator.tools.assistant`** unless **`MDENGINE_SKILL_AI_ROOT`** points at another directory (for example the repo’s [`ai/`](ai/) while developing). You do **not** need **`mdengine skill build`** unless you maintain this repository and want to regenerate that bundle.
+
+Script index (all `project.scripts`): [`ai/skills/mdengine-reference/references/entrypoints.md`](ai/skills/mdengine-reference/references/entrypoints.md).
+
+### Regenerate skills, graph, and bundled data
+
+**Maintainers / repo clones.** From the **repository root**, refresh generated files under [`ai/`](ai/) and copy the bundle into **`src/md_generator/tools/assistant/data/`** for packaging:
+
+```bash
+# Windows PowerShell
+$env:PYTHONPATH = "src"
+python -m md_generator.tools.skill_builder
+```
+
+```bash
+# macOS / Linux
+PYTHONPATH=src python -m md_generator.tools.skill_builder
+```
+
+After `pip install -e .`, you can run **`mdengine skill build`** from the repo root without setting `PYTHONPATH` (same as `python -m md_generator.tools.skill_builder` when `PYTHONPATH=src`).
+
+- Refreshes `ai/skills/**/SKILL.md`, `ai/skills/global-skill.md`, `ai/skills/modules/*.md`, `ai/registry.json` (routing), and `ai/dependency-graph.json`.
+- Copies the same bundle into **`src/md_generator/tools/assistant/data/`** for the installed wheel.
+
+**Incremental** (only area skills touched under `src/md_generator/` since a git ref; global/graph/registry still refresh):
+
+```bash
+mdengine skill build --since HEAD~1
+```
+
+```bash
+PYTHONPATH=src python -m md_generator.tools.skill_builder --since HEAD~1
+```
+
+**Custom repo root** (CI or non-default cwd):
+
+```bash
+mdengine skill build --root /path/to/md-generator
+```
+
+### Use the live `ai/` tree from a clone (optional)
+
+When developing skills, point the SDK at the repo’s `ai/` directory instead of the copy under `src/md_generator/tools/assistant/data/`:
+
+```powershell
+# PowerShell
+$env:MDENGINE_SKILL_AI_ROOT = (Resolve-Path ".\ai").Path
+```
+
+```bash
+export MDENGINE_SKILL_AI_ROOT="$(pwd)/ai"
+```
+
+### Console: `mdengine ai assist` and `mdengine ai export`
+
+After `pip install mdengine` (or `pip install -e .` from a clone), use the **`mdengine`** entry point (no separate AI script is installed).
+
+- **`mdengine ai assist`** — prints a single **Markdown** context (selected skills + global architecture) to **stdout**, meant to paste into an LLM or pipe to a file.
+- **`mdengine ai export`** — prints **host-shaped** output to stdout, or writes with **`-o` / `--output`**: OpenAI **`messages`** JSON (`--format openai`), a Claude-style project prompt (`claude`), or Cursor-style rules Markdown (`cursor`).
+
+**Flags (both subcommands):** **`--rag`** — use optional Chroma RAG when **`mdengine[skill-rag-chroma]`** is installed (falls back to full skills if Chroma is unavailable). **`--ai-root PATH`** — same effect as **`MDENGINE_SKILL_AI_ROOT`** for that run only.
+
+**Discover flags:** `mdengine ai assist --help` · `mdengine ai export --help`
+
+**`ai export` arguments:** **`--format`** `openai` \| `claude` \| `cursor` (required) · **`--query`** `…` (required) · **`-o` / `--output`** `file` (optional; default stdout).
+
+```bash
+mdengine ai assist "How do I run md-pdf on a file?"
+mdengine ai assist "openapi mcp" --rag
+mdengine ai export --format openai --query "md-db ERD" -o bundle.json
+mdengine ai export --format claude --query "playwright"
+mdengine ai export --format cursor --query "xlsx excel"
+```
+
+One-off override without env var: `mdengine ai assist "…" --ai-root /path/to/ai`.
+
+### Python API
+
+```python
+from md_generator.tools.assistant import MasterAgent, Registry
+
+reg = Registry.load_default()
+agent = MasterAgent(reg)
+result = agent.ask("openapi mcp", use_rag=False)
+print(result.resolved_skill_ids)
+print(result.context_markdown[:2000])
+```
+
+Optional RAG: install **`mdengine[skill-rag-chroma]`**, then `agent.ask(..., use_rag=True)` (falls back to full skills if Chroma is unavailable). The same optional stack is used by **`mdengine ai assist --rag`** and **`mdengine ai export … --rag`**.
+
+### Tests (skill SDK)
+
+```bash
+PYTHONPATH=src python -m pytest tool-assistant/tests -q
+```
+
+---
+
 ## Development
 
 ```bash
@@ -597,7 +703,30 @@ pip install -e ".[dev,all]"   # or a smaller subset of extras
 python -m pytest
 ```
 
-Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`), **`graph-to-md/tests/`**, and **`openapi-to-md/tests/`**; `pyproject.toml` configures `pythonpath = ["src"]` so `md_generator` resolves without a separate `PYTHONPATH`.
+Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/tests/`), **`graph-to-md/tests/`**, **`openapi-to-md/tests/`**, and **[`tool-assistant/tests/`](tool-assistant/tests/)** for the skill SDK; `pyproject.toml` sets `pythonpath = ["src"]` so **`md_generator`** (including **`md_generator.tools.assistant`**) resolves without a manual `PYTHONPATH` when you use `pytest` from the config.
+
+### Codeflow (`md_generator.codeflow`)
+
+Static call graphs and per-entry Markdown/Mermaid/JSON for **Python**, **Java** (Spring REST, Kafka listeners, **Liferay / JSR-286 portlets**), and other parsers. Output is **static only** (`graph-full.json` can include unresolved `unknown::*` callees).
+
+More detail: **[`codeflow-to-md/docs/graph-and-outputs.md`](codeflow-to-md/docs/graph-and-outputs.md)** (node/edge model, `graph-schema.json`, `methods/` layout).
+
+- **Formats:** pass **`--formats md,mermaid,json,html`** (comma-separated). **`--output`** is the **output directory**, not the format list.
+- **Graph schema export:** optional **`--emit-graph-schema`** (with `json` in formats) writes **`graph-schema.json`** — stable Node/Edge view with derived File/Class nodes and `CONTAINS` edges. **`graph-full.json`** stays unchanged for existing consumers.
+- **Markdown intelligence:** flow and entry Markdown include **Called By** and **Impact** lists (capped with **`--intelligence-list-cap`**, default 80).
+- **IR / CFG (optional):** **`--emit-cfg`** builds a language-agnostic IR and writes **`cfg.json`**, **`cfg.mmd`**, and appends a CFG Mermaid block to **`flow.md`** (when `md` is in formats). Cap with **`--cfg-max-nodes`** (default 500).
+- **Entries:** use `--entry` (comma-separated symbol ids), **`--entries-file`** (one id per line, `#` comments allowed), or rely on detectors. **`--emit-entry-per-method`** emits one output directory per method/entry symbol under **`methods/<slug>/`**; cap volume with **`--emit-entry-max`** (default cap **10000** when the flag is set and max is omitted). **`--entry-fallback`** `none` \| `roots` \| `first_n` controls behavior when nothing is detected (default **`roots`**, in-degree 0 symbols, limited by **`--entry-fallback-max`**).
+- **Java flow:** edges can record **branch context** from enclosing `if`, `switch`, or ternary expressions (`condition` / `labels` on edges). **Business rules** Markdown picks up validation-style **`@NotNull`** / **`@Size`** / **`@Valid`** (name heuristics) and throws such as **`IllegalArgumentException`**.
+- **JavaScript / TypeScript / TSX:** Tree-sitter adds **`condition_label`** from enclosing control flow, **`fr.branches`** for `if` / `switch`, and **`fr.rules`** for **`throw`** and common **`assert` / `console.assert`** calls (slice-scoped in the collector).
+- **C++ (tree-sitter path):** **`condition_label`** on calls, **`fr.branches`** for `if` / `switch`, plus **`rules/cpp_rules.py`** for **`throw`** / **`assert`** when methods appear in the slice. (Clang-only parses still get throw/assert via the same extractor when tree-sitter is installed.)
+- **Go:** the **`codeflow_go_dump`** helper emits **`condition`** on calls inside `if` / `switch` / `for` / `range`, and **`rules`** for **`panic`** / **`log.Fatal` / `Fatalf`**; **`fr.rules`** are merged slice-scoped.
+- **PHP:** the **`codeflow_php_dump`** helper adds **`condition`** on calls (from enclosing **`if` / `case`**), **`rules`** for **`throw`**, and **`branches`** for documentation. Install PHP + Composer deps under `codeflow-to-md/examples/codeflow_php_dump` (or mirror under `tools/codeflow_php_dump`) for PHP scans.
+- **Kinds / filters:** `--include` accepts aliases such as **`portlet`** / **`liferay`** for portlet entries.
+- **Operators:** each run writes **`scan-summary.md`** at the output root (counts, warnings, fallback mode) unless **`--no-scan-summary`** is set.
+
+```bash
+python -m md_generator.codeflow.cli.main scan path/to/src --output ./codeflow-out --lang java
+```
 
 ---
 
@@ -609,6 +738,10 @@ Tests live under each legacy folder’s `tests/` directory (e.g. `pdf-to-md/test
 | `CODE_OF_CONDUCT.md` | [Contributor Covenant](https://www.contributor-covenant.org/) 2.1 |
 | `src/md_generator/` | **Library source** (all formats + `api` subpackages); **audio/video** under [`media/audio/`](src/md_generator/media/audio/) and [`media/video/`](src/md_generator/media/video/); **graph-to-md** under [`graph/`](src/md_generator/graph/) |
 | `pyproject.toml` | Packaging, extras, CLI entry points, pytest |
+| `ai/` | **Distributable AI skills** (`SKILL.md` trees, `registry.json`, `dependency-graph.json`); regenerate with `PYTHONPATH=src python -m md_generator.tools.skill_builder` or **`mdengine skill build`** (after `pip install -e .`) |
+| `src/md_generator/tools/skill_builder/` | **Skill generator** — scans `src/md_generator` and `pyproject.toml` |
+| `src/md_generator/tools/assistant/` | **Skill SDK** (`md_generator.tools.assistant`): `MasterAgent`, `Registry`, bundled `data/` copy of `ai/` |
+| `tool-assistant/tests/` | **Pytests** for the assistant SDK (`md_generator.tools.assistant`) |
 | `*-to-md/` | **Docs, tests, fixtures**, thin `converter.py` shims, some `run.py` helpers |
 | `README.md` | This document |
 

{mdengine-0.7.0 → mdengine-0.8.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mdengine"
-version = "0.7.0"
+version = "0.8.1"
 description = "Convert PDF, Office, images, audio, video, text/JSON/XML, ZIP archives, web URLs, databases, graphs, and OpenAPI specs to Markdown."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -115,6 +115,11 @@ codeflow = [
   "networkx>=3.2.0",
   "javalang>=0.13.0",
 ]
+# Optional: horizontal workers for codeflow API (Celery + Redis). Default job path uses threads.
+codeflow-worker = [
+  "celery>=5.3.0",
+  "redis>=5.0.0",
+]
 codeflow-treesitter = [
   "tree-sitter>=0.21.0",
   "tree-sitter-javascript>=0.21.0",
@@ -124,6 +129,12 @@ codeflow-treesitter = [
 codeflow-clang = [
   "clang>=16.0.0",
 ]
+# Embeddings + semantic clustering (SentenceTransformers pulls torch; large install).
+codeflow-semantic = [
+  "numpy>=1.24.0",
+  "scikit-learn>=1.3.0",
+  "sentence-transformers>=3.0.0",
+]
 api = [
   "fastapi>=0.115.0",
   "uvicorn[standard]>=0.32.0",
@@ -132,6 +143,9 @@ api = [
   "pydantic-settings>=2.0.0",
 ]
 mcp = ["mcp>=1.2.0", "fastmcp>=2.3.0"]
+# Optional: AI skill bundle helpers (md_generator.tools.assistant)
+skill-openai = ["openai>=1.30.0"]
+skill-rag-chroma = ["chromadb>=0.4.22"]
 dev = [
   "pytest>=8.0.0",
   "httpx>=0.27.0",
@@ -219,15 +233,18 @@ mdengine = "md_generator.engine_cli:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]
+include = ["md_generator*"]
 
 [tool.setuptools.package-data]
 "md_generator.db.config" = ["*.yaml"]
 "md_generator.graph.config" = ["*.yaml"]
 "md_generator.openapi.config" = ["*.yaml"]
+"md_generator.tools.assistant" = ["data/**/*"]
 
 [tool.pytest.ini_options]
 pythonpath = ["src"]
 testpaths = [
+  "tool-assistant/tests",
   "pdf-to-md/tests",
   "word-to-md/tests",
   "ppt-to-md/tests",

{mdengine-0.7.0 → mdengine-0.8.1}/src/md_generator/__init__.py

@@ -1,5 +1,5 @@
 """md_generator: PDF, Office, image, text, and ZIP to Markdown conversion."""
 
-__version__ = "0.7.0"
+__version__ = "0.8.1"
 
 __all__ = ["__version__"]

{mdengine-0.7.0 → mdengine-0.8.1}/src/md_generator/codeflow/analyzers/flow_analyzer.py

@@ -4,6 +4,9 @@ from dataclasses import dataclass, field
 
 import networkx as nx
 
+from md_generator.codeflow.graph import relations as rel
+from md_generator.codeflow.graph.multigraph_utils import CodeflowGraph, collapsed_digraph_for_relations
+
 
 @dataclass(slots=True)
 class FlowSlice:
@@ -15,16 +18,25 @@ class FlowSlice:
     truncated: bool = False
 
 
+_DEFAULT_FLOW_RELATIONS = frozenset({rel.REL_CALLS})
+
+
 def walk_with_depth(
-    g: nx.DiGraph,
+    g: CodeflowGraph,
     start: str,
     max_depth: int,
+    *,
+    relations: frozenset[str] | None = None,
 ) -> tuple[set[str], list[tuple[str, str, dict]], set[str], bool]:
-    """Reachability from start within ``max_depth`` hops (shortest-path layering)."""
+    """Reachability from ``start`` within ``max_depth`` hops over selected relations (default: CALLS only)."""
     if start not in g:
         return {start}, [], set(), False
+    rels = relations if relations is not None else _DEFAULT_FLOW_RELATIONS
+    cg = collapsed_digraph_for_relations(g, rels)
+    if start not in cg:
+        return {start}, [], set(), False
     try:
-        plen = nx.single_source_shortest_path_length(g, start, cutoff=max_depth)
+        plen = nx.single_source_shortest_path_length(cg, start, cutoff=max_depth)
     except nx.NetworkXError:
         plen = {start: 0}
 
@@ -36,13 +48,18 @@ def walk_with_depth(
     for u in nodes:
         du = plen[u]
         if du >= max_depth:
-            succ = list(g.successors(u))
+            succ = list(cg.successors(u))
             if succ:
                 truncated = True
             continue
-        for v in g.successors(u):
-            ed = dict(g.edges[u, v])
-            edges_out.append((u, v, ed))
+        for v in cg.successors(u):
+            if not g.has_edge(u, v):
+                continue
+            for _k, ed in g[u][v].items():
+                er = ed.get("relation") or ed.get("kind") or rel.REL_CALLS
+                if er not in rels:
+                    continue
+                edges_out.append((u, v, dict(ed)))
             dv = plen.get(v)
             if dv is not None and dv <= du:
                 cycle_guess.add(v)
@@ -53,11 +70,13 @@ def walk_with_depth(
 
 
 def slice_from_entry(
-    g: nx.DiGraph,
+    g: CodeflowGraph,
     entry_id: str,
     max_depth: int,
+    *,
+    relations: frozenset[str] | None = None,
 ) -> FlowSlice:
-    nodes, edges, cycles, truncated = walk_with_depth(g, entry_id, max_depth)
+    nodes, edges, cycles, truncated = walk_with_depth(g, entry_id, max_depth, relations=relations)
     return FlowSlice(
         entry_id=entry_id,
         nodes=nodes,

{mdengine-0.7.0 → mdengine-0.8.1}/src/md_generator/codeflow/api/job_manager.py

@@ -7,7 +7,7 @@ import threading
 import time
 import uuid
 import zipfile
-from dataclasses import dataclass
+from dataclasses import dataclass, replace
 from pathlib import Path
 from typing import Any
 
@@ -101,24 +101,7 @@ class CodeflowJobManager:
             zf.extractall(src)
 
         out = ws / "out"
-        cfg = ScanConfig(
-            project_root=src,
-            output_path=out,
-            formats=cfg_template.formats,
-            depth=cfg_template.depth,
-            languages=cfg_template.languages,
-            entry=cfg_template.entry,
-            include=cfg_template.include,
-            exclude=cfg_template.exclude,
-            include_internal=cfg_template.include_internal,
-            async_mode=cfg_template.async_mode,
-            jobs=True,
-            runtime=cfg_template.runtime,
-            paths_override=cfg_template.paths_override,
-            business_rules=cfg_template.business_rules,
-            business_rules_sql=cfg_template.business_rules_sql,
-            business_rules_combined=cfg_template.business_rules_combined,
-        )
+        cfg = replace(cfg_template, project_root=src, output_path=out, jobs=True)
         now = time.time()
         cfg_dump = json.dumps(scan_config_dump(cfg), sort_keys=True)
         assert self._conn is not None