markdownizer 0.2.1__tar.gz → 0.3.0__tar.gz

{markdownizer-0.2.1 → markdownizer-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: markdownizer
-Version: 0.2.1
+Version: 0.3.0
 Summary: Extract documentation from Python projects into Markdown without rewriting it
 Author-email: Mohammad Hasan Khoddami <mohammadh.khoddami@gmail.com>
 License: MIT License
@@ -89,8 +89,17 @@ Requires Python 3.9+. No runtime dependencies.
 markdownizer /path/to/project -o ./docs
 ```
 
-This recursively scans the project, parses every Python file with the AST, and
-writes one Markdown file per package into `./docs`.
+This recursively scans the project, parses every Python file with the AST,
+builds a deterministic Project IR, and writes one Markdown file per package
+into `./docs`.
+
+The compiler-style form is equivalent and supports format selection:
+
+```bash
+markdownizer build /path/to/project -o ./docs --format markdown
+markdownizer build /path/to/project -o ./docs --format json
+markdownizer build /path/to/project -o ./docs --format compact
+```
 
 Common options:
 
@@ -105,6 +114,7 @@ markdownizer . -o ./docs \
 | `-o, --output DIR` | Output directory (default: `./docs`) |
 | `--root-name NAME` | Filename for files at the project root (default: `_root`) |
 | `--exclude GLOB` | Skip matching paths; may be repeated |
+| `--format FMT` | Output backend: `markdown`, `json`, or `compact` |
 | `--no-source` | Omit the `## Source Code` section |
 | `--no-comments` | Omit the `## Comments` section |
 | `--only-documented` | Only include objects with a docstring |
@@ -118,7 +128,7 @@ Run `markdownizer --help` for the full list.
 
 ```python
 from pathlib import Path
-from markdownizer import extract_project
+from markdownizer import extract_project, build_project_ir
 
 written = extract_project(
     Path("."),
@@ -126,9 +136,60 @@ written = extract_project(
     exclude=["tests/*"],
     include_source=False,
 )
-print(written)  # list of written Markdown files
+print(written)  # list of written output files
+
+# Or build the Project IR directly:
+ir = build_project_ir(Path("."), exclude=["tests/*"])
+print(ir.ir_version, ir.hash, ir.stats.symbol_count)
+```
+
+### Output formats
+
+The pipeline builds a deterministic **Project IR** (packages → modules →
+symbols, plus import/inherit/define edges) and renders it through a backend:
+
+| Format | Command | Output |
+| --- | --- | --- |
+| `markdown` (default) | `markdownizer build . -o ./docs` | One `.md` file per package |
+| `json` | `markdownizer build . -o ./docs --format json` | `project.json` — full IR serialization |
+| `compact` | `markdownizer build . -o ./docs --format compact` | `context.compact.md` — signatures, docstrings, inheritance, decorators (no bodies) |
+
+The legacy invocation `markdownizer <project> -o <out>` is kept as a
+compatibility alias for `markdownizer build <project> --format markdown`.
+
+### Signature mode
+
+Instead of full source or no source, `extract_project()` accepts
+`include_source="signature"` to emit only declaration lines:
+
+```python
+extract_project(Path("."), Path("docs"), include_source="signature")
 ```
 
+Functions render as `def foo(x: int = 1) -> str:`, async functions as
+`async def ...`, classes as `class User(models.Model):` (with base classes),
+and methods with their parameters. Modules render without source. The
+boolean modes (`True`/`False`) are unchanged.
+
+### Project IR
+
+`build_project_ir(project_root, exclude=None)` returns a `ProjectIR` with:
+
+- `ir_version` — schema version (currently `1`), independent of the package version
+- `packages`, `modules`, `symbols` — the project hierarchy
+- `imports`, `inherits`, `defines` — relationship edges
+- `stats` — file/module/symbol counts
+- `hash` — deterministic `blake2b` of the canonical IR content
+
+The hash and JSON serialization are deterministic: the same repository
+content always produces the same hash and the same `project.json`, making
+the output suitable for version control and caching. Machine-specific
+metadata (`root`, `python_version`, `git`) is excluded from the hash.
+
+Import resolution is conservative and fully static: project code is never
+imported or executed. Imports that cannot be resolved to a project module
+are marked `external`.
+
 ## What is extracted
 
 For every documented object (modules, packages, classes, dataclasses, enums,

{markdownizer-0.2.1 → markdownizer-0.3.0}/README.md

@@ -33,8 +33,17 @@ Requires Python 3.9+. No runtime dependencies.
 markdownizer /path/to/project -o ./docs
 ```
 
-This recursively scans the project, parses every Python file with the AST, and
-writes one Markdown file per package into `./docs`.
+This recursively scans the project, parses every Python file with the AST,
+builds a deterministic Project IR, and writes one Markdown file per package
+into `./docs`.
+
+The compiler-style form is equivalent and supports format selection:
+
+```bash
+markdownizer build /path/to/project -o ./docs --format markdown
+markdownizer build /path/to/project -o ./docs --format json
+markdownizer build /path/to/project -o ./docs --format compact
+```
 
 Common options:
 
@@ -49,6 +58,7 @@ markdownizer . -o ./docs \
 | `-o, --output DIR` | Output directory (default: `./docs`) |
 | `--root-name NAME` | Filename for files at the project root (default: `_root`) |
 | `--exclude GLOB` | Skip matching paths; may be repeated |
+| `--format FMT` | Output backend: `markdown`, `json`, or `compact` |
 | `--no-source` | Omit the `## Source Code` section |
 | `--no-comments` | Omit the `## Comments` section |
 | `--only-documented` | Only include objects with a docstring |
@@ -62,7 +72,7 @@ Run `markdownizer --help` for the full list.
 
 ```python
 from pathlib import Path
-from markdownizer import extract_project
+from markdownizer import extract_project, build_project_ir
 
 written = extract_project(
     Path("."),
@@ -70,9 +80,60 @@ written = extract_project(
     exclude=["tests/*"],
     include_source=False,
 )
-print(written)  # list of written Markdown files
+print(written)  # list of written output files
+
+# Or build the Project IR directly:
+ir = build_project_ir(Path("."), exclude=["tests/*"])
+print(ir.ir_version, ir.hash, ir.stats.symbol_count)
+```
+
+### Output formats
+
+The pipeline builds a deterministic **Project IR** (packages → modules →
+symbols, plus import/inherit/define edges) and renders it through a backend:
+
+| Format | Command | Output |
+| --- | --- | --- |
+| `markdown` (default) | `markdownizer build . -o ./docs` | One `.md` file per package |
+| `json` | `markdownizer build . -o ./docs --format json` | `project.json` — full IR serialization |
+| `compact` | `markdownizer build . -o ./docs --format compact` | `context.compact.md` — signatures, docstrings, inheritance, decorators (no bodies) |
+
+The legacy invocation `markdownizer <project> -o <out>` is kept as a
+compatibility alias for `markdownizer build <project> --format markdown`.
+
+### Signature mode
+
+Instead of full source or no source, `extract_project()` accepts
+`include_source="signature"` to emit only declaration lines:
+
+```python
+extract_project(Path("."), Path("docs"), include_source="signature")
 ```
 
+Functions render as `def foo(x: int = 1) -> str:`, async functions as
+`async def ...`, classes as `class User(models.Model):` (with base classes),
+and methods with their parameters. Modules render without source. The
+boolean modes (`True`/`False`) are unchanged.
+
+### Project IR
+
+`build_project_ir(project_root, exclude=None)` returns a `ProjectIR` with:
+
+- `ir_version` — schema version (currently `1`), independent of the package version
+- `packages`, `modules`, `symbols` — the project hierarchy
+- `imports`, `inherits`, `defines` — relationship edges
+- `stats` — file/module/symbol counts
+- `hash` — deterministic `blake2b` of the canonical IR content
+
+The hash and JSON serialization are deterministic: the same repository
+content always produces the same hash and the same `project.json`, making
+the output suitable for version control and caching. Machine-specific
+metadata (`root`, `python_version`, `git`) is excluded from the hash.
+
+Import resolution is conservative and fully static: project code is never
+imported or executed. Imports that cannot be resolved to a project module
+are marked `external`.
+
 ## What is extracted
 
 For every documented object (modules, packages, classes, dataclasses, enums,

{markdownizer-0.2.1 → markdownizer-0.3.0}/markdownizer/__init__.py

@@ -5,6 +5,7 @@ It only extracts what already exists in the source code.
 """
 
 from markdownizer.extractor import extract_project
+from markdownizer.ir import IR_VERSION, ProjectIR, build_project_ir
 
-__version__ = "0.2.1"
-__all__ = ["extract_project"]
+__version__ = "0.3.0"
+__all__ = ["extract_project", "build_project_ir", "IR_VERSION", "ProjectIR"]

markdownizer-0.3.0/markdownizer/backends/__init__.py

@@ -0,0 +1,26 @@
+"""Output backends for the Project IR."""
+
+from __future__ import annotations
+
+from markdownizer.backends.base import Backend, RenderOptions
+from markdownizer.backends.compact import CompactBackend
+from markdownizer.backends.json import JSONBackend
+from markdownizer.backends.markdown import MarkdownBackend
+
+_BACKENDS: dict[str, type[Backend]] = {
+    "markdown": MarkdownBackend,
+    "json": JSONBackend,
+    "compact": CompactBackend,
+}
+
+
+def get_backend(name: str) -> Backend:
+    """Instantiate a backend by name (``markdown``, ``json``, or ``compact``)."""
+    try:
+        return _BACKENDS[name]()
+    except KeyError:
+        choices = ", ".join(sorted(_BACKENDS))
+        raise ValueError(f"unknown format: {name!r} (expected one of: {choices})") from None
+
+
+__all__ = ["Backend", "RenderOptions", "get_backend"]

markdownizer-0.3.0/markdownizer/backends/base.py

@@ -0,0 +1,36 @@
+"""Backend abstraction: deterministic rendering of a ProjectIR into output files."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Literal, Protocol, Union
+
+from markdownizer.ir import ProjectIR
+
+SourceMode = Union[bool, Literal["signature"]]
+
+
+@dataclass
+class RenderOptions:
+    """Options controlling how a backend renders an IR.
+
+    ``include_source`` may be ``True`` (full source), ``False`` (no source),
+    or ``"signature"`` (declaration line only).
+    """
+
+    root_name: str = "_root"
+    include_source: SourceMode = True
+    include_comments: bool = True
+    include_undocumented: bool = True
+
+
+class Backend(Protocol):
+    """A backend renders a ProjectIR into ``{filename: content}`` pairs.
+
+    Implementations must be deterministic: the same IR always produces the
+    same output.
+    """
+
+    name: str
+
+    def render(self, ir: ProjectIR, options: RenderOptions) -> dict[str, str]: ...

markdownizer-0.3.0/markdownizer/backends/compact.py

@@ -0,0 +1,75 @@
+"""Compact backend: a low-volume, structure-oriented representation.
+
+Contains packages, modules, public symbols, signatures, inheritance,
+decorators, and docstrings — without full source bodies. The output is
+deterministic and intended as a high-signal overview for AI consumption.
+"""
+
+from __future__ import annotations
+
+from markdownizer.backends.base import RenderOptions
+from markdownizer.ir import ModuleInfo, ProjectIR, Symbol
+from markdownizer.renderer import format_signature
+
+
+def _symbol_line(symbol: Symbol, options: RenderOptions) -> list[str]:
+    lines = [f"# {symbol.framework}: {symbol.qualified_name} — {symbol.file_path}:{symbol.lineno}"]
+    if options.include_source is not False:
+        signature = format_signature(symbol)
+        if signature:
+            lines.append(f"```{signature}```")
+    if symbol.base_classes:
+        lines.append(f"inherits: {', '.join(symbol.base_classes)}")
+    if symbol.decorators:
+        lines.append(f"decorators: {', '.join(symbol.decorators)}")
+    if symbol.docstring not in (None, ""):
+        lines.append("")
+        lines.append(symbol.docstring.rstrip())
+    lines.append("")
+    return lines
+
+
+class CompactBackend:
+    """Render a single structure-oriented ``context.compact.md`` file."""
+
+    name = "compact"
+
+    def render(self, ir: ProjectIR, options: RenderOptions) -> dict[str, str]:
+        lines = [
+            f"# Project: {ir.name} (IR v{ir.ir_version})",
+            "",
+            (
+                f"# Packages: {ir.stats.package_count} | "
+                f"Modules: {ir.stats.module_count} | "
+                f"Symbols: {ir.stats.symbol_count} | "
+                f"Lines: {ir.stats.line_count}"
+            ),
+            "",
+        ]
+
+        modules_by_package: dict[str, list[ModuleInfo]] = {}
+        symbols_by_package: dict[str, list[Symbol]] = {}
+        for module in ir.modules:
+            modules_by_package.setdefault(module.package, []).append(module)
+        for symbol in ir.symbols:
+            package = symbol.file_path.rsplit("/", 1)[0] if "/" in symbol.file_path else ""
+            symbols_by_package.setdefault(package, []).append(symbol)
+
+        for package in sorted(set(modules_by_package) | set(symbols_by_package)):
+            lines.append(f"# Package: {package if package else options.root_name}")
+            lines.append("")
+            for module in modules_by_package.get(package, []):
+                lines.append(f"# Module: {module.path}")
+                if module.docstring not in (None, ""):
+                    lines.append("")
+                    lines.append(module.docstring.rstrip())
+                lines.append("")
+            symbols = symbols_by_package.get(package, [])
+            if not options.include_undocumented:
+                symbols = [s for s in symbols if s.docstring not in (None, "")]
+            for symbol in symbols:
+                if not symbol.is_public:
+                    continue
+                lines.extend(_symbol_line(symbol, options))
+
+        return {"context.compact.md": "\n".join(lines).rstrip() + "\n"}

markdownizer-0.3.0/markdownizer/backends/json.py

@@ -0,0 +1,19 @@
+"""JSON backend: deterministic serialization of the Project IR."""
+
+from __future__ import annotations
+
+from markdownizer.backends.base import RenderOptions
+from markdownizer.ir import ProjectIR
+
+
+class JSONBackend:
+    """Render the full ProjectIR as ``project.json``.
+
+    This backend serializes the complete IR regardless of ``RenderOptions``;
+    it is a faithful machine-readable representation of the analysis.
+    """
+
+    name = "json"
+
+    def render(self, ir: ProjectIR, options: RenderOptions) -> dict[str, str]:
+        return {"project.json": ir.to_json() + "\n"}

markdownizer-0.3.0/markdownizer/backends/markdown.py

@@ -0,0 +1,62 @@
+"""Markdown backend: one Markdown file per package (the historical output).
+
+The rendering logic lives in :mod:`markdownizer.renderer`; this backend
+groups the IR by package and drives it.
+"""
+
+from __future__ import annotations
+
+from markdownizer.backends.base import RenderOptions
+from markdownizer.ir import ModuleInfo, ProjectIR, Symbol
+from markdownizer.parser import DocObject
+from markdownizer.renderer import render_package_markdown
+
+
+def _module_docobject(module: ModuleInfo) -> DocObject:
+    """Adapt an IR module back into a DocObject for the renderer."""
+    return DocObject(
+        name=module.path,
+        qualified_name=module.path,
+        kind="module",
+        lineno=1,
+        end_lineno=module.line_count,
+        anchor_lineno=1,
+        docstring=module.docstring,
+        source=module.source,
+        inline_comments=list(module.inline_comments),
+        file_path=module.path,
+    )
+
+
+class MarkdownBackend:
+    """Render one ``{package}.md`` file per package."""
+
+    name = "markdown"
+
+    def render(self, ir: ProjectIR, options: RenderOptions) -> dict[str, str]:
+        modules_by_package: dict[str, list[ModuleInfo]] = {}
+        symbols_by_package: dict[str, list[Symbol]] = {}
+
+        for module in ir.modules:
+            modules_by_package.setdefault(module.package, []).append(module)
+        for symbol in ir.symbols:
+            package = symbol.file_path.rsplit("/", 1)[0] if "/" in symbol.file_path else ""
+            symbols_by_package.setdefault(package, []).append(symbol)
+
+        out: dict[str, str] = {}
+        for package in sorted(set(modules_by_package) | set(symbols_by_package)):
+            label = package.replace("/", ".") if package else options.root_name
+            modules = modules_by_package.get(package, [])
+            symbols = symbols_by_package.get(package, [])
+            if not options.include_undocumented:
+                symbols = [s for s in symbols if s.docstring not in (None, "")]
+            markdown = render_package_markdown(
+                label,
+                [_module_docobject(m) for m in modules],
+                symbols,  # Symbols duck-type DocObject for the renderer
+                None,
+                include_source=options.include_source,
+                include_comments=options.include_comments,
+            )
+            out[f"{label}.md"] = markdown
+        return out

{markdownizer-0.2.1 → markdownizer-0.3.0}/markdownizer/classifier.py

@@ -1,8 +1,8 @@
-"""Classify a DocObject into a specialized documentation header kind."""
+"""Classify a renderable object into a specialized documentation header kind."""
 
 from __future__ import annotations
 
-from markdownizer.parser import DocObject
+from markdownizer.parser import Renderable
 
 _DJANGO_MODEL_BASES = {"Model", "AbstractUser", "AbstractBaseUser", "PermissionsMixin"}
 
@@ -69,7 +69,7 @@ def _decorator_base(dec: str) -> str:
     return _rightmost(head)
 
 
-def classify(obj: DocObject) -> str:
+def classify(obj: Renderable) -> str:
     """Return a human-readable header label like 'DRF Serializer' or 'Async Function'."""
     if obj.kind == "module":
         if obj.file_path.endswith("__init__.py"):

{markdownizer-0.2.1 → markdownizer-0.3.0}/markdownizer/cli.py

@@ -13,22 +13,13 @@ from markdownizer.extractor import extract_project
 logger = logging.getLogger(__name__)
 
 
-def build_parser() -> argparse.ArgumentParser:
-    parser = argparse.ArgumentParser(
-        prog="markdownizer",
-        description="Extract existing documentation from a Python project into Markdown files.",
-    )
-    parser.add_argument(
-        "project",
-        type=Path,
-        help="Path to the Python project root to scan.",
-    )
+def _add_common_arguments(parser: argparse.ArgumentParser) -> None:
     parser.add_argument(
         "-o",
         "--output",
         type=Path,
         default=Path("docs"),
-        help="Directory where generated Markdown files will be written (default: ./docs).",
+        help="Directory where generated files will be written (default: ./docs).",
     )
     parser.add_argument(
         "--root-name",
@@ -59,6 +50,15 @@ def build_parser() -> argparse.ArgumentParser:
         action="store_true",
         help="Only include objects that have a docstring.",
     )
+    parser.add_argument(
+        "--format",
+        choices=["markdown", "json", "compact"],
+        default="markdown",
+        help="Output backend (default: markdown).",
+    )
+
+
+def _add_verbose_arguments(parser: argparse.ArgumentParser) -> None:
     parser.add_argument(
         "-v",
         "--verbose",
@@ -72,6 +72,25 @@ def build_parser() -> argparse.ArgumentParser:
         action="store_true",
         help="Suppress non-error output.",
     )
+
+
+def build_parser() -> argparse.ArgumentParser:
+    """Parser for the legacy invocation ``markdownizer <project> -o <out>``."""
+    parser = argparse.ArgumentParser(
+        prog="markdownizer",
+        description=(
+            "Extract existing documentation from a Python project into "
+            "Markdown files. Use 'markdownizer build <project>' for format "
+            "selection; the legacy form below is kept as a compatibility alias."
+        ),
+    )
+    parser.add_argument(
+        "project",
+        type=Path,
+        help="Path to the Python project root to scan.",
+    )
+    _add_common_arguments(parser)
+    _add_verbose_arguments(parser)
     parser.add_argument(
         "--version",
         action="version",
@@ -80,10 +99,26 @@ def build_parser() -> argparse.ArgumentParser:
     return parser
 
 
-def main(argv: list[str] | None = None) -> int:
-    parser = build_parser()
-    args = parser.parse_args(argv)
+def build_build_parser() -> argparse.ArgumentParser:
+    """Parser for ``markdownizer build <project>``."""
+    parser = argparse.ArgumentParser(
+        prog="markdownizer build",
+        description=(
+            "Scan a project and emit a deterministic representation: "
+            "Markdown (default), JSON project IR, or compact context."
+        ),
+    )
+    parser.add_argument(
+        "project",
+        type=Path,
+        help="Path to the Python project root to scan.",
+    )
+    _add_common_arguments(parser)
+    _add_verbose_arguments(parser)
+    return parser
 
+
+def _run(args: argparse.Namespace) -> int:
     if args.quiet:
         level = logging.ERROR
     elif args.verbose >= 2:
@@ -113,17 +148,29 @@ def main(argv: list[str] | None = None) -> int:
             include_source=not args.no_source,
             include_comments=not args.no_comments,
             include_undocumented=not args.only_documented,
+            format=args.format,
         )
-    except OSError as exc:
+    except (OSError, ValueError) as exc:
         print(f"error: {exc}", file=sys.stderr)
         return 1
 
     if not args.quiet:
-        print(f"Wrote {len(written)} Markdown file(s) to {output}")
+        print(f"Wrote {len(written)} file(s) to {output}")
         for path in written:
             print(f"  - {path}")
     return 0
 
 
+def main(argv: list[str] | None = None) -> int:
+    args_list = list(sys.argv[1:] if argv is None else argv)
+    if args_list and args_list[0] == "build":
+        parser = build_build_parser()
+        args = parser.parse_args(args_list[1:])
+    else:
+        parser = build_parser()
+        args = parser.parse_args(args_list)
+    return _run(args)
+
+
 if __name__ == "__main__":
     sys.exit(main())

markdownizer-0.3.0/markdownizer/extractor.py

@@ -0,0 +1,88 @@
+"""Orchestrates scanning, parsing, IR building, and backend output writing."""
+
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+
+from markdownizer.backends import RenderOptions, get_backend
+from markdownizer.ir import build_project_ir
+from markdownizer.renderer import SourceMode
+
+logger = logging.getLogger(__name__)
+
+
+def _package_key(file_path: Path, project_root: Path) -> str:
+    try:
+        rel = file_path.resolve().relative_to(project_root.resolve())
+    except ValueError:
+        rel = Path(file_path)
+    parent = rel.parent
+    if parent == Path("."):
+        return ""
+    return str(parent)
+
+
+def _package_label(key: str, root_name: str) -> str:
+    if key == "":
+        return root_name
+    return key.replace("/", ".").replace("\\", ".")
+
+
+def extract_project(
+    project_root: Path,
+    output_dir: Path,
+    root_name: str = "_root",
+    exclude: list[str] | None = None,
+    include_source: SourceMode = True,
+    include_comments: bool = True,
+    include_undocumented: bool = True,
+    format: str = "markdown",
+) -> list[Path]:
+    """Scan the project, build a Project IR, and write backend output.
+
+    Args:
+        project_root: Directory to scan recursively.
+        output_dir: Directory where generated files will be written.
+        root_name: Filename (without extension) for files that live at the
+            project root.
+        exclude: Optional glob patterns (e.g. ``"tests/*"``, ``"migrations"``)
+            matched against paths relative to ``project_root``.
+        include_source: Include source: ``True`` (full), ``False`` (none), or
+            ``"signature"`` (declaration line only).
+        include_comments: Include the ``## Comments`` section.
+        include_undocumented: Keep objects that have no docstring.
+        format: Output backend: ``"markdown"`` (default), ``"json"``, or
+            ``"compact"``.
+    """
+    project_root = project_root.resolve()
+    output_dir = output_dir.resolve()
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    ir = build_project_ir(project_root, exclude=exclude)
+    backend = get_backend(format)
+    options = RenderOptions(
+        root_name=root_name,
+        include_source=include_source,
+        include_comments=include_comments,
+        include_undocumented=include_undocumented,
+    )
+    files = backend.render(ir, options)
+
+    written: list[Path] = []
+    for filename, content in files.items():
+        out_path = output_dir / filename
+        try:
+            out_path.write_text(content, encoding="utf-8")
+        except OSError as exc:
+            logger.error("could not write %s: %s", out_path, exc)
+            raise
+        written.append(out_path)
+
+    logger.info(
+        "Wrote %d file(s) to %s (%d Python file(s) scanned)",
+        len(written),
+        output_dir,
+        len(ir.modules),
+    )
+    return written