llmstxt-standalone 0.1.0__tar.gz → 0.2.0__tar.gz

{llmstxt_standalone-0.1.0 → llmstxt_standalone-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: llmstxt-standalone
-Version: 0.1.0
+Version: 0.2.0
 Summary: Generate llms.txt from built HTML documentation
 Keywords: llms,documentation,markdown,mkdocs
 Author: Shaan Majid
@@ -20,10 +20,12 @@ Classifier: Topic :: Documentation
 Classifier: Typing :: Typed
 Requires-Dist: typer>=0.9.0
 Requires-Dist: pyyaml>=6.0
+Requires-Dist: ruamel-yaml>=0.18
 Requires-Dist: beautifulsoup4>=4.12
 Requires-Dist: markdownify>=0.14,<2.0
 Requires-Dist: mdformat>=0.7,<2.0
 Requires-Dist: mdformat-tables>=1.0
+Requires-Dist: pydantic>=2.12.5
 Requires-Python: >=3.10
 Project-URL: Repository, https://github.com/shaanmajid/llmstxt-standalone
 Project-URL: Issues, https://github.com/shaanmajid/llmstxt-standalone/issues
@@ -59,21 +61,25 @@ uv add llmstxt-standalone  # or: pip install
 
 ## Usage
 
+### build
+
+Generate llms.txt from a built MkDocs site:
+
 ```bash
 # Run from project root (expects mkdocs.yml and site/)
-llmstxt-standalone
+llmstxt-standalone build
 
 # Explicit paths
-llmstxt-standalone --config mkdocs.yml --site-dir ./build --output-dir ./dist
+llmstxt-standalone build --config mkdocs.yml --site-dir ./build --output-dir ./dist
 
 # Preview without writing files
-llmstxt-standalone --dry-run
+llmstxt-standalone build --dry-run
 
 # Suppress output
-llmstxt-standalone --quiet
+llmstxt-standalone build --quiet
 
 # Show detailed progress
-llmstxt-standalone --verbose
+llmstxt-standalone build --verbose
 ```
 
 | Option | Short | Default | Description |
@@ -84,11 +90,65 @@ llmstxt-standalone --verbose
 | `--dry-run` | `-n` | | Preview without writing |
 | `--quiet` | `-q` | | Suppress output |
 | `--verbose` | `-v` | | Show detailed progress |
-| `--version` | `-V` | | Show version |
+
+### init
+
+Add llmstxt plugin configuration to an existing mkdocs.yml:
+
+```bash
+llmstxt-standalone init
+
+# Specify config path
+llmstxt-standalone init --config path/to/mkdocs.yml
+
+# Overwrite existing llmstxt config
+llmstxt-standalone init --force
+
+# Show detailed progress
+llmstxt-standalone init --verbose
+```
+
+| Option | Short | Description |
+|--------|-------|-------------|
+| `--config` | `-c` | Path to mkdocs.yml (default: mkdocs.yml) |
+| `--force` | `-f` | Overwrite existing llmstxt section |
+| `--quiet` | `-q` | Suppress output |
+| `--verbose` | `-v` | Show detailed progress |
+
+### validate
+
+Check that a config file is valid:
+
+```bash
+$ llmstxt-standalone validate
+Config valid: mkdocs.yml
+  Site: My Project
+  Sections: 3
+  Pages: 12
+
+# Exit code only (for scripts)
+llmstxt-standalone validate --quiet
+
+# Show section details
+llmstxt-standalone validate --verbose
+```
+
+| Option | Short | Description |
+|--------|-------|-------------|
+| `--config` | `-c` | Path to mkdocs.yml (default: mkdocs.yml) |
+| `--quiet` | `-q` | Suppress output |
+| `--verbose` | `-v` | Show detailed config information |
+
+### Global options
+
+```bash
+llmstxt-standalone --version  # Show version
+llmstxt-standalone --help     # Show available commands
+```
 
 ## Output
 
-The tool generates three outputs:
+The `build` command generates three outputs:
 
 1. `llms.txt` — an index file with markdown links to all pages
 1. `llms-full.txt` — concatenated content of all pages

{llmstxt_standalone-0.1.0 → llmstxt_standalone-0.2.0}/README.md

@@ -28,21 +28,25 @@ uv add llmstxt-standalone  # or: pip install
 
 ## Usage
 
+### build
+
+Generate llms.txt from a built MkDocs site:
+
 ```bash
 # Run from project root (expects mkdocs.yml and site/)
-llmstxt-standalone
+llmstxt-standalone build
 
 # Explicit paths
-llmstxt-standalone --config mkdocs.yml --site-dir ./build --output-dir ./dist
+llmstxt-standalone build --config mkdocs.yml --site-dir ./build --output-dir ./dist
 
 # Preview without writing files
-llmstxt-standalone --dry-run
+llmstxt-standalone build --dry-run
 
 # Suppress output
-llmstxt-standalone --quiet
+llmstxt-standalone build --quiet
 
 # Show detailed progress
-llmstxt-standalone --verbose
+llmstxt-standalone build --verbose
 ```
 
 | Option | Short | Default | Description |
@@ -53,11 +57,65 @@ llmstxt-standalone --verbose
 | `--dry-run` | `-n` | | Preview without writing |
 | `--quiet` | `-q` | | Suppress output |
 | `--verbose` | `-v` | | Show detailed progress |
-| `--version` | `-V` | | Show version |
+
+### init
+
+Add llmstxt plugin configuration to an existing mkdocs.yml:
+
+```bash
+llmstxt-standalone init
+
+# Specify config path
+llmstxt-standalone init --config path/to/mkdocs.yml
+
+# Overwrite existing llmstxt config
+llmstxt-standalone init --force
+
+# Show detailed progress
+llmstxt-standalone init --verbose
+```
+
+| Option | Short | Description |
+|--------|-------|-------------|
+| `--config` | `-c` | Path to mkdocs.yml (default: mkdocs.yml) |
+| `--force` | `-f` | Overwrite existing llmstxt section |
+| `--quiet` | `-q` | Suppress output |
+| `--verbose` | `-v` | Show detailed progress |
+
+### validate
+
+Check that a config file is valid:
+
+```bash
+$ llmstxt-standalone validate
+Config valid: mkdocs.yml
+  Site: My Project
+  Sections: 3
+  Pages: 12
+
+# Exit code only (for scripts)
+llmstxt-standalone validate --quiet
+
+# Show section details
+llmstxt-standalone validate --verbose
+```
+
+| Option | Short | Description |
+|--------|-------|-------------|
+| `--config` | `-c` | Path to mkdocs.yml (default: mkdocs.yml) |
+| `--quiet` | `-q` | Suppress output |
+| `--verbose` | `-v` | Show detailed config information |
+
+### Global options
+
+```bash
+llmstxt-standalone --version  # Show version
+llmstxt-standalone --help     # Show available commands
+```
 
 ## Output
 
-The tool generates three outputs:
+The `build` command generates three outputs:
 
 1. `llms.txt` — an index file with markdown links to all pages
 1. `llms-full.txt` — concatenated content of all pages

{llmstxt_standalone-0.1.0 → llmstxt_standalone-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "llmstxt-standalone"
-version = "0.1.0"
+version = "0.2.0"
 description = "Generate llms.txt from built HTML documentation"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -26,10 +26,12 @@ authors = [{ name = "Shaan Majid", email = "shaanmajid64@gmail.com" }]
 dependencies = [
     "typer>=0.9.0",
     "pyyaml>=6.0",
+    "ruamel.yaml>=0.18",
     "beautifulsoup4>=4.12",
     "markdownify>=0.14,<2.0",
     "mdformat>=0.7,<2.0",
     "mdformat-tables>=1.0",
+    "pydantic>=2.12.5",
 ]
 
 [project.scripts]

llmstxt_standalone-0.2.0/src/llmstxt_standalone/cli.py

@@ -0,0 +1,422 @@
+"""Command-line interface."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from pathlib import Path
+from typing import Annotated
+
+import typer
+import yaml
+from ruamel.yaml import YAML
+from ruamel.yaml import YAMLError as RuamelYAMLError
+
+from llmstxt_standalone import __version__
+from llmstxt_standalone.config import load_config
+from llmstxt_standalone.generate import (
+    build_llms_output,
+    ensure_safe_md_path,
+    write_markdown_files,
+)
+
+
+def _make_logger(
+    quiet: bool, verbose: bool = False
+) -> tuple[Callable[..., None], Callable[..., None]]:
+    """Create log and log_verbose functions for CLI output.
+
+    Args:
+        quiet: If True, suppress all output.
+        verbose: If True, enable verbose logging (quiet overrides this).
+
+    Returns:
+        Tuple of (log, log_verbose) functions.
+    """
+    effective_verbose = verbose and not quiet
+
+    def log(msg: str, color: str = "green", err: bool = False) -> None:
+        if not quiet:
+            typer.secho(msg, fg=color, err=err)
+
+    def log_verbose(msg: str, color: str = "green", err: bool = False) -> None:
+        if effective_verbose:
+            typer.secho(msg, fg=color, err=err)
+
+    return log, log_verbose
+
+
+def version_callback(value: bool) -> None:
+    """Print version and exit if --version flag is set."""
+    if value:
+        typer.echo(f"llmstxt-standalone {__version__}")
+        raise typer.Exit()
+
+
+app = typer.Typer(
+    help="Generate llms.txt from built HTML documentation.",
+    no_args_is_help=True,
+    context_settings={"help_option_names": ["-h", "--help"]},
+)
+
+
+@app.callback(invoke_without_command=True)
+def main(
+    version: Annotated[
+        bool,
+        typer.Option(
+            "--version",
+            "-V",
+            callback=version_callback,
+            is_eager=True,
+            help="Show version and exit",
+        ),
+    ] = False,
+) -> None:
+    """Generate llms.txt from built HTML documentation."""
+
+
+@app.command()
+def build(
+    config: Annotated[
+        Path,
+        typer.Option("--config", "-c", help="Path to mkdocs.yml config file"),
+    ] = Path("mkdocs.yml"),
+    site_dir: Annotated[
+        Path,
+        typer.Option("--site-dir", "-s", help="Path to built HTML site directory"),
+    ] = Path("site"),
+    output_dir: Annotated[
+        Path | None,
+        typer.Option(
+            "--output-dir", "-o", help="Output directory (defaults to site-dir)"
+        ),
+    ] = None,
+    dry_run: Annotated[
+        bool,
+        typer.Option(
+            "--dry-run",
+            "-n",
+            help="Preview what would be generated without writing files",
+        ),
+    ] = False,
+    quiet: Annotated[
+        bool,
+        typer.Option("--quiet", "-q", help="Suppress output (exit code only)"),
+    ] = False,
+    verbose: Annotated[
+        bool,
+        typer.Option("--verbose", "-v", help="Show detailed progress"),
+    ] = False,
+) -> None:
+    """Generate llms.txt and llms-full.txt from built MkDocs site."""
+    # Resolve output directory
+    out_dir = output_dir or site_dir
+    log, log_verbose = _make_logger(quiet, verbose)
+
+    # Validate inputs
+    if not config.exists():
+        log(f"Error: Config file not found: {config}", color="red", err=True)
+        raise typer.Exit(1)
+
+    if not site_dir.exists():
+        log(f"Error: Site directory not found: {site_dir}", color="red", err=True)
+        log(
+            "Hint: Run 'mkdocs build' first to generate the HTML documentation.",
+            color="yellow",
+            err=True,
+        )
+        raise typer.Exit(1)
+
+    # Load config
+    try:
+        cfg = load_config(config)
+    except (FileNotFoundError, ValueError, yaml.YAMLError) as e:
+        log(f"Error loading config: {e}", color="red", err=True)
+        raise typer.Exit(1) from None
+
+    # Validate sections
+    if not cfg.sections:
+        log("Error: No sections configured.", color="red", err=True)
+        log(
+            "Add a 'nav' to your mkdocs.yml, or configure 'sections' "
+            "in the llmstxt plugin.",
+            color="yellow",
+            err=True,
+        )
+        raise typer.Exit(1)
+
+    log_verbose(f"Site: {cfg.site_name}")
+    log_verbose(f"Sections: {list(cfg.sections.keys())}")
+    if dry_run:
+        log_verbose("Dry run - no files will be written")
+
+    # Generate content
+    llms_build = build_llms_output(
+        config=cfg,
+        site_dir=site_dir,
+    )
+    try:
+        markdown_files = write_markdown_files(
+            llms_build.pages,
+            output_dir=out_dir,
+            use_directory_urls=cfg.use_directory_urls,
+            dry_run=dry_run,
+        )
+    except (OSError, ValueError) as exc:
+        log(f"Error writing markdown files: {exc}", color="red", err=True)
+        raise typer.Exit(1) from None
+
+    # Define output paths
+    llms_path = out_dir / "llms.txt"
+    try:
+        full_output_path = ensure_safe_md_path(cfg.full_output)
+    except ValueError:
+        log(
+            "Error: Invalid full_output: must be a relative path without '..'",
+            color="red",
+            err=True,
+        )
+        raise typer.Exit(1) from None
+    full_path = out_dir / full_output_path
+
+    # Write output files (skip in dry-run mode)
+    if dry_run:
+        action = "Would generate"
+        color = "yellow"
+    else:
+        action = "Generated"
+        color = "green"
+        try:
+            out_dir.mkdir(parents=True, exist_ok=True)
+            llms_path.write_text(llms_build.llms_txt, encoding="utf-8")
+            full_path.write_text(llms_build.llms_full_txt, encoding="utf-8")
+        except OSError as exc:
+            log(f"Error writing output files: {exc}", color="red", err=True)
+            raise typer.Exit(1) from None
+
+    log(f"{action} {llms_path} ({len(llms_build.llms_txt):,} bytes)", color)
+    log(f"{action} {full_path} ({len(llms_build.llms_full_txt):,} bytes)", color)
+    log(f"{action} {len(markdown_files)} markdown files", color)
+
+    if llms_build.skipped:
+        log_verbose("Skipped files:", color="yellow", err=True)
+        for path, reason in llms_build.skipped:
+            log_verbose(f"- {path} ({reason})", color="yellow", err=True)
+
+    if llms_build.warnings:
+        log("Warnings:", color="yellow", err=True)
+        for warning in llms_build.warnings:
+            log(f"- {warning}", color="yellow", err=True)
+
+
+@app.command()
+def init(
+    config: Annotated[
+        Path,
+        typer.Option("--config", "-c", help="Path to mkdocs.yml config file"),
+    ] = Path("mkdocs.yml"),
+    force: Annotated[
+        bool,
+        typer.Option("--force", "-f", help="Overwrite existing llmstxt section"),
+    ] = False,
+    quiet: Annotated[
+        bool,
+        typer.Option("--quiet", "-q", help="Suppress output (exit code only)"),
+    ] = False,
+    verbose: Annotated[
+        bool,
+        typer.Option("--verbose", "-v", help="Show detailed progress"),
+    ] = False,
+) -> None:
+    """Add llmstxt plugin config to mkdocs.yml."""
+    log, log_verbose = _make_logger(quiet, verbose)
+
+    if not config.exists():
+        log(f"Error: Config file not found: {config}", color="red", err=True)
+        log(
+            "Create one first or specify path with --config.",
+            color="yellow",
+            err=True,
+        )
+        raise typer.Exit(1)
+
+    yaml_parser = YAML()
+    yaml_parser.preserve_quotes = True
+
+    try:
+        with open(config, encoding="utf-8") as f:
+            data = yaml_parser.load(f)
+    except RuamelYAMLError as e:
+        log(f"Error: Invalid YAML: {e}", color="red", err=True)
+        raise typer.Exit(1) from None
+
+    if data is None:
+        data = {}
+
+    # Check for existing llmstxt plugin
+    plugins = data.get("plugins", [])
+    if plugins is None:
+        plugins = []
+    if not isinstance(plugins, (list, dict)):
+        log(
+            "Error: 'plugins' must be a list or mapping in mkdocs.yml.",
+            color="red",
+            err=True,
+        )
+        raise typer.Exit(1)
+    data["plugins"] = plugins
+
+    if isinstance(plugins, list):
+        has_llmstxt = any(
+            p == "llmstxt" or (isinstance(p, dict) and "llmstxt" in p) for p in plugins
+        )
+    elif isinstance(plugins, dict):
+        has_llmstxt = "llmstxt" in plugins
+    else:
+        has_llmstxt = False
+
+    if has_llmstxt and not force:
+        log("Error: llmstxt plugin already configured.", color="red", err=True)
+        log(
+            "Use --force to overwrite existing configuration.",
+            color="yellow",
+            err=True,
+        )
+        raise typer.Exit(1)
+
+    # Remove existing llmstxt if force is set
+    if has_llmstxt and force:
+        if isinstance(plugins, list):
+            plugins = [
+                p
+                for p in plugins
+                if p != "llmstxt" and not (isinstance(p, dict) and "llmstxt" in p)
+            ]
+            data["plugins"] = plugins
+        elif isinstance(plugins, dict):
+            del plugins["llmstxt"]
+
+    # Create the llmstxt plugin entry with commented example
+    llmstxt_entry = {
+        "llmstxt": {
+            # We'll add comments after writing
+        }
+    }
+
+    if isinstance(data["plugins"], list):
+        data["plugins"].append(llmstxt_entry)
+    else:
+        # Preserve dict-style plugins
+        data["plugins"]["llmstxt"] = {}
+
+    # Write the file
+    try:
+        with open(config, "w", encoding="utf-8") as f:
+            yaml_parser.dump(data, f)
+    except PermissionError:
+        log(f"Error: Permission denied writing to {config}", color="red", err=True)
+        raise typer.Exit(1) from None
+
+    # Now add comments using string manipulation since ruamel.yaml comment API is complex
+    content = config.read_text(encoding="utf-8")
+    ends_with_newline = content.endswith("\n")
+
+    # Find the llmstxt entry and add commented example below it
+    commented_example_lines = [
+        "# markdown_description: |",
+        "#   Additional context for LLMs.",
+        "# sections:",
+        "#   Getting Started:",
+        "#     - index.md",
+    ]
+
+    def _comment_indent(line: str) -> int:
+        leading = len(line) - len(line.lstrip(" "))
+        if line.lstrip().startswith("- "):
+            return leading + 4
+        return leading + 2
+
+    def _format_commented_example(indent: int) -> list[str]:
+        prefix = " " * indent
+        return [f"{prefix}{line}" for line in commented_example_lines]
+
+    # Look for the llmstxt entry and add commented example below it
+    lines = content.splitlines()
+    new_lines: list[str] = []
+    inserted = False
+    for line in lines:
+        stripped = line.strip()
+        if not inserted and stripped == "llmstxt: {}":
+            indent = _comment_indent(line)
+            new_lines.append(line.replace("llmstxt: {}", "llmstxt:"))
+            new_lines.extend(_format_commented_example(indent))
+            inserted = True
+            continue
+        if not inserted and stripped == "llmstxt:":
+            indent = _comment_indent(line)
+            new_lines.append(line)
+            new_lines.extend(_format_commented_example(indent))
+            inserted = True
+            continue
+        new_lines.append(line)
+    content = "\n".join(new_lines)
+    if ends_with_newline:
+        content += "\n"
+
+    try:
+        config.write_text(content, encoding="utf-8")
+    except PermissionError:
+        log(f"Error: Permission denied writing to {config}", color="red", err=True)
+        raise typer.Exit(1) from None
+
+    log(f"Added llmstxt plugin to {config}")
+    log_verbose(
+        "Configuration includes commented example for sections and markdown_description"
+    )
+
+
+@app.command()
+def validate(
+    config: Annotated[
+        Path,
+        typer.Option("--config", "-c", help="Path to mkdocs.yml config file"),
+    ] = Path("mkdocs.yml"),
+    quiet: Annotated[
+        bool,
+        typer.Option("--quiet", "-q", help="Suppress output (exit code only)"),
+    ] = False,
+    verbose: Annotated[
+        bool,
+        typer.Option("--verbose", "-v", help="Show detailed config information"),
+    ] = False,
+) -> None:
+    """Check config file validity."""
+    log, log_verbose = _make_logger(quiet, verbose)
+
+    try:
+        cfg = load_config(config)
+    except FileNotFoundError:
+        log(f"Config invalid: {config}", color="red", err=True)
+        log(f"  Error: File not found: {config}", color="red", err=True)
+        raise typer.Exit(1) from None
+    except (ValueError, yaml.YAMLError) as e:
+        log(f"Config invalid: {config}", color="red", err=True)
+        log(f"  Error: {e}", color="red", err=True)
+        raise typer.Exit(1) from None
+
+    total_pages = sum(len(pages) for pages in cfg.sections.values())
+
+    log(f"Config valid: {config}")
+    log(f"  Site: {cfg.site_name}")
+    log(f"  Sections: {len(cfg.sections)}")
+    log(f"  Pages: {total_pages}")
+
+    # Verbose: show section details
+    for section_name, pages in cfg.sections.items():
+        log_verbose(f"  {section_name}: {len(pages)} pages")
+        for page in pages:
+            log_verbose(f"    - {page}")
+
+
+if __name__ == "__main__":
+    app()

{llmstxt_standalone-0.1.0 → llmstxt_standalone-0.2.0}/src/llmstxt_standalone/config/derive.py

@@ -10,10 +10,13 @@ def nav_to_sections(nav: list[Any]) -> dict[str, list[str]]:
     sections: dict[str, list[str]] = {}
 
     for item in nav:
-        if isinstance(item, dict):
+        if isinstance(item, str):
+            # Bare top-level page: - index.md
+            sections.setdefault("Pages", []).append(item)
+        elif isinstance(item, dict):
             for key, value in item.items():
                 if isinstance(value, str):
-                    # Top-level page, add to "Pages" section
+                    # Top-level page with title: - Home: index.md
                     sections.setdefault("Pages", []).append(value)
                 elif isinstance(value, list):
                     # Section with children

llmstxt_standalone-0.2.0/src/llmstxt_standalone/config/load.py

@@ -0,0 +1,172 @@
+"""Configuration loading from mkdocs.yml."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import yaml
+from pydantic import BaseModel, Field, ValidationError, field_validator
+
+from llmstxt_standalone.config.derive import nav_to_sections
+from llmstxt_standalone.config.model import Config
+from llmstxt_standalone.config.plugin import get_llmstxt_config
+
+DEFAULT_SITE_NAME = "Documentation"
+DEFAULT_FULL_OUTPUT = "llms-full.txt"
+
+
+class _PermissiveLoader(yaml.SafeLoader):
+    """SafeLoader that ignores unknown Python tags.
+
+    MkDocs extensions like pymdownx.slugs use Python-specific YAML tags
+    like !python/object/apply which SafeLoader rejects. This loader
+    treats them as raw strings to allow parsing the rest of the config.
+    """
+
+
+def _ignore_unknown(loader: yaml.Loader, tag_suffix: str, node: yaml.Node) -> str:
+    """Return the raw tag as a placeholder string."""
+    return f"<{node.tag}>"
+
+
+# Register handler for all Python tags (both full and shorthand forms)
+_PermissiveLoader.add_multi_constructor("tag:yaml.org,2002:python/", _ignore_unknown)
+_PermissiveLoader.add_multi_constructor("!python/", _ignore_unknown)
+
+
+class LlmstxtPluginConfig(BaseModel):
+    """Pydantic model for llmstxt plugin configuration."""
+
+    markdown_description: str = ""
+    full_output: str = DEFAULT_FULL_OUTPUT
+    content_selector: str | None = None
+    sections: dict[str, list[str]] = Field(default_factory=dict)
+
+    @field_validator("sections", mode="before")
+    @classmethod
+    def validate_sections(cls, v: Any) -> dict[str, list[str]]:
+        """Validate sections is a dict with string keys and list[str] values."""
+        if v is None:
+            return {}
+        if not isinstance(v, dict):
+            raise ValueError(f"'sections' must be a mapping, got {type(v).__name__}")
+        for section_name, pages in v.items():
+            if not isinstance(section_name, str):
+                raise ValueError(
+                    f"'sections' keys must be strings, got {type(section_name).__name__}"
+                )
+            if not isinstance(pages, list):
+                raise ValueError(
+                    f"'sections.{section_name}' must be a list of strings, "
+                    f"got {type(pages).__name__}"
+                )
+            for page in pages:
+                if not isinstance(page, str):
+                    raise ValueError(
+                        f"'sections.{section_name}' entries must be strings, "
+                        f"got {type(page).__name__}"
+                    )
+        return v
+
+
+class MkDocsConfig(BaseModel):
+    """Pydantic model for mkdocs.yml top-level fields we care about."""
+
+    site_name: str = DEFAULT_SITE_NAME
+    site_description: str = ""
+    site_url: str = ""
+    nav: list[Any] = Field(default_factory=list)
+    use_directory_urls: bool = True
+
+    @field_validator("site_name", mode="before")
+    @classmethod
+    def coerce_site_name(cls, v: Any) -> str:
+        """Coerce None to default."""
+        return v if v is not None else DEFAULT_SITE_NAME
+
+    @field_validator("site_description", "site_url", mode="before")
+    @classmethod
+    def coerce_str_fields(cls, v: Any) -> str:
+        """Coerce None to empty string."""
+        return v if v is not None else ""
+
+    @field_validator("nav", mode="before")
+    @classmethod
+    def coerce_nav(cls, v: Any) -> list[Any]:
+        """Coerce None to empty list."""
+        return v if v is not None else []
+
+    @field_validator("site_url", mode="after")
+    @classmethod
+    def strip_trailing_slash(cls, v: str) -> str:
+        """Remove trailing slash from site_url."""
+        return v.rstrip("/")
+
+
+def load_config(config_path: Path) -> Config:
+    """Load and resolve configuration from mkdocs.yml.
+
+    Args:
+        config_path: Path to mkdocs.yml file.
+
+    Returns:
+        Resolved Config object.
+
+    Raises:
+        FileNotFoundError: If config file doesn't exist.
+        ValueError: If config is invalid.
+    """
+    if not config_path.exists():
+        raise FileNotFoundError(f"Config file not found: {config_path}")
+
+    try:
+        with open(config_path, encoding="utf-8") as f:
+            raw = yaml.load(f, Loader=_PermissiveLoader)
+    except RecursionError:
+        raise ValueError(
+            f"Config file has nav structure too deeply nested: {config_path}"
+        ) from None
+
+    if not isinstance(raw, dict):
+        raise ValueError(f"Config file must be a mapping: {config_path}")
+
+    return _config_from_mkdocs(raw)
+
+
+def _config_from_mkdocs(raw: dict[str, Any]) -> Config:
+    """Build a Config from a parsed mkdocs.yml mapping."""
+    try:
+        mkdocs = MkDocsConfig.model_validate(raw)
+    except ValidationError as e:
+        raise ValueError(str(e)) from None
+
+    llmstxt_raw = get_llmstxt_config(raw)
+
+    if llmstxt_raw is not None:
+        try:
+            plugin = LlmstxtPluginConfig.model_validate(llmstxt_raw)
+        except ValidationError as e:
+            # Extract the core error message for cleaner output
+            raise ValueError(f"llmstxt {e.errors()[0]['msg']}") from None
+        sections = plugin.sections
+        markdown_description = plugin.markdown_description
+        full_output = plugin.full_output
+        content_selector = plugin.content_selector
+    else:
+        sections = nav_to_sections(mkdocs.nav)
+        markdown_description = ""
+        full_output = DEFAULT_FULL_OUTPUT
+        content_selector = None
+
+    return Config(
+        site_name=mkdocs.site_name,
+        site_description=mkdocs.site_description,
+        site_url=mkdocs.site_url,
+        markdown_description=markdown_description,
+        full_output=full_output,
+        content_selector=content_selector,
+        sections=sections,
+        nav=mkdocs.nav,
+        use_directory_urls=mkdocs.use_directory_urls,
+    )

{llmstxt_standalone-0.1.0 → llmstxt_standalone-0.2.0}/src/llmstxt_standalone/config/model.py

@@ -2,12 +2,12 @@
 
 from __future__ import annotations
 
-from dataclasses import dataclass
 from typing import Any
 
+from pydantic import BaseModel
 
-@dataclass
-class Config:
+
+class Config(BaseModel):
     """Resolved configuration for llmstxt generation."""
 
     site_name: str

{llmstxt_standalone-0.1.0 → llmstxt_standalone-0.2.0}/src/llmstxt_standalone/convert.py

@@ -6,6 +6,11 @@ import mdformat
 from bs4 import BeautifulSoup, NavigableString, Tag
 from markdownify import ATX, MarkdownConverter
 
+__all__ = [
+    "extract_title_from_html",
+    "html_to_markdown",
+]
+
 
 def _should_remove(tag: Tag) -> bool:
     """Check if a tag should be removed during autoclean."""
@@ -34,9 +39,15 @@ def _autoclean(soup: BeautifulSoup | Tag) -> None:
     for element in soup.find_all("table", attrs={"class": "highlighttable"}):
         code = element.find("code")
         if code:
-            element.replace_with(
-                BeautifulSoup(f"<pre>{code.get_text()}</pre>", "html.parser")
+            # Find the root BeautifulSoup document to create new tags
+            # (soup parameter may be a Tag, which doesn't have new_tag)
+            doc = next(
+                (p for p in element.parents if isinstance(p, BeautifulSoup)), None
             )
+            if doc:
+                pre_tag = doc.new_tag("pre")
+                pre_tag.string = code.get_text()
+                element.replace_with(pre_tag)
 
 
 def _get_language(tag: Tag) -> str:
@@ -64,13 +75,14 @@ def _get_language(tag: Tag) -> str:
     return ""
 
 
-# Converter with mkdocs-llmstxt-compatible settings
-_converter = MarkdownConverter(
-    bullets="-",
-    code_language_callback=_get_language,
-    escape_underscores=False,
-    heading_style=ATX,
-)
+def _make_converter() -> MarkdownConverter:
+    """Create a MarkdownConverter with mkdocs-llmstxt-compatible settings."""
+    return MarkdownConverter(
+        bullets="-",
+        code_language_callback=_get_language,
+        escape_underscores=False,
+        heading_style=ATX,
+    )
 
 
 def extract_title_from_html(html: str, site_name: str | None = None) -> str | None:
@@ -148,5 +160,6 @@ def html_to_markdown(html: str, content_selector: str | None = None) -> str:
         return ""
 
     _autoclean(content)
-    md = _converter.convert_soup(content)
+    converter = _make_converter()
+    md = converter.convert_soup(content)
     return mdformat.text(md, options={"wrap": "no"}, extensions=("tables",))

{llmstxt_standalone-0.1.0 → llmstxt_standalone-0.2.0}/src/llmstxt_standalone/generate.py

@@ -8,6 +8,19 @@ from pathlib import Path
 from llmstxt_standalone.config import Config
 from llmstxt_standalone.convert import extract_title_from_html, html_to_markdown
 
+__all__ = [
+    "BuildResult",
+    "GenerateResult",
+    "PageMarkdown",
+    "build_llms_output",
+    "ensure_safe_md_path",
+    "generate_llms_txt",
+    "md_path_to_html_path",
+    "md_path_to_output_md_path",
+    "md_path_to_page_url",
+    "write_markdown_files",
+]
+
 
 def _escape_markdown_link_text(text: str) -> str:
     r"""Escape characters that break markdown link syntax.
@@ -32,7 +45,18 @@ def _is_index_md(md_path: str) -> bool:
     return md_path == "index.md" or md_path.endswith("/index.md")
 
 
-def _ensure_safe_md_path(md_path: str) -> Path:
+def ensure_safe_md_path(md_path: str) -> Path:
+    """Validate and convert a markdown path to a safe Path object.
+
+    Args:
+        md_path: Relative markdown file path (e.g., "install.md").
+
+    Returns:
+        Path object for the markdown file.
+
+    Raises:
+        ValueError: If path is absolute or contains '..'.
+    """
     path = Path(md_path)
     if path.is_absolute() or path.drive:
         raise ValueError(f"Markdown path must be relative: {md_path}")
@@ -63,7 +87,7 @@ def md_path_to_html_path(
         Path to the corresponding HTML file.
     """
     # Handle index.md at any level (root or nested like foo/bar/index.md)
-    safe_md_path = _ensure_safe_md_path(md_path)
+    safe_md_path = ensure_safe_md_path(md_path)
     if _is_index_md(md_path):
         html_path = site_dir / safe_md_path.with_suffix(".html")
         return _ensure_within_dir(site_dir, html_path, "HTML path")
@@ -117,7 +141,7 @@ def md_path_to_output_md_path(
         Path where the markdown file should be written.
     """
     # Handle index.md at any level (root or nested like foo/bar/index.md)
-    safe_md_path = _ensure_safe_md_path(md_path)
+    safe_md_path = ensure_safe_md_path(md_path)
     if _is_index_md(md_path):
         output_path = site_dir / safe_md_path
         return _ensure_within_dir(site_dir, output_path, "Output path")
@@ -149,7 +173,12 @@ class BuildResult:
 
 @dataclass
 class GenerateResult:
-    """Result of llms.txt generation with files written."""
+    """Result of llms.txt generation with files written.
+
+    Used by generate_llms_txt() for programmatic use cases that want
+    file writing handled automatically. The CLI uses BuildResult +
+    write_markdown_files() for more control over the write step.
+    """
 
     llms_txt: str
     llms_full_txt: str

llmstxt_standalone-0.1.0/src/llmstxt_standalone/cli.py

@@ -1,172 +0,0 @@
-"""Command-line interface."""
-
-from __future__ import annotations
-
-from pathlib import Path
-from typing import Annotated
-
-import typer
-
-from llmstxt_standalone import __version__
-from llmstxt_standalone.config import load_config
-from llmstxt_standalone.generate import build_llms_output, write_markdown_files
-
-app = typer.Typer(
-    help="Generate llms.txt from built HTML documentation.",
-    no_args_is_help=False,
-    context_settings={"help_option_names": ["-h", "--help"]},
-)
-
-
-def version_callback(value: bool) -> None:
-    """Print version and exit if --version flag is set."""
-    if value:
-        typer.echo(f"llmstxt-standalone {__version__}")
-        raise typer.Exit()
-
-
-@app.command()
-def main(
-    config: Annotated[
-        Path,
-        typer.Option("--config", "-c", help="Path to mkdocs.yml config file"),
-    ] = Path("mkdocs.yml"),
-    site_dir: Annotated[
-        Path,
-        typer.Option("--site-dir", "-s", help="Path to built HTML site directory"),
-    ] = Path("site"),
-    output_dir: Annotated[
-        Path | None,
-        typer.Option(
-            "--output-dir", "-o", help="Output directory (defaults to site-dir)"
-        ),
-    ] = None,
-    dry_run: Annotated[
-        bool,
-        typer.Option(
-            "--dry-run",
-            "-n",
-            help="Preview what would be generated without writing files",
-        ),
-    ] = False,
-    quiet: Annotated[
-        bool,
-        typer.Option("--quiet", "-q", help="Suppress output (exit code only)"),
-    ] = False,
-    verbose: Annotated[
-        bool,
-        typer.Option("--verbose", "-v", help="Show detailed progress"),
-    ] = False,
-    version: Annotated[
-        bool,
-        typer.Option(
-            "--version",
-            "-V",
-            callback=version_callback,
-            is_eager=True,
-            help="Show version",
-        ),
-    ] = False,
-) -> None:
-    """Generate llms.txt and llms-full.txt from built HTML documentation."""
-    # Resolve output directory
-    out_dir = output_dir or site_dir
-
-    # quiet overrides verbose
-    if quiet:
-        verbose = False
-
-    def log(msg: str, color: str = "green", err: bool = False) -> None:
-        if not quiet:
-            typer.secho(msg, fg=color, err=err)
-
-    # Validate inputs
-    if not config.exists():
-        typer.secho(f"Error: Config file not found: {config}", fg="red", err=True)
-        raise typer.Exit(1)
-
-    if not site_dir.exists():
-        typer.secho(f"Error: Site directory not found: {site_dir}", fg="red", err=True)
-        typer.secho(
-            "Hint: Run 'mkdocs build' first to generate the HTML documentation.",
-            fg="yellow",
-            err=True,
-        )
-        raise typer.Exit(1)
-
-    # Load config
-    try:
-        cfg = load_config(config)
-    except Exception as e:
-        typer.secho(f"Error loading config: {e}", fg="red", err=True)
-        raise typer.Exit(1) from None
-
-    # Validate sections
-    if not cfg.sections:
-        typer.secho("Error: No sections configured.", fg="red", err=True)
-        typer.secho(
-            "Add a 'nav' to your mkdocs.yml, or configure 'sections' "
-            "in the llmstxt plugin.",
-            fg="yellow",
-            err=True,
-        )
-        raise typer.Exit(1)
-
-    if verbose:
-        typer.echo(f"Site: {cfg.site_name}")
-        typer.echo(f"Sections: {list(cfg.sections.keys())}")
-        if dry_run:
-            typer.echo("Dry run - no files will be written")
-
-    # Generate content
-    build = build_llms_output(
-        config=cfg,
-        site_dir=site_dir,
-    )
-    try:
-        markdown_files = write_markdown_files(
-            build.pages,
-            output_dir=out_dir,
-            use_directory_urls=cfg.use_directory_urls,
-            dry_run=dry_run,
-        )
-    except (OSError, ValueError) as exc:
-        typer.secho(f"Error writing markdown files: {exc}", fg="red", err=True)
-        raise typer.Exit(1) from None
-
-    # Define output paths
-    llms_path = out_dir / "llms.txt"
-    full_path = out_dir / cfg.full_output
-
-    # Write output files (skip in dry-run mode)
-    if dry_run:
-        action = "Would generate"
-        color = "yellow"
-    else:
-        action = "Generated"
-        color = "green"
-        try:
-            out_dir.mkdir(parents=True, exist_ok=True)
-            llms_path.write_text(build.llms_txt, encoding="utf-8")
-            full_path.write_text(build.llms_full_txt, encoding="utf-8")
-        except OSError as exc:
-            typer.secho(f"Error writing output files: {exc}", fg="red", err=True)
-            raise typer.Exit(1) from None
-
-    log(f"{action} {llms_path} ({len(build.llms_txt):,} bytes)", color)
-    log(f"{action} {full_path} ({len(build.llms_full_txt):,} bytes)", color)
-    log(f"{action} {len(markdown_files)} markdown files", color)
-
-    if verbose and build.skipped:
-        log("Skipped files:", color="yellow", err=True)
-        for path, reason in build.skipped:
-            log(f"- {path} ({reason})", color="yellow", err=True)
-
-    if build.warnings:
-        log("Warnings:", color="yellow", err=True)
-        for warning in build.warnings:
-            log(f"- {warning}", color="yellow", err=True)
-
-
-if __name__ == "__main__":
-    app()

llmstxt_standalone-0.1.0/src/llmstxt_standalone/config/load.py

@@ -1,93 +0,0 @@
-"""Configuration loading from mkdocs.yml."""
-
-from __future__ import annotations
-
-from pathlib import Path
-from typing import Any
-
-import yaml
-
-from llmstxt_standalone.config.derive import nav_to_sections
-from llmstxt_standalone.config.model import Config
-from llmstxt_standalone.config.plugin import get_llmstxt_config
-
-DEFAULT_SITE_NAME = "Documentation"
-DEFAULT_FULL_OUTPUT = "llms-full.txt"
-
-
-class _PermissiveLoader(yaml.SafeLoader):
-    """SafeLoader that ignores unknown Python tags.
-
-    MkDocs extensions like pymdownx.slugs use Python-specific YAML tags
-    like !python/object/apply which SafeLoader rejects. This loader
-    treats them as raw strings to allow parsing the rest of the config.
-    """
-
-
-def _ignore_unknown(loader: yaml.Loader, tag_suffix: str, node: yaml.Node) -> str:
-    """Return the raw tag as a placeholder string."""
-    return f"<{node.tag}>"
-
-
-# Register handler for all Python tags (both full and shorthand forms)
-_PermissiveLoader.add_multi_constructor("tag:yaml.org,2002:python/", _ignore_unknown)
-_PermissiveLoader.add_multi_constructor("!python/", _ignore_unknown)
-
-
-def load_config(config_path: Path) -> Config:
-    """Load and resolve configuration from mkdocs.yml.
-
-    Args:
-        config_path: Path to mkdocs.yml file.
-
-    Returns:
-        Resolved Config object.
-
-    Raises:
-        FileNotFoundError: If config file doesn't exist.
-    """
-    if not config_path.exists():
-        raise FileNotFoundError(f"Config file not found: {config_path}")
-
-    with open(config_path, encoding="utf-8") as f:
-        raw = yaml.load(f, Loader=_PermissiveLoader)
-
-    if not isinstance(raw, dict):
-        raise ValueError(f"Config file must be a mapping: {config_path}")
-
-    return _config_from_mkdocs(raw)
-
-
-def _config_from_mkdocs(raw: dict[str, Any]) -> Config:
-    """Build a Config from a parsed mkdocs.yml mapping."""
-    site_name = raw.get("site_name", DEFAULT_SITE_NAME)
-    site_description = raw.get("site_description", "")
-    site_url = raw.get("site_url", "").rstrip("/")
-    nav = raw.get("nav", [])
-    # MkDocs defaults use_directory_urls to true
-    use_directory_urls = raw.get("use_directory_urls", True)
-
-    llmstxt_config = get_llmstxt_config(raw)
-
-    if llmstxt_config is not None:
-        markdown_description = llmstxt_config.get("markdown_description", "")
-        full_output = llmstxt_config.get("full_output", DEFAULT_FULL_OUTPUT)
-        content_selector = llmstxt_config.get("content_selector")
-        sections = llmstxt_config.get("sections", {})
-    else:
-        markdown_description = ""
-        full_output = DEFAULT_FULL_OUTPUT
-        content_selector = None
-        sections = nav_to_sections(nav)
-
-    return Config(
-        site_name=site_name,
-        site_description=site_description,
-        site_url=site_url,
-        markdown_description=markdown_description,
-        full_output=full_output,
-        content_selector=content_selector,
-        sections=sections,
-        nav=nav,
-        use_directory_urls=use_directory_urls,
-    )