llmstxt-standalone 0.1.0__tar.gz

llmstxt_standalone-0.1.0/PKG-INFO

@@ -0,0 +1,179 @@
+Metadata-Version: 2.3
+Name: llmstxt-standalone
+Version: 0.1.0
+Summary: Generate llms.txt from built HTML documentation
+Keywords: llms,documentation,markdown,mkdocs
+Author: Shaan Majid
+Author-email: Shaan Majid <shaanmajid64@gmail.com>
+License: MIT
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Documentation
+Classifier: Typing :: Typed
+Requires-Dist: typer>=0.9.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: markdownify>=0.14,<2.0
+Requires-Dist: mdformat>=0.7,<2.0
+Requires-Dist: mdformat-tables>=1.0
+Requires-Python: >=3.10
+Project-URL: Repository, https://github.com/shaanmajid/llmstxt-standalone
+Project-URL: Issues, https://github.com/shaanmajid/llmstxt-standalone/issues
+Description-Content-Type: text/markdown
+
+# llmstxt-standalone
+
+[![CI](https://github.com/shaanmajid/llmstxt-standalone/actions/workflows/ci.yml/badge.svg)](https://github.com/shaanmajid/llmstxt-standalone/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/llmstxt-standalone)](https://pypi.org/project/llmstxt-standalone/)
+[![Python](https://img.shields.io/pypi/pyversions/llmstxt-standalone)](https://pypi.org/project/llmstxt-standalone/)
+[![License](https://img.shields.io/pypi/l/llmstxt-standalone)](https://github.com/shaanmajid/llmstxt-standalone/blob/main/LICENSE)
+[![codecov](https://codecov.io/gh/shaanmajid/llmstxt-standalone/graph/badge.svg)](https://codecov.io/gh/shaanmajid/llmstxt-standalone)
+[![prek](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/j178/prek/master/docs/assets/badge-v0.json)](https://github.com/j178/prek)
+
+Generate `/llms.txt`, `/llms-full.txt`, and per-page markdown files from built HTML documentation, following the [llms.txt spec](https://llmstxt.org/).
+
+This tool works on pre-built HTML, making it useful for environments that cannot run MkDocs plugins (e.g., [Zensical](https://zensical.com/)) or when you want llms.txt generation as a separate build step. For standard MkDocs workflows, see [mkdocs-llmstxt](https://github.com/pawamoy/mkdocs-llmstxt).
+
+## Installation
+
+Requires Python 3.10+.
+
+```bash
+# Run without installing
+uvx llmstxt-standalone
+
+# Install as a CLI tool
+uv tool install llmstxt-standalone  # or: pipx install
+
+# Add to a project
+uv add llmstxt-standalone  # or: pip install
+```
+
+## Usage
+
+```bash
+# Run from project root (expects mkdocs.yml and site/)
+llmstxt-standalone
+
+# Explicit paths
+llmstxt-standalone --config mkdocs.yml --site-dir ./build --output-dir ./dist
+
+# Preview without writing files
+llmstxt-standalone --dry-run
+
+# Suppress output
+llmstxt-standalone --quiet
+
+# Show detailed progress
+llmstxt-standalone --verbose
+```
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--config` | `-c` | `mkdocs.yml` | Path to MkDocs config file |
+| `--site-dir` | `-s` | `site` | Path to built HTML directory |
+| `--output-dir` | `-o` | same as site-dir | Where to write output files |
+| `--dry-run` | `-n` | | Preview without writing |
+| `--quiet` | `-q` | | Suppress output |
+| `--verbose` | `-v` | | Show detailed progress |
+| `--version` | `-V` | | Show version |
+
+## Output
+
+The tool generates three outputs:
+
+1. `llms.txt` — an index file with markdown links to all pages
+1. `llms-full.txt` — concatenated content of all pages
+1. Per-page `.md` files alongside the HTML
+
+The per-page markdown files make the URLs in `llms.txt` resolve to actual content. If your site is at `https://docs.example.com/`, the URL `https://docs.example.com/install/index.md` returns markdown instead of HTML.
+
+## Configuration
+
+The tool reads your `mkdocs.yml` for site metadata. You can configure llmstxt output explicitly or let it derive structure from your nav.
+
+### Explicit configuration
+
+```yaml
+plugins:
+  - llmstxt:
+      markdown_description: |
+        Extra context for LLMs about your project.
+      full_output: llms-full.txt
+      content_selector: article.md-content__inner
+      sections:
+        Getting Started:
+          - index.md
+          - install.md
+        Usage:
+          - guide/basics.md
+          - guide/advanced.md
+```
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `markdown_description` | `""` | Additional context for LLMs, appears after site description |
+| `full_output` | `llms-full.txt` | Filename for concatenated content |
+| `content_selector` | auto-detect | CSS selector for main content |
+| `sections` | derived from nav | Section names mapped to page lists |
+
+### Automatic fallback
+
+Without an explicit `llmstxt` plugin config, sections derive from your `nav` structure. Top-level pages go into a "Pages" section; nested nav items become sections named by their keys.
+
+### MkDocs settings
+
+The tool respects `use_directory_urls` from your mkdocs.yml. When enabled (the default), `install.md` maps to `install/index.md`; when disabled, it maps to `install.md`.
+
+### Content extraction
+
+If `content_selector` is not set, the tool tries these selectors in order:
+
+1. `.md-content__inner` (Material for MkDocs)
+1. `[role="main"]` (default MkDocs theme)
+1. `article`
+1. `main`
+1. The entire document
+
+### Title resolution
+
+Page titles resolve in this order:
+
+1. The title from your `nav` structure
+1. The HTML `<title>` tag (with site name suffix stripped)
+1. The first `<h1>` tag
+1. A title derived from the filename
+
+## Programmatic use
+
+```python
+from pathlib import Path
+from llmstxt_standalone.config import load_config
+from llmstxt_standalone.generate import generate_llms_txt
+
+config = load_config(Path("mkdocs.yml"))
+result = generate_llms_txt(config, site_dir=Path("site"))
+
+print(result.llms_txt)       # Index content
+print(result.llms_full_txt)  # Full content
+print(result.markdown_files) # List of written .md paths
+```
+
+## Compatibility
+
+- Produces output identical to mkdocs-llmstxt when configured the same way
+- Handles Unicode, international characters, and special characters
+- Works with Material for MkDocs, ReadTheDocs, and the default MkDocs theme
+- Parses configs containing Python YAML tags like `!python/object/apply`
+
+## License
+
+MIT

llmstxt_standalone-0.1.0/README.md

@@ -0,0 +1,148 @@
+# llmstxt-standalone
+
+[![CI](https://github.com/shaanmajid/llmstxt-standalone/actions/workflows/ci.yml/badge.svg)](https://github.com/shaanmajid/llmstxt-standalone/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/llmstxt-standalone)](https://pypi.org/project/llmstxt-standalone/)
+[![Python](https://img.shields.io/pypi/pyversions/llmstxt-standalone)](https://pypi.org/project/llmstxt-standalone/)
+[![License](https://img.shields.io/pypi/l/llmstxt-standalone)](https://github.com/shaanmajid/llmstxt-standalone/blob/main/LICENSE)
+[![codecov](https://codecov.io/gh/shaanmajid/llmstxt-standalone/graph/badge.svg)](https://codecov.io/gh/shaanmajid/llmstxt-standalone)
+[![prek](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/j178/prek/master/docs/assets/badge-v0.json)](https://github.com/j178/prek)
+
+Generate `/llms.txt`, `/llms-full.txt`, and per-page markdown files from built HTML documentation, following the [llms.txt spec](https://llmstxt.org/).
+
+This tool works on pre-built HTML, making it useful for environments that cannot run MkDocs plugins (e.g., [Zensical](https://zensical.com/)) or when you want llms.txt generation as a separate build step. For standard MkDocs workflows, see [mkdocs-llmstxt](https://github.com/pawamoy/mkdocs-llmstxt).
+
+## Installation
+
+Requires Python 3.10+.
+
+```bash
+# Run without installing
+uvx llmstxt-standalone
+
+# Install as a CLI tool
+uv tool install llmstxt-standalone  # or: pipx install
+
+# Add to a project
+uv add llmstxt-standalone  # or: pip install
+```
+
+## Usage
+
+```bash
+# Run from project root (expects mkdocs.yml and site/)
+llmstxt-standalone
+
+# Explicit paths
+llmstxt-standalone --config mkdocs.yml --site-dir ./build --output-dir ./dist
+
+# Preview without writing files
+llmstxt-standalone --dry-run
+
+# Suppress output
+llmstxt-standalone --quiet
+
+# Show detailed progress
+llmstxt-standalone --verbose
+```
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--config` | `-c` | `mkdocs.yml` | Path to MkDocs config file |
+| `--site-dir` | `-s` | `site` | Path to built HTML directory |
+| `--output-dir` | `-o` | same as site-dir | Where to write output files |
+| `--dry-run` | `-n` | | Preview without writing |
+| `--quiet` | `-q` | | Suppress output |
+| `--verbose` | `-v` | | Show detailed progress |
+| `--version` | `-V` | | Show version |
+
+## Output
+
+The tool generates three outputs:
+
+1. `llms.txt` — an index file with markdown links to all pages
+1. `llms-full.txt` — concatenated content of all pages
+1. Per-page `.md` files alongside the HTML
+
+The per-page markdown files make the URLs in `llms.txt` resolve to actual content. If your site is at `https://docs.example.com/`, the URL `https://docs.example.com/install/index.md` returns markdown instead of HTML.
+
+## Configuration
+
+The tool reads your `mkdocs.yml` for site metadata. You can configure llmstxt output explicitly or let it derive structure from your nav.
+
+### Explicit configuration
+
+```yaml
+plugins:
+  - llmstxt:
+      markdown_description: |
+        Extra context for LLMs about your project.
+      full_output: llms-full.txt
+      content_selector: article.md-content__inner
+      sections:
+        Getting Started:
+          - index.md
+          - install.md
+        Usage:
+          - guide/basics.md
+          - guide/advanced.md
+```
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `markdown_description` | `""` | Additional context for LLMs, appears after site description |
+| `full_output` | `llms-full.txt` | Filename for concatenated content |
+| `content_selector` | auto-detect | CSS selector for main content |
+| `sections` | derived from nav | Section names mapped to page lists |
+
+### Automatic fallback
+
+Without an explicit `llmstxt` plugin config, sections derive from your `nav` structure. Top-level pages go into a "Pages" section; nested nav items become sections named by their keys.
+
+### MkDocs settings
+
+The tool respects `use_directory_urls` from your mkdocs.yml. When enabled (the default), `install.md` maps to `install/index.md`; when disabled, it maps to `install.md`.
+
+### Content extraction
+
+If `content_selector` is not set, the tool tries these selectors in order:
+
+1. `.md-content__inner` (Material for MkDocs)
+1. `[role="main"]` (default MkDocs theme)
+1. `article`
+1. `main`
+1. The entire document
+
+### Title resolution
+
+Page titles resolve in this order:
+
+1. The title from your `nav` structure
+1. The HTML `<title>` tag (with site name suffix stripped)
+1. The first `<h1>` tag
+1. A title derived from the filename
+
+## Programmatic use
+
+```python
+from pathlib import Path
+from llmstxt_standalone.config import load_config
+from llmstxt_standalone.generate import generate_llms_txt
+
+config = load_config(Path("mkdocs.yml"))
+result = generate_llms_txt(config, site_dir=Path("site"))
+
+print(result.llms_txt)       # Index content
+print(result.llms_full_txt)  # Full content
+print(result.markdown_files) # List of written .md paths
+```
+
+## Compatibility
+
+- Produces output identical to mkdocs-llmstxt when configured the same way
+- Handles Unicode, international characters, and special characters
+- Works with Material for MkDocs, ReadTheDocs, and the default MkDocs theme
+- Parses configs containing Python YAML tags like `!python/object/apply`
+
+## License
+
+MIT

llmstxt_standalone-0.1.0/pyproject.toml

@@ -0,0 +1,61 @@
+[project]
+name = "llmstxt-standalone"
+version = "0.1.0"
+description = "Generate llms.txt from built HTML documentation"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+keywords = ["llms", "documentation", "markdown", "mkdocs"]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Documentation",
+    "Typing :: Typed",
+]
+
+authors = [{ name = "Shaan Majid", email = "shaanmajid64@gmail.com" }]
+
+dependencies = [
+    "typer>=0.9.0",
+    "pyyaml>=6.0",
+    "beautifulsoup4>=4.12",
+    "markdownify>=0.14,<2.0",
+    "mdformat>=0.7,<2.0",
+    "mdformat-tables>=1.0",
+]
+
+[project.scripts]
+llmstxt-standalone = "llmstxt_standalone.cli:app"
+
+[project.urls]
+Repository = "https://github.com/shaanmajid/llmstxt-standalone"
+Issues = "https://github.com/shaanmajid/llmstxt-standalone/issues"
+
+[dependency-groups]
+dev = [
+    { include-group = "hooks" },
+    { include-group = "lint" },
+    { include-group = "typecheck" },
+    { include-group = "test" },
+]
+hooks = ["prek>=0.3.0"]
+lint = ["ruff>=0.14.14"]
+typecheck = ["ty>=0.0.14"]
+test = ["pytest>=8.0.0", "pytest-cov>=6.0.0"]
+
+[tool.uv]
+# renovate: datasource=pypi depName=uv
+required-version = ">=0.9.27"
+
+[build-system]
+# renovate: datasource=pypi depName=uv_build
+requires = ["uv_build>=0.9.0,<=0.9.27"]
+build-backend = "uv_build"

llmstxt_standalone-0.1.0/src/llmstxt_standalone/__init__.py

@@ -0,0 +1,5 @@
+"""llmstxt-standalone: Generate llms.txt from built HTML documentation."""
+
+from importlib.metadata import version
+
+__version__ = version("llmstxt-standalone")

llmstxt_standalone-0.1.0/src/llmstxt_standalone/__main__.py

@@ -0,0 +1,6 @@
+"""Allow running as python -m llmstxt_standalone."""
+
+from llmstxt_standalone.cli import app
+
+if __name__ == "__main__":
+    app()

llmstxt_standalone-0.1.0/src/llmstxt_standalone/cli.py

@@ -0,0 +1,172 @@
+"""Command-line interface."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+from llmstxt_standalone import __version__
+from llmstxt_standalone.config import load_config
+from llmstxt_standalone.generate import build_llms_output, write_markdown_files
+
+app = typer.Typer(
+    help="Generate llms.txt from built HTML documentation.",
+    no_args_is_help=False,
+    context_settings={"help_option_names": ["-h", "--help"]},
+)
+
+
+def version_callback(value: bool) -> None:
+    """Print version and exit if --version flag is set."""
+    if value:
+        typer.echo(f"llmstxt-standalone {__version__}")
+        raise typer.Exit()
+
+
+@app.command()
+def main(
+    config: Annotated[
+        Path,
+        typer.Option("--config", "-c", help="Path to mkdocs.yml config file"),
+    ] = Path("mkdocs.yml"),
+    site_dir: Annotated[
+        Path,
+        typer.Option("--site-dir", "-s", help="Path to built HTML site directory"),
+    ] = Path("site"),
+    output_dir: Annotated[
+        Path | None,
+        typer.Option(
+            "--output-dir", "-o", help="Output directory (defaults to site-dir)"
+        ),
+    ] = None,
+    dry_run: Annotated[
+        bool,
+        typer.Option(
+            "--dry-run",
+            "-n",
+            help="Preview what would be generated without writing files",
+        ),
+    ] = False,
+    quiet: Annotated[
+        bool,
+        typer.Option("--quiet", "-q", help="Suppress output (exit code only)"),
+    ] = False,
+    verbose: Annotated[
+        bool,
+        typer.Option("--verbose", "-v", help="Show detailed progress"),
+    ] = False,
+    version: Annotated[
+        bool,
+        typer.Option(
+            "--version",
+            "-V",
+            callback=version_callback,
+            is_eager=True,
+            help="Show version",
+        ),
+    ] = False,
+) -> None:
+    """Generate llms.txt and llms-full.txt from built HTML documentation."""
+    # Resolve output directory
+    out_dir = output_dir or site_dir
+
+    # quiet overrides verbose
+    if quiet:
+        verbose = False
+
+    def log(msg: str, color: str = "green", err: bool = False) -> None:
+        if not quiet:
+            typer.secho(msg, fg=color, err=err)
+
+    # Validate inputs
+    if not config.exists():
+        typer.secho(f"Error: Config file not found: {config}", fg="red", err=True)
+        raise typer.Exit(1)
+
+    if not site_dir.exists():
+        typer.secho(f"Error: Site directory not found: {site_dir}", fg="red", err=True)
+        typer.secho(
+            "Hint: Run 'mkdocs build' first to generate the HTML documentation.",
+            fg="yellow",
+            err=True,
+        )
+        raise typer.Exit(1)
+
+    # Load config
+    try:
+        cfg = load_config(config)
+    except Exception as e:
+        typer.secho(f"Error loading config: {e}", fg="red", err=True)
+        raise typer.Exit(1) from None
+
+    # Validate sections
+    if not cfg.sections:
+        typer.secho("Error: No sections configured.", fg="red", err=True)
+        typer.secho(
+            "Add a 'nav' to your mkdocs.yml, or configure 'sections' "
+            "in the llmstxt plugin.",
+            fg="yellow",
+            err=True,
+        )
+        raise typer.Exit(1)
+
+    if verbose:
+        typer.echo(f"Site: {cfg.site_name}")
+        typer.echo(f"Sections: {list(cfg.sections.keys())}")
+        if dry_run:
+            typer.echo("Dry run - no files will be written")
+
+    # Generate content
+    build = build_llms_output(
+        config=cfg,
+        site_dir=site_dir,
+    )
+    try:
+        markdown_files = write_markdown_files(
+            build.pages,
+            output_dir=out_dir,
+            use_directory_urls=cfg.use_directory_urls,
+            dry_run=dry_run,
+        )
+    except (OSError, ValueError) as exc:
+        typer.secho(f"Error writing markdown files: {exc}", fg="red", err=True)
+        raise typer.Exit(1) from None
+
+    # Define output paths
+    llms_path = out_dir / "llms.txt"
+    full_path = out_dir / cfg.full_output
+
+    # Write output files (skip in dry-run mode)
+    if dry_run:
+        action = "Would generate"
+        color = "yellow"
+    else:
+        action = "Generated"
+        color = "green"
+        try:
+            out_dir.mkdir(parents=True, exist_ok=True)
+            llms_path.write_text(build.llms_txt, encoding="utf-8")
+            full_path.write_text(build.llms_full_txt, encoding="utf-8")
+        except OSError as exc:
+            typer.secho(f"Error writing output files: {exc}", fg="red", err=True)
+            raise typer.Exit(1) from None
+
+    log(f"{action} {llms_path} ({len(build.llms_txt):,} bytes)", color)
+    log(f"{action} {full_path} ({len(build.llms_full_txt):,} bytes)", color)
+    log(f"{action} {len(markdown_files)} markdown files", color)
+
+    if verbose and build.skipped:
+        log("Skipped files:", color="yellow", err=True)
+        for path, reason in build.skipped:
+            log(f"- {path} ({reason})", color="yellow", err=True)
+
+    if build.warnings:
+        log("Warnings:", color="yellow", err=True)
+        for warning in build.warnings:
+            log(f"- {warning}", color="yellow", err=True)
+
+
+if __name__ == "__main__":
+    app()

llmstxt_standalone-0.1.0/src/llmstxt_standalone/config/__init__.py

@@ -0,0 +1,6 @@
+"""Configuration loading and resolution."""
+
+from llmstxt_standalone.config.load import load_config
+from llmstxt_standalone.config.model import Config
+
+__all__ = ["Config", "load_config"]

llmstxt_standalone-0.1.0/src/llmstxt_standalone/config/derive.py

@@ -0,0 +1,39 @@
+"""Helpers for deriving config sections from nav."""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+def nav_to_sections(nav: list[Any]) -> dict[str, list[str]]:
+    """Convert nav structure to sections dict."""
+    sections: dict[str, list[str]] = {}
+
+    for item in nav:
+        if isinstance(item, dict):
+            for key, value in item.items():
+                if isinstance(value, str):
+                    # Top-level page, add to "Pages" section
+                    sections.setdefault("Pages", []).append(value)
+                elif isinstance(value, list):
+                    # Section with children
+                    pages = _extract_pages(value)
+                    if pages:
+                        sections[key] = pages
+
+    return sections
+
+
+def _extract_pages(items: list[Any]) -> list[str]:
+    """Extract page paths from nav items."""
+    pages: list[str] = []
+    for item in items:
+        if isinstance(item, str):
+            pages.append(item)
+        elif isinstance(item, dict):
+            for value in item.values():
+                if isinstance(value, str):
+                    pages.append(value)
+                elif isinstance(value, list):
+                    pages.extend(_extract_pages(value))
+    return pages

llmstxt_standalone-0.1.0/src/llmstxt_standalone/config/load.py

@@ -0,0 +1,93 @@
+"""Configuration loading from mkdocs.yml."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+from llmstxt_standalone.config.derive import nav_to_sections
+from llmstxt_standalone.config.model import Config
+from llmstxt_standalone.config.plugin import get_llmstxt_config
+
+DEFAULT_SITE_NAME = "Documentation"
+DEFAULT_FULL_OUTPUT = "llms-full.txt"
+
+
+class _PermissiveLoader(yaml.SafeLoader):
+    """SafeLoader that ignores unknown Python tags.
+
+    MkDocs extensions like pymdownx.slugs use Python-specific YAML tags
+    like !python/object/apply which SafeLoader rejects. This loader
+    treats them as raw strings to allow parsing the rest of the config.
+    """
+
+
+def _ignore_unknown(loader: yaml.Loader, tag_suffix: str, node: yaml.Node) -> str:
+    """Return the raw tag as a placeholder string."""
+    return f"<{node.tag}>"
+
+
+# Register handler for all Python tags (both full and shorthand forms)
+_PermissiveLoader.add_multi_constructor("tag:yaml.org,2002:python/", _ignore_unknown)
+_PermissiveLoader.add_multi_constructor("!python/", _ignore_unknown)
+
+
+def load_config(config_path: Path) -> Config:
+    """Load and resolve configuration from mkdocs.yml.
+
+    Args:
+        config_path: Path to mkdocs.yml file.
+
+    Returns:
+        Resolved Config object.
+
+    Raises:
+        FileNotFoundError: If config file doesn't exist.
+    """
+    if not config_path.exists():
+        raise FileNotFoundError(f"Config file not found: {config_path}")
+
+    with open(config_path, encoding="utf-8") as f:
+        raw = yaml.load(f, Loader=_PermissiveLoader)
+
+    if not isinstance(raw, dict):
+        raise ValueError(f"Config file must be a mapping: {config_path}")
+
+    return _config_from_mkdocs(raw)
+
+
+def _config_from_mkdocs(raw: dict[str, Any]) -> Config:
+    """Build a Config from a parsed mkdocs.yml mapping."""
+    site_name = raw.get("site_name", DEFAULT_SITE_NAME)
+    site_description = raw.get("site_description", "")
+    site_url = raw.get("site_url", "").rstrip("/")
+    nav = raw.get("nav", [])
+    # MkDocs defaults use_directory_urls to true
+    use_directory_urls = raw.get("use_directory_urls", True)
+
+    llmstxt_config = get_llmstxt_config(raw)
+
+    if llmstxt_config is not None:
+        markdown_description = llmstxt_config.get("markdown_description", "")
+        full_output = llmstxt_config.get("full_output", DEFAULT_FULL_OUTPUT)
+        content_selector = llmstxt_config.get("content_selector")
+        sections = llmstxt_config.get("sections", {})
+    else:
+        markdown_description = ""
+        full_output = DEFAULT_FULL_OUTPUT
+        content_selector = None
+        sections = nav_to_sections(nav)
+
+    return Config(
+        site_name=site_name,
+        site_description=site_description,
+        site_url=site_url,
+        markdown_description=markdown_description,
+        full_output=full_output,
+        content_selector=content_selector,
+        sections=sections,
+        nav=nav,
+        use_directory_urls=use_directory_urls,
+    )

llmstxt_standalone-0.1.0/src/llmstxt_standalone/config/model.py

@@ -0,0 +1,60 @@
+"""Configuration model and derived helpers."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+
+@dataclass
+class Config:
+    """Resolved configuration for llmstxt generation."""
+
+    site_name: str
+    site_description: str
+    site_url: str
+    markdown_description: str
+    full_output: str
+    content_selector: str | None
+    sections: dict[str, list[str]]
+    nav: list[Any]
+    use_directory_urls: bool = True
+
+    def get_nav_title(self, md_path: str) -> str | None:
+        """Find the title for a page from the nav structure only.
+
+        Returns None if the page is not found in nav or has no explicit title.
+        """
+        return self._search_nav(self.nav, md_path, section_title=None)
+
+    def get_filename_title(self, md_path: str) -> str:
+        """Derive title from filename path."""
+        return md_path.replace(".md", "").replace("-", " ").replace("/", " - ").title()
+
+    def get_page_title(self, md_path: str) -> str:
+        """Find the title for a page from the nav structure with fallback."""
+        return self.get_nav_title(md_path) or self.get_filename_title(md_path)
+
+    def _search_nav(
+        self, items: list[Any], md_path: str, section_title: str | None
+    ) -> str | None:
+        """Recursively search nav for a page title.
+
+        Args:
+            items: Nav items to search (list of dicts or strings).
+            md_path: Markdown file path to find.
+            section_title: Title of the current section for bare string inheritance.
+        """
+        for item in items:
+            # Bare string in a section list inherits section title
+            if isinstance(item, str) and item == md_path:
+                return section_title
+            if isinstance(item, dict):
+                for key, value in item.items():
+                    if isinstance(value, str) and value == md_path:
+                        return key
+                    if isinstance(value, list):
+                        result = self._search_nav(value, md_path, section_title=key)
+                        if result:
+                            return result
+        return None

llmstxt_standalone-0.1.0/src/llmstxt_standalone/config/plugin.py

@@ -0,0 +1,42 @@
+"""MkDocs plugin configuration helpers."""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+def get_llmstxt_config(raw: dict[str, Any]) -> dict[str, Any] | None:
+    """Extract llmstxt plugin config from mkdocs.yml plugins.
+
+    MkDocs supports two plugin config styles:
+
+    List form:
+        plugins:
+          - llmstxt:
+              sections: ...
+
+    Mapping form:
+        plugins:
+          llmstxt:
+            sections: ...
+    """
+    plugins = raw.get("plugins")
+    if plugins is None:
+        return None
+
+    # Mapping form: plugins is a dict with plugin names as keys
+    if isinstance(plugins, dict):
+        if "llmstxt" in plugins:
+            config = plugins["llmstxt"]
+            # Plugin with no options is represented as empty dict or None
+            return config if isinstance(config, dict) else {}
+        return None
+
+    # List form: plugins is a list of strings or dicts
+    for plugin in plugins:
+        if isinstance(plugin, dict) and "llmstxt" in plugin:
+            config = plugin["llmstxt"]
+            return config if isinstance(config, dict) else {}
+        if plugin == "llmstxt":
+            return {}
+    return None

llmstxt_standalone-0.1.0/src/llmstxt_standalone/convert.py

@@ -0,0 +1,152 @@
+"""HTML to Markdown conversion."""
+
+from __future__ import annotations
+
+import mdformat
+from bs4 import BeautifulSoup, NavigableString, Tag
+from markdownify import ATX, MarkdownConverter
+
+
+def _should_remove(tag: Tag) -> bool:
+    """Check if a tag should be removed during autoclean."""
+    if tag.name in {"img", "svg"}:
+        return True
+    if tag.name == "a" and tag.img:
+        return True
+    classes = tag.get("class") or ()
+    if tag.name == "a" and "headerlink" in classes:
+        return True
+    if "twemoji" in classes:
+        return True
+    return "tabbed-labels" in classes
+
+
+def _autoclean(soup: BeautifulSoup | Tag) -> None:
+    """Remove unwanted elements from HTML."""
+    for element in soup.find_all(_should_remove):
+        element.decompose()
+
+    # Unwrap autoref elements
+    for element in soup.find_all("autoref"):
+        element.replace_with(NavigableString(element.get_text()))
+
+    # Remove line numbers from code blocks
+    for element in soup.find_all("table", attrs={"class": "highlighttable"}):
+        code = element.find("code")
+        if code:
+            element.replace_with(
+                BeautifulSoup(f"<pre>{code.get_text()}</pre>", "html.parser")
+            )
+
+
+def _get_language(tag: Tag) -> str:
+    """Extract language from code block classes.
+
+    The callback receives the <pre> tag, so we need to check:
+    1. Classes on the <pre> tag itself
+    2. Classes on the parent of <pre>
+    3. Classes on child <code> element (common pattern: <pre><code class="language-X">)
+    """
+    classes: list[str] = list(tag.get("class") or ())
+
+    # Check parent classes
+    if tag.parent:
+        classes.extend(tag.parent.get("class") or ())
+
+    # Check child <code> element classes
+    code_child = tag.find("code")
+    if code_child:
+        classes.extend(code_child.get("class") or ())
+
+    for css_class in classes:
+        if css_class.startswith("language-"):
+            return css_class[9:]
+    return ""
+
+
+# Converter with mkdocs-llmstxt-compatible settings
+_converter = MarkdownConverter(
+    bullets="-",
+    code_language_callback=_get_language,
+    escape_underscores=False,
+    heading_style=ATX,
+)
+
+
+def extract_title_from_html(html: str, site_name: str | None = None) -> str | None:
+    """Extract page title from HTML.
+
+    Tries <title> tag first, then falls back to first <h1>.
+    Strips site name suffixes (e.g., "Page - Site Name" -> "Page") when provided.
+
+    Args:
+        html: Raw HTML content.
+        site_name: Site name to strip from title suffixes (e.g., "Page - Site").
+
+    Returns:
+        The page title, or None if not found.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+
+    # Try <title> tag first
+    title_tag = soup.find("title")
+    if title_tag:
+        title = title_tag.get_text().strip()
+        # Strip site name suffix only when it matches the configured site name.
+        if site_name and " - " in title:
+            base, suffix = title.rsplit(" - ", 1)
+            if suffix.strip().casefold() == site_name.strip().casefold():
+                title = base.strip()
+        if title:
+            return title
+
+    # Fall back to first <h1>
+    h1_tag = soup.find("h1")
+    if h1_tag:
+        text = h1_tag.get_text().strip()
+        if text:
+            return text
+
+    return None
+
+
+def html_to_markdown(html: str, content_selector: str | None = None) -> str:
+    """Convert HTML to clean Markdown.
+
+    Args:
+        html: Raw HTML content.
+        content_selector: Optional CSS selector for main content.
+            Defaults to Material for MkDocs selectors.
+
+    Returns:
+        Cleaned Markdown text.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+
+    # Find main content
+    if content_selector:
+        try:
+            content = soup.select_one(content_selector)
+        except Exception:
+            content = None
+        else:
+            if content is None:
+                return ""
+    else:
+        content = None
+
+    if content is None:
+        content = (
+            soup.select_one(".md-content__inner")  # Material for MkDocs
+            or soup.select_one('[role="main"]')  # Default MkDocs theme
+            or soup.select_one("article")
+            or soup.select_one("main")
+            or soup
+        )
+
+    if content is None:
+        return ""
+
+    _autoclean(content)
+    md = _converter.convert_soup(content)
+    return mdformat.text(md, options={"wrap": "no"}, extensions=("tables",))

llmstxt_standalone-0.1.0/src/llmstxt_standalone/generate.py

@@ -0,0 +1,343 @@
+"""Main generation orchestration."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+from llmstxt_standalone.config import Config
+from llmstxt_standalone.convert import extract_title_from_html, html_to_markdown
+
+
+def _escape_markdown_link_text(text: str) -> str:
+    r"""Escape characters that break markdown link syntax.
+
+    Args:
+        text: The link text to escape.
+
+    Returns:
+        Text with backslashes, brackets escaped and newlines replaced with spaces.
+    """
+    return (
+        text.replace("\\", "\\\\")
+        .replace("[", r"\[")
+        .replace("]", r"\]")
+        .replace("\r\n", " ")
+        .replace("\n", " ")
+        .replace("\r", " ")
+    )
+
+
+def _is_index_md(md_path: str) -> bool:
+    return md_path == "index.md" or md_path.endswith("/index.md")
+
+
+def _ensure_safe_md_path(md_path: str) -> Path:
+    path = Path(md_path)
+    if path.is_absolute() or path.drive:
+        raise ValueError(f"Markdown path must be relative: {md_path}")
+    if ".." in path.parts:
+        raise ValueError(f"Markdown path must not contain '..': {md_path}")
+    return path
+
+
+def _ensure_within_dir(base_dir: Path, path: Path, label: str) -> Path:
+    base_resolved = base_dir.resolve(strict=False)
+    path_resolved = path.resolve(strict=False)
+    if not path_resolved.is_relative_to(base_resolved):
+        raise ValueError(f"{label} resolves outside {base_resolved}: {path}")
+    return path
+
+
+def md_path_to_html_path(
+    site_dir: Path, md_path: str, use_directory_urls: bool = True
+) -> Path:
+    """Convert docs/foo.md path to site HTML path.
+
+    Args:
+        site_dir: Path to the built site directory.
+        md_path: Relative markdown file path (e.g., "install.md").
+        use_directory_urls: If True, maps to foo/index.html; if False, maps to foo.html.
+
+    Returns:
+        Path to the corresponding HTML file.
+    """
+    # Handle index.md at any level (root or nested like foo/bar/index.md)
+    safe_md_path = _ensure_safe_md_path(md_path)
+    if _is_index_md(md_path):
+        html_path = site_dir / safe_md_path.with_suffix(".html")
+        return _ensure_within_dir(site_dir, html_path, "HTML path")
+    if use_directory_urls:
+        html_path = site_dir / safe_md_path.with_suffix("") / "index.html"
+        return _ensure_within_dir(site_dir, html_path, "HTML path")
+    html_path = site_dir / safe_md_path.with_suffix(".html")
+    return _ensure_within_dir(site_dir, html_path, "HTML path")
+
+
+def md_path_to_page_url(
+    site_url: str,
+    md_path: str,
+    use_directory_urls: bool = True,
+) -> str:
+    """Convert docs/foo.md path to markdown file URL on the deployed site.
+
+    Args:
+        site_url: Base URL of the site.
+        md_path: Relative markdown file path (e.g., "install.md").
+        use_directory_urls: If True, directory-style URLs; if False, flat URLs.
+
+    Returns:
+        URL to the markdown file on the deployed site.
+    """
+    if not site_url:
+        if _is_index_md(md_path):
+            return md_path
+        if use_directory_urls:
+            return md_path.replace(".md", "") + "/index.md"
+        return md_path
+    # Handle index.md at any level (root or nested like foo/bar/index.md)
+    if _is_index_md(md_path):
+        return f"{site_url}/{md_path}"
+    if use_directory_urls:
+        return f"{site_url}/{md_path.replace('.md', '')}/index.md"
+    return f"{site_url}/{md_path}"
+
+
+def md_path_to_output_md_path(
+    site_dir: Path, md_path: str, use_directory_urls: bool = True
+) -> Path:
+    """Convert docs/foo.md path to site markdown output path.
+
+    Args:
+        site_dir: Path to the built site directory.
+        md_path: Relative markdown file path (e.g., "install.md").
+        use_directory_urls: If True, outputs to foo/index.md; if False, outputs to foo.md.
+
+    Returns:
+        Path where the markdown file should be written.
+    """
+    # Handle index.md at any level (root or nested like foo/bar/index.md)
+    safe_md_path = _ensure_safe_md_path(md_path)
+    if _is_index_md(md_path):
+        output_path = site_dir / safe_md_path
+        return _ensure_within_dir(site_dir, output_path, "Output path")
+    if use_directory_urls:
+        output_path = site_dir / safe_md_path.with_suffix("") / "index.md"
+        return _ensure_within_dir(site_dir, output_path, "Output path")
+    output_path = site_dir / safe_md_path
+    return _ensure_within_dir(site_dir, output_path, "Output path")
+
+
+@dataclass
+class PageMarkdown:
+    """Per-page markdown output."""
+
+    md_path: str
+    content: str
+
+
+@dataclass
+class BuildResult:
+    """Result of building llms.txt content (no files written)."""
+
+    llms_txt: str
+    llms_full_txt: str
+    pages: list[PageMarkdown]
+    skipped: list[tuple[Path, str]]
+    warnings: list[str]
+
+
+@dataclass
+class GenerateResult:
+    """Result of llms.txt generation with files written."""
+
+    llms_txt: str
+    llms_full_txt: str
+    markdown_files: list[Path]
+    skipped: list[tuple[Path, str]]
+    warnings: list[str]
+
+
+def build_llms_output(
+    config: Config,
+    site_dir: Path,
+) -> BuildResult:
+    """Build llms.txt, llms-full.txt, and per-page markdown content.
+
+    Args:
+        config: Resolved configuration.
+        site_dir: Path to built HTML site directory.
+
+    Returns:
+        BuildResult with content and per-page markdown data.
+    """
+    # Build llms.txt (index)
+    llms_lines = [f"# {config.site_name}", ""]
+
+    if config.site_description:
+        llms_lines.append(f"> {config.site_description}")
+        llms_lines.append("")
+
+    if config.markdown_description:
+        llms_lines.append(config.markdown_description.strip())
+        llms_lines.append("")
+
+    # Build llms-full.txt header
+    full_lines = [f"# {config.site_name}", ""]
+
+    if config.site_description:
+        full_lines.append(f"> {config.site_description}")
+        full_lines.append("")
+
+    # Process sections - check HTML existence and extract titles first
+    page_outputs: list[PageMarkdown] = []
+    skipped: list[tuple[Path, str]] = []
+    warnings: list[str] = []
+
+    for section_name, section_pages in config.sections.items():
+        section_entries: list[str] = []
+
+        for md_path in section_pages:
+            try:
+                html_path = md_path_to_html_path(
+                    site_dir, md_path, config.use_directory_urls
+                )
+            except ValueError as exc:
+                skipped.append((site_dir / md_path, str(exc)))
+                continue
+
+            if not html_path.exists():
+                skipped.append((html_path, "HTML file not found"))
+                continue
+
+            try:
+                html = html_path.read_text(encoding="utf-8")
+            except UnicodeDecodeError:
+                skipped.append((html_path, "HTML file has encoding errors"))
+                continue
+            except OSError as exc:
+                skipped.append((html_path, f"Failed to read HTML file: {exc}"))
+                continue
+
+            # Prefer nav title (mkdocs-llmstxt compat), fall back to HTML, then filename
+            title = (
+                config.get_nav_title(md_path)
+                or extract_title_from_html(html, site_name=config.site_name)
+                or config.get_filename_title(md_path)
+            )
+
+            page_url = md_path_to_page_url(
+                config.site_url,
+                md_path,
+                config.use_directory_urls,
+            )
+            # Escape brackets in title to produce valid markdown links
+            escaped_title = _escape_markdown_link_text(title)
+            section_entries.append(f"- [{escaped_title}]({page_url})")
+
+            # Convert content for llms-full.txt
+            try:
+                content = html_to_markdown(html, config.content_selector)
+            except Exception as exc:
+                warning = f"Failed to convert HTML from {html_path}: {exc}"
+                warnings.append(warning)
+                content = ""
+
+            if content:
+                full_lines.append(f"## {title}")
+                full_lines.append("")
+                full_lines.append(content)
+                full_lines.append("")
+            else:
+                warning = (
+                    f"No markdown content extracted from {html_path}; content empty"
+                )
+                warnings.append(warning)
+
+            page_outputs.append(PageMarkdown(md_path=md_path, content=content))
+
+        # Only add section to llms.txt if it has entries
+        if section_entries:
+            llms_lines.append(f"## {section_name}")
+            llms_lines.append("")
+            llms_lines.extend(section_entries)
+            llms_lines.append("")
+
+    llms_txt = "\n".join(llms_lines)
+    llms_full_txt = "\n".join(full_lines)
+
+    return BuildResult(
+        llms_txt=llms_txt,
+        llms_full_txt=llms_full_txt,
+        pages=page_outputs,
+        skipped=skipped,
+        warnings=warnings,
+    )
+
+
+def write_markdown_files(
+    pages: list[PageMarkdown],
+    output_dir: Path,
+    use_directory_urls: bool,
+    dry_run: bool = False,
+) -> list[Path]:
+    """Write per-page markdown files to disk.
+
+    Args:
+        pages: Per-page markdown content.
+        output_dir: Path to write output files.
+        use_directory_urls: If True, outputs to foo/index.md; if False, outputs to foo.md.
+        dry_run: If True, don't write markdown files.
+
+    Returns:
+        List of output markdown paths (written or would-be).
+    """
+    markdown_files: list[Path] = []
+    for page in pages:
+        output_md_path = md_path_to_output_md_path(
+            output_dir, page.md_path, use_directory_urls
+        )
+        if not dry_run:
+            try:
+                output_md_path.parent.mkdir(parents=True, exist_ok=True)
+                output_md_path.write_text(page.content, encoding="utf-8")
+            except OSError as exc:
+                raise OSError(f"Failed to write {output_md_path}: {exc}") from exc
+        markdown_files.append(output_md_path)
+    return markdown_files
+
+
+def generate_llms_txt(
+    config: Config,
+    site_dir: Path,
+    output_dir: Path | None = None,
+    dry_run: bool = False,
+) -> GenerateResult:
+    """Generate llms.txt, llms-full.txt, and per-page markdown files.
+
+    Args:
+        config: Resolved configuration.
+        site_dir: Path to built HTML site directory.
+        output_dir: Path to write output files. Defaults to site_dir.
+        dry_run: If True, don't write markdown files.
+
+    Returns:
+        GenerateResult with content and list of markdown files (written or would-be).
+    """
+    build = build_llms_output(config=config, site_dir=site_dir)
+    if output_dir is None:
+        output_dir = site_dir
+    markdown_files = write_markdown_files(
+        build.pages,
+        output_dir=output_dir,
+        use_directory_urls=config.use_directory_urls,
+        dry_run=dry_run,
+    )
+
+    return GenerateResult(
+        llms_txt=build.llms_txt,
+        llms_full_txt=build.llms_full_txt,
+        markdown_files=markdown_files,
+        skipped=build.skipped,
+        warnings=build.warnings,
+    )