htomd 0.1.0__tar.gz

htomd-0.1.0/CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+## 0.1.0 — 2026-09-09
+
+Initial alpha release of htomd, a pure Python HTML-to-Markdown and metadata
+extractor for Python 3.12 and newer, with no runtime dependencies.
+
+- `convert()` extracts Markdown from decoded HTML.
+- `extract()` returns immutable Markdown, metadata, and diagnostic results.
+- Source URLs resolve relative references without fetching network content.
+- The `htomd` command and `python -m htomd` read UTF-8 HTML from stdin and write
+  Markdown or JSON, with help and installed-version commands.
+- The package includes typing information and an MIT license.
+
+Extraction is best suited to articles and documentation and may miss content or
+retain clutter. JavaScript, browser layout, math, and SVG are unsupported. Simple
+tables use GFM; complex tables become row/cell text.

htomd-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,33 @@
+# Contributing
+
+Follow the [README setup](README.md#development), then install the hooks:
+
+```sh
+mise exec -- uv run --locked pre-commit install
+mise run hooks
+mise run check
+```
+
+Hooks apply Ruff safe fixes and formatting to staged Python files, excluding
+fixtures. Review hook changes before staging. Checks include lint, formatting,
+strict mypy, offline tests, and workflow validation.
+
+Keep runtime dependencies empty and add focused regressions for behavior changes.
+See the [fixture guide](tests/fixtures/real/README.md) when updating snapshots.
+
+## Releases
+
+```sh
+mise run check
+mise run build
+mise exec -- uv run --locked twine check dist/*
+mise exec -- env RELEASE_TAG=v0.1.0 uv run --locked python tools/check_release.py
+```
+
+See [RELEASING.md](RELEASING.md) for the manual version, changelog, tag, and
+GitHub Release steps.
+
+The [release workflow](.github/workflows/release.yml) publishes to PyPI when a
+GitHub Release is published. It runs CI, verifies the tag matches the package
+version, validates both distributions, and uploads them to PyPI and the GitHub
+Release. Pushing a branch or tag alone does not publish a package.

htomd-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 JXD Ltd
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

htomd-0.1.0/MANIFEST.in

@@ -0,0 +1,6 @@
+include LICENSE README.md CONTRIBUTING.md RELEASING.md CHANGELOG.md
+recursive-include src/htomd *.py py.typed
+prune tests
+prune tools
+prune .github
+exclude .pre-commit-config.yaml uv.lock

htomd-0.1.0/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.4
+Name: htomd
+Version: 0.1.0
+Summary: Focused Markdown and metadata from messy HTML, in pure Python
+License-Expression: MIT
+Project-URL: Repository, https://github.com/jamiedavenport/htomd
+Project-URL: Issues, https://github.com/jamiedavenport/htomd/issues
+Project-URL: Changelog, https://github.com/jamiedavenport/htomd/blob/main/CHANGELOG.md
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# htomd
+
+Extract Markdown and metadata from HTML. Pure Python 3.12+, with no runtime
+dependencies or network access.
+
+## Installation
+
+```sh
+python -m pip install htomd
+```
+
+For the command line, install in an isolated tool environment:
+
+```sh
+uv tool install htomd
+```
+
+## Python API
+
+```python
+import htomd
+
+html = "<article><h1>Hello</h1><p>Readable text.</p></article>"
+markdown = htomd.convert(html)
+document = htomd.extract(html, url="https://example.com/article")
+print(document.markdown, document.metadata.title, document.diagnostics)
+```
+
+Pass decoded HTML strings. `url` resolves relative references. `extract()` returns
+immutable results; missing metadata is `None`. Empty content yields empty Markdown;
+invalid argument types raise `TypeError`.
+
+Best suited to articles and documentation. Extraction can miss content or retain
+clutter. JavaScript, browser layout, math, and SVG are unsupported. Simple tables
+use GFM; complex tables become row/cell text.
+
+## Command line
+
+Installing the package also installs the `htomd` command. To install a locally
+built wheel in a virtual environment:
+
+```sh
+python -m venv .venv
+source .venv/bin/activate
+python -m pip install dist/htomd-0.1.0-py3-none-any.whl
+```
+
+In Windows PowerShell, activate with `.venv\Scripts\Activate.ps1` instead.
+See [Contributing](https://github.com/jamiedavenport/htomd/blob/main/CONTRIBUTING.md#releases)
+for the build commands.
+
+```sh
+curl -s https://example.com/article | htomd convert --url https://example.com/article
+cat page.html | htomd convert > page.md
+cat page.html | htomd extract > page.json
+```
+
+Both commands read stdin to EOF and accept no file arguments. Input must be UTF-8
+(an optional UTF-8 BOM is accepted); output is UTF-8. `--url` supplies source
+context for relative references and metadata; it never fetches the URL.
+
+`convert` writes Markdown unchanged. `extract` writes indented JSON containing
+`markdown`, `metadata`, and `diagnostics`, matching the Python result structure.
+Missing metadata is `null`; diagnostic notes are an array. Empty or malformed
+HTML receives the same best-effort handling as the library.
+
+Run `htomd`, `htomd help`, or `htomd --help` for usage and pipeline examples.
+Use `htomd help convert` or `htomd convert --help` for command-specific help
+(likewise for `extract`). `htomd version` or `htomd --version` prints the installed
+package version. Help and version commands exit successfully without reading stdin.
+
+`python -m htomd` supports the same commands. Successful conversion exits with code 0, I/O and
+decoding failures with code 1, and usage errors with code 2. Errors go to stderr.
+
+## Development
+
+```sh
+mise trust
+mise install
+mise run setup
+mise exec -- uv run --locked pytest
+```
+
+`mise run check` runs all checks. See
+[CONTRIBUTING.md](https://github.com/jamiedavenport/htomd/blob/main/CONTRIBUTING.md)
+for hooks and releases.
+
+MIT license, copyright 2026 JXD Ltd.
+[Fixtures](https://github.com/jamiedavenport/htomd/blob/main/tests/fixtures/real/README.md)
+have separate licenses.

htomd-0.1.0/README.md

@@ -0,0 +1,90 @@
+# htomd
+
+Extract Markdown and metadata from HTML. Pure Python 3.12+, with no runtime
+dependencies or network access.
+
+## Installation
+
+```sh
+python -m pip install htomd
+```
+
+For the command line, install in an isolated tool environment:
+
+```sh
+uv tool install htomd
+```
+
+## Python API
+
+```python
+import htomd
+
+html = "<article><h1>Hello</h1><p>Readable text.</p></article>"
+markdown = htomd.convert(html)
+document = htomd.extract(html, url="https://example.com/article")
+print(document.markdown, document.metadata.title, document.diagnostics)
+```
+
+Pass decoded HTML strings. `url` resolves relative references. `extract()` returns
+immutable results; missing metadata is `None`. Empty content yields empty Markdown;
+invalid argument types raise `TypeError`.
+
+Best suited to articles and documentation. Extraction can miss content or retain
+clutter. JavaScript, browser layout, math, and SVG are unsupported. Simple tables
+use GFM; complex tables become row/cell text.
+
+## Command line
+
+Installing the package also installs the `htomd` command. To install a locally
+built wheel in a virtual environment:
+
+```sh
+python -m venv .venv
+source .venv/bin/activate
+python -m pip install dist/htomd-0.1.0-py3-none-any.whl
+```
+
+In Windows PowerShell, activate with `.venv\Scripts\Activate.ps1` instead.
+See [Contributing](https://github.com/jamiedavenport/htomd/blob/main/CONTRIBUTING.md#releases)
+for the build commands.
+
+```sh
+curl -s https://example.com/article | htomd convert --url https://example.com/article
+cat page.html | htomd convert > page.md
+cat page.html | htomd extract > page.json
+```
+
+Both commands read stdin to EOF and accept no file arguments. Input must be UTF-8
+(an optional UTF-8 BOM is accepted); output is UTF-8. `--url` supplies source
+context for relative references and metadata; it never fetches the URL.
+
+`convert` writes Markdown unchanged. `extract` writes indented JSON containing
+`markdown`, `metadata`, and `diagnostics`, matching the Python result structure.
+Missing metadata is `null`; diagnostic notes are an array. Empty or malformed
+HTML receives the same best-effort handling as the library.
+
+Run `htomd`, `htomd help`, or `htomd --help` for usage and pipeline examples.
+Use `htomd help convert` or `htomd convert --help` for command-specific help
+(likewise for `extract`). `htomd version` or `htomd --version` prints the installed
+package version. Help and version commands exit successfully without reading stdin.
+
+`python -m htomd` supports the same commands. Successful conversion exits with code 0, I/O and
+decoding failures with code 1, and usage errors with code 2. Errors go to stderr.
+
+## Development
+
+```sh
+mise trust
+mise install
+mise run setup
+mise exec -- uv run --locked pytest
+```
+
+`mise run check` runs all checks. See
+[CONTRIBUTING.md](https://github.com/jamiedavenport/htomd/blob/main/CONTRIBUTING.md)
+for hooks and releases.
+
+MIT license, copyright 2026 JXD Ltd.
+[Fixtures](https://github.com/jamiedavenport/htomd/blob/main/tests/fixtures/real/README.md)
+have separate licenses.

htomd-0.1.0/RELEASING.md

@@ -0,0 +1,97 @@
+# Releasing htomd
+
+Releases are prepared manually. Publishing a GitHub Release starts the
+[release workflow](.github/workflows/release.yml), which runs CI, validates the
+version and distributions, publishes to PyPI, and attaches the wheel and source
+archive to the GitHub Release.
+
+## Publishing identity
+
+PyPI Trusted Publishing uses these settings:
+
+| Field | Value |
+| --- | --- |
+| Project | `htomd` |
+| Repository owner | `jamiedavenport` |
+| Repository | `htomd` |
+| Workflow filename | `release.yml` |
+| GitHub environment | `pypi` |
+
+Configure the publisher in the project's PyPI settings, or use an account-level
+pending publisher for the first upload. No stored PyPI API token is needed.
+See [PyPI's setup instructions](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/).
+
+## 1. Prepare the version and release notes
+
+Set the version in `pyproject.toml` and refresh `uv.lock` if the version changed.
+Update `CHANGELOG.md` with the release date and user-facing changes. Version
+`0.1.0` is used in the examples below; substitute the new version for later
+releases.
+
+## 2. Validate, commit, and push
+
+```sh
+mise run check
+mise run build
+mise exec -- uv run --locked twine check dist/*
+mise exec -- env RELEASE_TAG=v0.1.0 uv run --locked python tools/check_release.py
+```
+
+The build inspects package contents and installs both artifacts in separate
+virtual environments to exercise the API and CLI. Keep exactly one wheel and
+one source archive in `dist/`; move aside old-version artifacts before building.
+
+Commit and push the release changes to `main`. Wait for all CI jobs to pass,
+including the Windows package build. The release workflow must be included in
+the commit you tag.
+
+## 3. Create the tag and GitHub Release
+
+From the clean, validated `main` checkout:
+
+```sh
+git tag -a v0.1.0 -m "htomd 0.1.0"
+git push origin v0.1.0
+gh release create v0.1.0 --verify-tag --title "htomd 0.1.0" --notes-file CHANGELOG.md
+```
+
+For later releases, pass a file containing only that version's changelog entry
+as `--notes-file`. Pushing the tag alone does not publish. Publishing the GitHub
+Release triggers the PyPI upload; saving a draft does not.
+
+## 4. Monitor publishing and verify installation
+
+Find the new Release run, then watch it using its numeric ID:
+
+```sh
+gh run list --workflow release.yml --limit 5
+gh run watch RUN_ID --exit-status
+```
+
+Verify PyPI installation in an isolated environment:
+
+```sh
+mise exec -- uv run --isolated --no-project --refresh-package htomd \
+  --default-index https://pypi.org/simple/ --with htomd==0.1.0 \
+  python -I -c "import htomd; assert htomd.convert('<h1>Hello</h1>') == '# Hello\n'"
+mise exec -- uv run --isolated --no-project --refresh-package htomd \
+  --default-index https://pypi.org/simple/ --with htomd==0.1.0 \
+  htomd --version
+```
+
+Check the package page on [PyPI](https://pypi.org/project/htomd/) and both
+attachments on [GitHub Releases](https://github.com/jamiedavenport/htomd/releases).
+
+If publisher configuration fails, correct it and rerun the failed jobs on the
+same release. If only the release-asset job fails, PyPI publication has already
+succeeded; rerun that job to attach the existing artifacts. Published artifacts
+cannot be replaced with changed files; corrections need a new version.
+
+## Documentation references
+
+Commands were checked against uv `0.12.11` using Context7 library ID
+`/astral-sh/uv` (requested version `0.12.11`; current unversioned documentation)
+and the [uv publishing guide](https://docs.astral.sh/uv/guides/package/).
+The GitHub CLI release command was checked against installed `gh` `2.100.0` and
+Context7 library ID `/websites/cli_github_manual` (unversioned), alongside the
+[GitHub CLI manual](https://cli.github.com/manual/gh_release_create).

htomd-0.1.0/pyproject.toml

@@ -0,0 +1,65 @@
+[build-system]
+requires = ["setuptools==82.0.1"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "htomd"
+version = "0.1.0"
+description = "Focused Markdown and metadata from messy HTML, in pure Python"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "MIT"
+license-files = ["LICENSE"]
+dependencies = []
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Typing :: Typed",
+]
+
+[project.scripts]
+htomd = "htomd._cli:main"
+
+[project.urls]
+Repository = "https://github.com/jamiedavenport/htomd"
+Issues = "https://github.com/jamiedavenport/htomd/issues"
+Changelog = "https://github.com/jamiedavenport/htomd/blob/main/CHANGELOG.md"
+
+[dependency-groups]
+dev = [
+  "mypy==1.20.2",
+  "pre-commit==4.5.1",
+  "pytest==9.0.2",
+  "ruff==0.15.6",
+  "setuptools==82.0.1",
+  "twine==6.2.0",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+htomd = ["py.typed"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-ra"
+
+[tool.ruff]
+target-version = "py312"
+line-length = 100
+extend-exclude = ["tests/fixtures"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM"]
+# Explicit branches are clearer for parser and renderer state transitions.
+ignore = ["SIM108"]
+
+[tool.mypy]
+python_version = "3.12"
+strict = true
+files = ["src", "tests", "tools"]
+exclude = ["tests/fixtures"]

htomd-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

htomd-0.1.0/src/htomd/__init__.py

@@ -0,0 +1,37 @@
+"""Focused Markdown and metadata from decoded HTML, with no network access."""
+
+from ._metadata import read_metadata, refine_metadata
+from ._models import Diagnostics, Document, Metadata
+from ._parser import parse
+from ._render import render
+from ._selection import select
+from ._urls import document_base
+
+__all__ = ["Diagnostics", "Document", "Metadata", "convert", "extract"]
+
+
+def extract(html: str, *, url: str | None = None) -> Document:
+    """Extract relevant Markdown and explicit metadata from an HTML string.
+
+    ``url`` provides source context and resolves references; no fetching occurs.
+    Malformed HTML receives best-effort recovery. Result dataclasses are immutable.
+    """
+    if not isinstance(html, str):
+        raise TypeError("html must be a decoded str")
+    if url is not None and not isinstance(url, str):
+        raise TypeError("url must be a str or None")
+    root, parsing_notes = parse(html)
+    base = document_base(root, url)
+    metadata = read_metadata(root, url, base)
+    selected, diagnostics = select(root)
+    metadata = refine_metadata(metadata, selected)
+    markdown = render(selected, base)
+    diagnostics = Diagnostics(
+        diagnostics.strategy if markdown else "none", parsing_notes + diagnostics.notes
+    )
+    return Document(markdown, metadata, diagnostics)
+
+
+def convert(html: str, *, url: str | None = None) -> str:
+    """Return the Markdown produced by :func:`extract`."""
+    return extract(html, url=url).markdown

htomd-0.1.0/src/htomd/__main__.py

@@ -0,0 +1,6 @@
+"""Support ``python -m htomd``."""
+
+from ._cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

htomd-0.1.0/src/htomd/_cli.py

@@ -0,0 +1,92 @@
+"""Command-line conversion of HTML supplied on stdin."""
+
+import argparse
+import json
+import os
+import sys
+from dataclasses import asdict
+from importlib.metadata import version
+
+from . import convert, extract
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="htomd",
+        description=(
+            "Extract Markdown and metadata from UTF-8 HTML on stdin. "
+            "Pipe input from cat or curl; output goes to stdout. No file arguments or fetching."
+        ),
+        epilog=(
+            "Examples:\n"
+            "  cat page.html | htomd convert > page.md\n"
+            "  cat page.html | htomd extract > page.json\n"
+            "  curl -fsSL https://example.com | htomd convert --url https://example.com\n"
+            "\nUse 'htomd help COMMAND' for command-specific help."
+        ),
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    version_text = f"htomd {version('htomd')}"
+    parser.add_argument("--version", action="version", version=version_text)
+    commands = parser.add_subparsers(dest="command")
+    command_parsers: dict[str, argparse.ArgumentParser] = {}
+    for name, description in (
+        ("convert", "Write Markdown to stdout."),
+        ("extract", "Write Markdown, metadata, and diagnostics as JSON to stdout."),
+    ):
+        command = commands.add_parser(
+            name,
+            help=description,
+            description=f"Read UTF-8 HTML from stdin to EOF. {description}",
+            epilog=f"Example: cat page.html | htomd {name}",
+        )
+        command.add_argument(
+            "--url", help="Source URL for resolving references; no fetching occurs."
+        )
+        command_parsers[name] = command
+    command_parsers["version"] = commands.add_parser(
+        "version",
+        help="Show the installed package version.",
+        description="Show the installed version.",
+    )
+    help_parser = commands.add_parser(
+        "help",
+        help="Show general or command-specific help.",
+        description="Show help for a command.",
+    )
+    command_parsers["help"] = help_parser
+    help_parser.add_argument(
+        "topic", nargs="?", choices=command_parsers, help="Command to describe."
+    )
+    args = parser.parse_args(argv)
+
+    if args.command is None or args.command == "help":
+        topic = args.topic if args.command == "help" else None
+        (command_parsers[topic] if topic else parser).print_help()
+        return 0
+    if args.command == "version":
+        print(version_text)
+        return 0
+
+    try:
+        html = sys.stdin.buffer.read().decode("utf-8-sig")
+        if args.command == "convert":
+            output = convert(html, url=args.url)
+        else:
+            output = json.dumps(asdict(extract(html, url=args.url)), ensure_ascii=False, indent=2)
+            output += "\n"
+    except (OSError, UnicodeError) as error:
+        print(f"htomd: {error}", file=sys.stderr)
+        return 1
+
+    try:
+        sys.stdout.buffer.write(output.encode("utf-8"))
+        sys.stdout.buffer.flush()
+    except OSError as error:
+        # Prevent a second output failure when Python flushes stdout during shutdown.
+        with open(os.devnull, "wb") as sink:
+            os.dup2(sink.fileno(), sys.stdout.fileno())
+        if not isinstance(error, BrokenPipeError):
+            print(f"htomd: {error}", file=sys.stderr)
+        return 1
+    return 0