mwextract 0.1.0__tar.gz

mwextract-0.1.0/.gitignore

@@ -0,0 +1,27 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+
+# Tooling
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.DS_Store
+
+# Local data / outputs
+source_data/
+extracted_data/
+out/
+*.7z
+*.xml
+*.xml.bz2
+
+# ...but keep test fixtures
+!tests/fixtures/
+!tests/fixtures/*.xml

mwextract-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aditya Parab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mwextract-0.1.0/PKG-INFO

@@ -0,0 +1,183 @@
+Metadata-Version: 2.5
+Name: mwextract
+Version: 0.1.0
+Summary: Convert a MediaWiki XML dump (.7z / .xml / .xml.bz2) into structured Markdown, JSON, or XML.
+Project-URL: Homepage, https://github.com/adityaparab/mwextract
+Project-URL: Repository, https://github.com/adityaparab/mwextract
+Project-URL: Issues, https://github.com/adityaparab/mwextract/issues
+Author-email: Aditya Parab <mradityaparab@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: converter,dump,fandom,markdown,mediawiki,rag,wiki,wikia,wikitext
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Markup
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Requires-Dist: mwparserfromhell>=0.6
+Requires-Dist: mwxml>=0.3.3
+Requires-Dist: py7zr>=0.20
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# mwextract
+
+Convert a **MediaWiki XML dump** (as shipped by Fandom/Wikia and other MediaWiki
+sites) into clean, structured **Markdown**, **JSON**, or **XML** — one command,
+no pandoc, no other binaries.
+
+It uses only the Wikimedia parsing stack:
+
+- [`mwxml`](https://pypi.org/project/mwxml/) streams pages out of the dump
+  (memory-safe, namespace-aware).
+- [`mwparserfromhell`](https://pypi.org/project/mwparserfromhell/) parses
+  wikitext: infobox/template fields become metadata, and the article body is
+  rendered to clean Markdown.
+- [`py7zr`](https://pypi.org/project/py7zr/) extracts the `.7z` archive the dump
+  usually comes in.
+
+## Install
+
+```bash
+uv add mwextract
+# or
+pip install mwextract
+```
+
+## Get a dump
+
+`mwextract` does **not** download anything — you fetch the dump yourself. Fandom
+wikis expose current-page dumps under a URL like:
+
+```
+https://s3.amazonaws.com/wikia_xml_dumps/<x>/<xx>/<wiki>_pages_current.xml.7z
+```
+
+For example, the Game of Thrones wiki:
+`https://s3.amazonaws.com/wikia_xml_dumps/g/ga/gameofthrones_pages_current.xml.7z`
+
+Download the `.7z` and point `mwextract` at it.
+
+## Use it from Python
+
+```python
+import mwextract
+
+result = mwextract.convert(
+    "gameofthrones_pages_current.xml.7z",   # .7z, .xml, or .xml.bz2
+    "out/",                                  # output directory
+    base_url="https://gameofthrones.fandom.com/wiki/",  # optional
+    fmt="md",                                # "md" | "json" | "xml"
+    layout="combined",                       # "combined" | "per-file"
+)
+
+print(result.pages, "pages ->", result.outputs)
+```
+
+`convert()` extracts the archive to a temporary directory (removed afterwards),
+streams every main-namespace, non-redirect article, and writes the chosen format.
+
+## Use it from the command line
+
+```bash
+mwextract gameofthrones_pages_current.xml.7z out/ \
+  --base-url https://gameofthrones.fandom.com/wiki/ \
+  --format md \
+  --layout combined
+```
+
+```
+usage: mwextract [-h] [-b BASE_URL] [-f {md,json,xml}]
+                 [-l {combined,per-file}] [-q] [-V]
+                 dump output_dir
+```
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `dump` | — | Path to the dump: `.7z`, `.xml`, or `.xml.bz2`. |
+| `output_dir` | — | Directory to write into (created if missing). |
+| `-b`, `--base-url` | *(none)* | Wiki URL prefix; when set, each page gets a `url`. |
+| `-f`, `--format` | `md` | `md`, `json`, or `xml`. |
+| `-l`, `--layout` | `combined` | `combined` (one file) or `per-file` (one per article). |
+| `-q`, `--quiet` | off | Only warnings/errors. |
+
+## Output
+
+### Layouts
+
+- **`combined`** (default) — a single file for the whole corpus:
+  `corpus.md`, `corpus.json` (a JSON array), or `corpus.xml`
+  (`<pages><page>…</page></pages>`).
+- **`per-file`** — one file per article, named after the title:
+  `Daenerys_Targaryen.md`, `.json`, or `.xml`.
+
+Both layouts are written incrementally as pages stream in, so memory stays flat
+no matter how large the dump is.
+
+### Formats
+
+**Markdown** — YAML front-matter (title, url, infobox fields) + an optional
+summary line + the rendered body:
+
+```markdown
+---
+title: "Jon Snow"
+url: https://gameofthrones.fandom.com/wiki/Jon_Snow
+house: "Stark"
+allegiance: "Night's Watch"
+---
+
+# Jon Snow
+
+> **House:** Stark · **Allegiance:** Night's Watch
+
+Jon Snow is the illegitimate son of ...
+```
+
+**JSON** — the same content, structured:
+
+```json
+{
+  "title": "Jon Snow",
+  "url": "https://gameofthrones.fandom.com/wiki/Jon_Snow",
+  "metadata": { "house": "Stark", "allegiance": "Night's Watch" },
+  "body": "Jon Snow is the illegitimate son of ..."
+}
+```
+
+**XML** — the same content as elements:
+
+```xml
+<page>
+  <title>Jon Snow</title>
+  <url>https://gameofthrones.fandom.com/wiki/Jon_Snow</url>
+  <metadata>
+    <field name="house">Stark</field>
+    <field name="allegiance">Night's Watch</field>
+  </metadata>
+  <body>Jon Snow is the illegitimate son of ...</body>
+</page>
+```
+
+`body` is the rendered Markdown body only — the structured `metadata`/`url`
+carry everything the Markdown front-matter would, so nothing is duplicated.
+
+## What gets kept / dropped
+
+- **Kept:** main-namespace articles, section hierarchy, bold/italic, list items,
+  wikilink and external-link visible text, infobox fields (as metadata).
+- **Dropped:** redirects and non-article namespaces, file/image/category/media
+  links, citation footnotes (`<ref>`), templates (after their infobox fields are
+  extracted), and comments.
+
+## License
+
+MIT © Aditya Parab

mwextract-0.1.0/README.md

@@ -0,0 +1,153 @@
+# mwextract
+
+Convert a **MediaWiki XML dump** (as shipped by Fandom/Wikia and other MediaWiki
+sites) into clean, structured **Markdown**, **JSON**, or **XML** — one command,
+no pandoc, no other binaries.
+
+It uses only the Wikimedia parsing stack:
+
+- [`mwxml`](https://pypi.org/project/mwxml/) streams pages out of the dump
+  (memory-safe, namespace-aware).
+- [`mwparserfromhell`](https://pypi.org/project/mwparserfromhell/) parses
+  wikitext: infobox/template fields become metadata, and the article body is
+  rendered to clean Markdown.
+- [`py7zr`](https://pypi.org/project/py7zr/) extracts the `.7z` archive the dump
+  usually comes in.
+
+## Install
+
+```bash
+uv add mwextract
+# or
+pip install mwextract
+```
+
+## Get a dump
+
+`mwextract` does **not** download anything — you fetch the dump yourself. Fandom
+wikis expose current-page dumps under a URL like:
+
+```
+https://s3.amazonaws.com/wikia_xml_dumps/<x>/<xx>/<wiki>_pages_current.xml.7z
+```
+
+For example, the Game of Thrones wiki:
+`https://s3.amazonaws.com/wikia_xml_dumps/g/ga/gameofthrones_pages_current.xml.7z`
+
+Download the `.7z` and point `mwextract` at it.
+
+## Use it from Python
+
+```python
+import mwextract
+
+result = mwextract.convert(
+    "gameofthrones_pages_current.xml.7z",   # .7z, .xml, or .xml.bz2
+    "out/",                                  # output directory
+    base_url="https://gameofthrones.fandom.com/wiki/",  # optional
+    fmt="md",                                # "md" | "json" | "xml"
+    layout="combined",                       # "combined" | "per-file"
+)
+
+print(result.pages, "pages ->", result.outputs)
+```
+
+`convert()` extracts the archive to a temporary directory (removed afterwards),
+streams every main-namespace, non-redirect article, and writes the chosen format.
+
+## Use it from the command line
+
+```bash
+mwextract gameofthrones_pages_current.xml.7z out/ \
+  --base-url https://gameofthrones.fandom.com/wiki/ \
+  --format md \
+  --layout combined
+```
+
+```
+usage: mwextract [-h] [-b BASE_URL] [-f {md,json,xml}]
+                 [-l {combined,per-file}] [-q] [-V]
+                 dump output_dir
+```
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `dump` | — | Path to the dump: `.7z`, `.xml`, or `.xml.bz2`. |
+| `output_dir` | — | Directory to write into (created if missing). |
+| `-b`, `--base-url` | *(none)* | Wiki URL prefix; when set, each page gets a `url`. |
+| `-f`, `--format` | `md` | `md`, `json`, or `xml`. |
+| `-l`, `--layout` | `combined` | `combined` (one file) or `per-file` (one per article). |
+| `-q`, `--quiet` | off | Only warnings/errors. |
+
+## Output
+
+### Layouts
+
+- **`combined`** (default) — a single file for the whole corpus:
+  `corpus.md`, `corpus.json` (a JSON array), or `corpus.xml`
+  (`<pages><page>…</page></pages>`).
+- **`per-file`** — one file per article, named after the title:
+  `Daenerys_Targaryen.md`, `.json`, or `.xml`.
+
+Both layouts are written incrementally as pages stream in, so memory stays flat
+no matter how large the dump is.
+
+### Formats
+
+**Markdown** — YAML front-matter (title, url, infobox fields) + an optional
+summary line + the rendered body:
+
+```markdown
+---
+title: "Jon Snow"
+url: https://gameofthrones.fandom.com/wiki/Jon_Snow
+house: "Stark"
+allegiance: "Night's Watch"
+---
+
+# Jon Snow
+
+> **House:** Stark · **Allegiance:** Night's Watch
+
+Jon Snow is the illegitimate son of ...
+```
+
+**JSON** — the same content, structured:
+
+```json
+{
+  "title": "Jon Snow",
+  "url": "https://gameofthrones.fandom.com/wiki/Jon_Snow",
+  "metadata": { "house": "Stark", "allegiance": "Night's Watch" },
+  "body": "Jon Snow is the illegitimate son of ..."
+}
+```
+
+**XML** — the same content as elements:
+
+```xml
+<page>
+  <title>Jon Snow</title>
+  <url>https://gameofthrones.fandom.com/wiki/Jon_Snow</url>
+  <metadata>
+    <field name="house">Stark</field>
+    <field name="allegiance">Night's Watch</field>
+  </metadata>
+  <body>Jon Snow is the illegitimate son of ...</body>
+</page>
+```
+
+`body` is the rendered Markdown body only — the structured `metadata`/`url`
+carry everything the Markdown front-matter would, so nothing is duplicated.
+
+## What gets kept / dropped
+
+- **Kept:** main-namespace articles, section hierarchy, bold/italic, list items,
+  wikilink and external-link visible text, infobox fields (as metadata).
+- **Dropped:** redirects and non-article namespaces, file/image/category/media
+  links, citation footnotes (`<ref>`), templates (after their infobox fields are
+  extracted), and comments.
+
+## License
+
+MIT © Aditya Parab

mwextract-0.1.0/pyproject.toml

@@ -0,0 +1,58 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "mwextract"
+version = "0.1.0"
+description = "Convert a MediaWiki XML dump (.7z / .xml / .xml.bz2) into structured Markdown, JSON, or XML."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Aditya Parab", email = "mradityaparab@gmail.com" }]
+keywords = [
+  "mediawiki",
+  "wiki",
+  "fandom",
+  "wikia",
+  "dump",
+  "wikitext",
+  "markdown",
+  "converter",
+  "rag",
+]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Environment :: Console",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Text Processing :: Markup",
+  "Topic :: Utilities",
+]
+dependencies = [
+  "py7zr>=0.20",
+  "mwxml>=0.3.3",
+  "mwparserfromhell>=0.6",
+]
+
+[project.urls]
+Homepage = "https://github.com/adityaparab/mwextract"
+Repository = "https://github.com/adityaparab/mwextract"
+Issues = "https://github.com/adityaparab/mwextract/issues"
+
+[project.scripts]
+mwextract = "mwextract.cli:main"
+
+[project.optional-dependencies]
+dev = ["pytest>=7"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/mwextract"]
+
+[tool.hatch.build.targets.sdist]
+include = ["src/mwextract", "README.md", "LICENSE"]

mwextract-0.1.0/src/mwextract/__init__.py

@@ -0,0 +1,111 @@
+"""mwextract -- convert a MediaWiki XML dump into Markdown, JSON, or XML.
+
+Typical use::
+
+    import mwextract
+
+    result = mwextract.convert(
+        "gameofthrones_pages_current.xml.7z",
+        "out/",
+        base_url="https://gameofthrones.fandom.com/wiki/",
+        fmt="md",
+    )
+    print(result.pages, "pages written to", result.output_dir)
+
+The input may be a ``.7z`` archive (extracted automatically), or an
+already-extracted ``.xml`` / ``.xml.bz2`` dump.
+"""
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import List, Optional
+
+from .archive import resolve_dump
+from .parser import PageDoc, iter_pages, parse_page
+from .writers import write
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "convert",
+    "ConvertResult",
+    "PageDoc",
+    "iter_pages",
+    "parse_page",
+    "FORMATS",
+    "LAYOUTS",
+    "__version__",
+]
+
+# Library code shouldn't configure logging; attach a no-op handler so callers
+# who never set up logging don't see "No handlers could be found" noise.
+logging.getLogger("mwextract").addHandler(logging.NullHandler())
+
+FORMATS = ("md", "json", "xml")
+LAYOUTS = ("combined", "per-file")
+
+
+@dataclass
+class ConvertResult:
+    """Summary of a completed conversion."""
+
+    pages: int
+    fmt: str
+    layout: str
+    output_dir: Path
+    outputs: List[Path] = field(default_factory=list)
+
+
+def convert(
+    dump_path,
+    output_dir,
+    *,
+    base_url: Optional[str] = None,
+    fmt: str = "md",
+    layout: str = "combined",
+) -> ConvertResult:
+    """Convert a MediaWiki dump to Markdown, JSON, or XML.
+
+    Args:
+        dump_path: Path to the dump -- a ``.7z`` archive, or an ``.xml`` /
+            ``.xml.bz2`` file. ``.7z`` archives are extracted to a temporary
+            directory that is removed when conversion finishes.
+        output_dir: Directory to write into (created if missing).
+        base_url: Optional wiki URL prefix, e.g.
+            ``"https://gameofthrones.fandom.com/wiki/"``. When given, each page
+            gets a ``url`` built by appending its underscored title. When
+            ``None``, no url is emitted.
+        fmt: Output format -- one of ``"md"``, ``"json"``, ``"xml"``.
+        layout: ``"combined"`` (one ``corpus.<fmt>`` file, the default) or
+            ``"per-file"`` (one file per article).
+
+    Returns:
+        A :class:`ConvertResult` with the page count and the path(s) written.
+    """
+    fmt = fmt.lower()
+    if fmt not in FORMATS:
+        raise ValueError(f"fmt must be one of {FORMATS}, got {fmt!r}")
+    if layout not in LAYOUTS:
+        raise ValueError(f"layout must be one of {LAYOUTS}, got {layout!r}")
+
+    if base_url and not base_url.endswith("/"):
+        base_url += "/"
+
+    output_dir = Path(output_dir)
+
+    with resolve_dump(dump_path) as dump_file:
+        pages = iter_pages(dump_file, base_url=base_url)
+        count, outputs = write(pages, output_dir, fmt, layout)
+
+    logging.getLogger("mwextract").info(
+        "Done: %d pages -> %s", count, ", ".join(str(o) for o in outputs)
+    )
+    return ConvertResult(
+        pages=count,
+        fmt=fmt,
+        layout=layout,
+        output_dir=output_dir,
+        outputs=outputs,
+    )

mwextract-0.1.0/src/mwextract/archive.py

@@ -0,0 +1,74 @@
+"""Resolve a user-supplied dump path into a readable MediaWiki XML dump.
+
+The user downloads the dump manually (Fandom/Wikia ship them as ``.7z``). This
+module takes whatever they hand us -- a ``.7z`` archive, or an already-extracted
+``.xml`` / ``.xml.bz2`` -- and yields a path the parser can stream from,
+cleaning up any temporary extraction afterwards.
+"""
+from __future__ import annotations
+
+import contextlib
+import logging
+import shutil
+import tempfile
+from pathlib import Path
+from typing import Iterator
+
+log = logging.getLogger("mwextract")
+
+# Suffixes the parser can read directly (see parser.open_dump).
+_READABLE_SUFFIXES = (".xml", ".xml.bz2", ".bz2")
+
+
+def _find_dump_file(root: Path) -> Path:
+    """Locate the dump inside an extracted archive."""
+    candidates = [
+        p
+        for p in root.rglob("*")
+        if p.is_file() and p.name.lower().endswith(_READABLE_SUFFIXES)
+    ]
+    if candidates:
+        return max(candidates, key=lambda p: p.stat().st_size)
+
+    # Fall back to the largest file (some dumps extract without a suffix).
+    all_files = [p for p in root.rglob("*") if p.is_file()]
+    if not all_files:
+        raise FileNotFoundError("archive contained no files")
+    return max(all_files, key=lambda p: p.stat().st_size)
+
+
+@contextlib.contextmanager
+def resolve_dump(dump_path) -> Iterator[Path]:
+    """Yield a path to a readable dump file.
+
+    For a ``.7z`` archive, extract into a temporary directory (removed on exit)
+    and yield the dump found inside. For ``.xml`` / ``.xml.bz2`` / ``.bz2``,
+    yield the path unchanged.
+    """
+    dump_path = Path(dump_path)
+    if not dump_path.exists():
+        raise FileNotFoundError(f"dump not found: {dump_path}")
+
+    name = dump_path.name.lower()
+
+    if name.endswith(".7z"):
+        import py7zr
+
+        tmp = Path(tempfile.mkdtemp(prefix="mwextract_"))
+        try:
+            log.info("Extracting %s ...", dump_path.name)
+            with py7zr.SevenZipFile(dump_path, mode="r") as archive:
+                archive.extractall(path=tmp)
+            found = _find_dump_file(tmp)
+            log.info("Extracted dump: %s", found.name)
+            yield found
+        finally:
+            shutil.rmtree(tmp, ignore_errors=True)
+
+    elif name.endswith(_READABLE_SUFFIXES):
+        yield dump_path
+
+    else:
+        raise ValueError(
+            f"unsupported input {dump_path.name!r}: expected .7z, .xml, or .xml.bz2"
+        )

mwextract-0.1.0/src/mwextract/cli.py

@@ -0,0 +1,104 @@
+"""Command-line interface for mwextract.
+
+    mwextract DUMP OUTPUT_DIR [-b URL] [-f {md,json,xml}] [-l {combined,per-file}]
+"""
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+from typing import Optional, Sequence
+
+from . import FORMATS, LAYOUTS, __version__, convert
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="mwextract",
+        description=(
+            "Convert a MediaWiki XML dump (.7z / .xml / .xml.bz2) into "
+            "structured Markdown, JSON, or XML."
+        ),
+    )
+    parser.add_argument(
+        "dump",
+        help="Path to the dump: a .7z archive, or an .xml / .xml.bz2 file.",
+    )
+    parser.add_argument(
+        "output_dir",
+        help="Directory to write output into (created if missing).",
+    )
+    parser.add_argument(
+        "-b",
+        "--base-url",
+        default=None,
+        help=(
+            "Optional wiki URL prefix, e.g. "
+            "https://gameofthrones.fandom.com/wiki/ . When set, each page gets "
+            "a url built from its title."
+        ),
+    )
+    parser.add_argument(
+        "-f",
+        "--format",
+        dest="fmt",
+        choices=FORMATS,
+        default="md",
+        help="Output format (default: md).",
+    )
+    parser.add_argument(
+        "-l",
+        "--layout",
+        choices=LAYOUTS,
+        default="combined",
+        help=(
+            "combined = one corpus.<fmt> file (default); "
+            "per-file = one file per article."
+        ),
+    )
+    parser.add_argument(
+        "-q",
+        "--quiet",
+        action="store_true",
+        help="Only report warnings and errors.",
+    )
+    parser.add_argument(
+        "-V",
+        "--version",
+        action="version",
+        version=f"mwextract {__version__}",
+    )
+    return parser
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+    args = build_parser().parse_args(argv)
+
+    logging.basicConfig(
+        level=logging.WARNING if args.quiet else logging.INFO,
+        format="%(message)s",
+        stream=sys.stderr,
+    )
+
+    try:
+        result = convert(
+            args.dump,
+            args.output_dir,
+            base_url=args.base_url,
+            fmt=args.fmt,
+            layout=args.layout,
+        )
+    except (FileNotFoundError, ValueError) as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 1
+
+    print(
+        f"{result.pages} pages -> {', '.join(str(o) for o in result.outputs)} "
+        f"({result.fmt}, {result.layout})",
+        file=sys.stderr,
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

mwextract-0.1.0/src/mwextract/parser.py

@@ -0,0 +1,177 @@
+"""Parse a MediaWiki XML dump into structured page documents.
+
+Uses only the Wikimedia stack:
+
+    mwxml            -> stream <page>/<revision> out of the dump (memory-safe,
+                        namespace-aware)
+    mwparserfromhell -> parse wikitext into a node tree; pull infobox/template
+                        fields into metadata, render the body to clean Markdown
+
+The rendering logic here is a faithful refactor of the original
+``ingest_mwparser.py`` script, restructured so the writers can serialize the
+same :class:`PageDoc` to Markdown, JSON, or XML.
+"""
+from __future__ import annotations
+
+import bz2
+import logging
+import re
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Iterator, Optional
+
+import mwparserfromhell as mw
+import mwxml
+from mwparserfromhell.nodes import (
+    ExternalLink,
+    Heading,
+    HTMLEntity,
+    Tag,
+    Text,
+    Wikilink,
+)
+
+log = logging.getLogger("mwextract")
+
+# Wikilink namespaces we never want as prose.
+SKIP_LINK_PREFIXES = ("file:", "image:", "category:", "media:")
+
+
+@dataclass
+class PageDoc:
+    """One article, parsed into structured pieces.
+
+    ``body`` is the rendered Markdown body only -- it does *not* include the
+    YAML front-matter or the ``# Title`` heading. Each writer serializes these
+    fields into its own format.
+    """
+
+    title: str
+    body: str
+    metadata: dict = field(default_factory=dict)
+    url: Optional[str] = None
+
+
+# --------------------------------------------------------------------------- #
+# 1. Render a sequence of wikitext nodes -> Markdown string                    #
+# --------------------------------------------------------------------------- #
+def render(nodes) -> str:
+    out = []
+    for node in nodes:
+        if isinstance(node, Text):
+            out.append(str(node))
+
+        elif isinstance(node, Heading):
+            title = render(node.title.nodes).strip()
+            out.append(f"\n\n{'#' * node.level} {title}\n\n")
+
+        elif isinstance(node, Wikilink):
+            target = str(node.title).strip()
+            if target.lower().startswith(SKIP_LINK_PREFIXES):
+                continue  # drop media/category links entirely
+            # keep only the visible text, not the link itself
+            out.append(render(node.text.nodes) if node.text else target)
+
+        elif isinstance(node, ExternalLink):
+            # keep only the visible text; drop the URL entirely
+            if node.title:
+                out.append(render(node.title.nodes))
+
+        elif isinstance(node, Tag):
+            tag = str(node.tag)
+            content = render(node.contents.nodes) if node.contents else ""
+            if tag == "b":
+                out.append(f"**{content}**")
+            elif tag == "i":
+                out.append(f"*{content}*")
+            elif tag == "li":
+                marker = "1. " if node.wiki_markup == "#" else "- "
+                out.append(marker)
+            elif tag in ("dt", "dd"):
+                out.append("")  # definition lists -> flatten to text
+            elif tag in ("ref", "references"):
+                continue  # drop citation footnotes
+            else:
+                out.append(content)  # br, span, small, etc. -> keep inner text
+
+        elif isinstance(node, HTMLEntity):
+            out.append(node.normalize())
+
+        # Template, Comment, Argument -> intentionally dropped here
+    return "".join(out)
+
+
+# --------------------------------------------------------------------------- #
+# 2. Tidy whitespace / list spacing in the rendered Markdown                   #
+# --------------------------------------------------------------------------- #
+def tidy(md: str) -> str:
+    md = re.sub(r"[ \t]+\n", "\n", md)  # trailing spaces
+    md = re.sub(r"^(\s*(?:- |\d+\. ))[ \t]+", r"\1", md, flags=re.M)  # marker spacing
+    md = re.sub(r"\n{3,}", "\n\n", md)  # collapse blank lines
+    return md.strip()
+
+
+# --------------------------------------------------------------------------- #
+# 3. Pull infobox / template fields into a metadata dict                       #
+# --------------------------------------------------------------------------- #
+def extract_metadata(code) -> dict:
+    meta: dict = {}
+    for tmpl in code.filter_templates():
+        if "infobox" in str(tmpl.name).strip().lower():
+            for param in tmpl.params:
+                key = str(param.name).strip()
+                val = mw.parse(str(param.value)).strip_code().strip()
+                val = re.sub(r"\s+", " ", val)
+                if key and val and not key.isdigit():
+                    meta[key] = val
+    return meta
+
+
+def strip_templates(code):
+    for tmpl in list(code.filter_templates()):
+        try:
+            code.remove(tmpl)
+        except ValueError:
+            pass
+    return code
+
+
+# --------------------------------------------------------------------------- #
+# 4. Parse one page into a structured PageDoc                                  #
+# --------------------------------------------------------------------------- #
+def parse_page(title: str, wikitext: str, base_url: Optional[str] = None) -> PageDoc:
+    code = mw.parse(wikitext)
+    meta = extract_metadata(code)
+    strip_templates(code)
+    body = tidy(render(code.nodes))
+
+    url = base_url + title.replace(" ", "_") if base_url else None
+    return PageDoc(title=title, body=body, metadata=meta, url=url)
+
+
+# --------------------------------------------------------------------------- #
+# 5. Stream the dump -> PageDoc per article                                    #
+# --------------------------------------------------------------------------- #
+def open_dump(path: str):
+    """Open an .xml or .xml.bz2/.bz2 dump as a UTF-8 text stream."""
+    if path.endswith(".bz2"):
+        import io
+
+        return io.TextIOWrapper(bz2.open(path, "rb"), encoding="utf-8")
+    return open(path, encoding="utf-8")
+
+
+def iter_pages(dump_path, base_url: Optional[str] = None) -> Iterator[PageDoc]:
+    """Yield one :class:`PageDoc` per main-namespace, non-redirect article."""
+    dump = mwxml.Dump.from_file(open_dump(str(dump_path)))
+    for page in dump:
+        if page.namespace != 0 or page.redirect:  # main namespace, no redirects
+            continue
+        wikitext = next((rev.text for rev in page if rev.text), None)
+        if not wikitext:
+            continue
+        try:
+            yield parse_page(page.title, wikitext, base_url)
+        except Exception as exc:  # noqa: BLE001 - one bad page shouldn't stop the run
+            log.warning("skip %r: %s", page.title, exc)
+            continue

mwextract-0.1.0/src/mwextract/writers.py

@@ -0,0 +1,171 @@
+"""Serialize a stream of :class:`~mwextract.parser.PageDoc` to disk.
+
+Every format supports two layouts:
+
+    combined  -> one file for the whole corpus (``corpus.md`` / ``corpus.json``
+                 / ``corpus.xml``). This is the default.
+    per-file  -> one file per article (``Title.md`` / ``Title.json`` /
+                 ``Title.xml``), mirroring the original ingest script.
+
+Both layouts are written incrementally as pages stream in, so memory use stays
+flat regardless of dump size.
+"""
+from __future__ import annotations
+
+import json
+import logging
+import re
+import xml.etree.ElementTree as ET
+from pathlib import Path
+from typing import Iterable, List, Tuple
+
+from .parser import PageDoc
+
+log = logging.getLogger("mwextract")
+
+# Infobox fields promoted into the human-readable Markdown summary line.
+SUMMARY_KEYS = ("house", "allegiance", "status", "culture", "born", "died")
+
+_PROGRESS_EVERY = 200
+
+
+def safe_filename(title: str) -> str:
+    name = re.sub(r"[^\w\-]+", "_", title).strip("_")[:120]
+    return name or "untitled"
+
+
+# --------------------------------------------------------------------------- #
+# Per-format serialization of a single PageDoc                                 #
+# --------------------------------------------------------------------------- #
+def render_markdown_doc(doc: PageDoc) -> str:
+    """Full Markdown document: YAML front-matter + title + summary + body."""
+    fm: List[str] = ["---", f'title: "{doc.title}"']
+    if doc.url:
+        fm.append(f"url: {doc.url}")
+    for key, val in doc.metadata.items():
+        safe = val.replace('"', "'")
+        fm.append(f'{key}: "{safe}"')
+    fm.append("---")
+
+    summary = ""
+    keys = [k for k in SUMMARY_KEYS if k in doc.metadata]
+    if keys:
+        summary = (
+            "> "
+            + " · ".join(f"**{k.title()}:** {doc.metadata[k]}" for k in keys)
+            + "\n\n"
+        )
+
+    return "\n".join(fm) + f"\n\n# {doc.title}\n\n{summary}{doc.body}\n"
+
+
+def _json_obj(doc: PageDoc) -> dict:
+    obj: dict = {"title": doc.title}
+    if doc.url:
+        obj["url"] = doc.url
+    obj["metadata"] = doc.metadata
+    obj["body"] = doc.body
+    return obj
+
+
+def _xml_elem(doc: PageDoc) -> ET.Element:
+    page = ET.Element("page")
+    ET.SubElement(page, "title").text = doc.title
+    if doc.url:
+        ET.SubElement(page, "url").text = doc.url
+    metadata = ET.SubElement(page, "metadata")
+    for key, val in doc.metadata.items():
+        field_el = ET.SubElement(metadata, "field", {"name": key})
+        field_el.text = val
+    ET.SubElement(page, "body").text = doc.body
+    return page
+
+
+# --------------------------------------------------------------------------- #
+# Combined layout: one file for the whole corpus                               #
+# --------------------------------------------------------------------------- #
+def _write_combined(
+    pages: Iterable[PageDoc], out_dir: Path, fmt: str
+) -> Tuple[int, List[Path]]:
+    out_path = out_dir / f"corpus.{fmt}"
+    count = 0
+
+    with open(out_path, "w", encoding="utf-8") as fh:
+        if fmt == "md":
+            for count, doc in _enumerate(pages):
+                if count > 1:
+                    fh.write("\n\n\n")
+                fh.write(render_markdown_doc(doc))
+
+        elif fmt == "json":
+            fh.write("[\n")
+            for count, doc in _enumerate(pages):
+                if count > 1:
+                    fh.write(",\n")
+                fh.write(json.dumps(_json_obj(doc), ensure_ascii=False))
+            fh.write("\n]\n")
+
+        elif fmt == "xml":
+            fh.write('<?xml version="1.0" encoding="utf-8"?>\n<pages>\n')
+            for count, doc in _enumerate(pages):
+                fh.write(ET.tostring(_xml_elem(doc), encoding="unicode"))
+                fh.write("\n")
+            fh.write("</pages>\n")
+
+    return count, [out_path]
+
+
+# --------------------------------------------------------------------------- #
+# Per-file layout: one file per article                                        #
+# --------------------------------------------------------------------------- #
+def _write_per_file(
+    pages: Iterable[PageDoc], out_dir: Path, fmt: str
+) -> Tuple[int, List[Path]]:
+    seen: dict[str, int] = {}
+    count = 0
+
+    for count, doc in _enumerate(pages):
+        stem = safe_filename(doc.title)
+        # Disambiguate distinct titles that collapse to the same filename.
+        if stem in seen:
+            seen[stem] += 1
+            stem = f"{stem}_{seen[stem]}"
+        else:
+            seen[stem] = 1
+
+        out_path = out_dir / f"{stem}.{fmt}"
+        with open(out_path, "w", encoding="utf-8") as fh:
+            if fmt == "md":
+                fh.write(render_markdown_doc(doc))
+            elif fmt == "json":
+                fh.write(json.dumps(_json_obj(doc), ensure_ascii=False, indent=2))
+                fh.write("\n")
+            elif fmt == "xml":
+                elem = _xml_elem(doc)
+                ET.indent(elem)
+                fh.write('<?xml version="1.0" encoding="utf-8"?>\n')
+                fh.write(ET.tostring(elem, encoding="unicode"))
+                fh.write("\n")
+
+    return count, [out_dir]
+
+
+def _enumerate(pages: Iterable[PageDoc]):
+    """Enumerate pages from 1, logging progress every _PROGRESS_EVERY pages."""
+    for i, doc in enumerate(pages, start=1):
+        if i % _PROGRESS_EVERY == 0:
+            log.info("  ... %d pages", i)
+        yield i, doc
+
+
+# --------------------------------------------------------------------------- #
+# Public entry point                                                           #
+# --------------------------------------------------------------------------- #
+def write(
+    pages: Iterable[PageDoc], out_dir, fmt: str, layout: str
+) -> Tuple[int, List[Path]]:
+    out_dir = Path(out_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    if layout == "combined":
+        return _write_combined(pages, out_dir, fmt)
+    return _write_per_file(pages, out_dir, fmt)