wikiq 0.3.0__tar.gz

wikiq-0.3.0/.gitignore

@@ -0,0 +1,17 @@
+*.xml.gz
+*.7z
+*.xml.bz2
+*.xml.xz
+*.swp
+
+# JetBrains
+/.idea
+
+/.claude/settings.local.json*
+
+# Python build and test output
+__pycache__/
+/test/test_output/
+
+# Virtualenv (created with create_cdsc_venv on klone)
+/.venv/

wikiq-0.3.0/CHANGELOG.md

@@ -0,0 +1,73 @@
+# Changelog
+
+## 0.3.0 (2026-11-01)
+
+Changes relative to what people have been running. Earlier version tags
+in this repository were internal markers rather than releases, so there
+is no previous release to compare against.
+
+### Incompatible changes
+
+- **Redirect detection is now per revision.** The page-level
+  `redirect_target` column has been replaced by `revision_is_redirect`
+  and `revision_redirect_target`, which classify each revision from its
+  own text. The old column reported a page's redirect status as of the
+  time the dump was exported and applied that single value to every
+  revision in the page's history, which is wrong for any page that was
+  turned into a redirect, or turned back, at some point in its life.
+  Revisions with deleted text report null in both columns rather than
+  false. Wikis using a localized redirect keyword can add it with
+  `--redirect-aliases`; `#REDIRECT` is always recognized.
+
+  Code reading `redirect_target` needs updating.
+
+- **Parquet output has been removed**, along with the
+  `--partition-namespaces` and `--max-revisions-per-file` options that
+  only applied to it. Use JSONL, which supports `--resume`.
+
+- **pyspark is no longer a dependency** and the `wikiq-spark` entry point
+  is gone. The second pass of that pipeline was never in this
+  repository.
+
+- **Python 3.11 or later is required.** Python 3.9 reached end of life in
+  October 2025, and the floor unlocks current pyarrow, more-itertools and
+  pandas, along with the CPython 3.11 speedups on the CPU-bound work
+  wikiq does.
+
+- **The project is now called wikiq**, rather than mediawiki_dump_tools.
+
+### Installation
+
+- **wikiq installs with plain `pip` and needs no C++ compiler.**
+  Previously every install required pywikidiff2, built from an ssh-only
+  git URL, so anyone outside the collective could not install the tool at
+  all. It is now optional and needed only for `--diff` and
+  `-p wikidiff2`.
+
+- **`-p legacy` needs the new `legacy` extra.** It was the only user of
+  `mediawiki-utilities`, a 2015 package that also pulls in a MySQL client
+  and six other packages. Everyone was paying for that to support one
+  flag.
+
+- **Dependencies resolve from PyPI** rather than from pinned forks under
+  personal accounts. Requires mwxml >= 0.3.8, which fixes eleven upstream
+  issues; note that mwxml 0.3.7 changed namespace handling to trust the
+  dump's embedded `<ns>` tag.
+
+### Added
+
+- `--version`, and the version is printed to stderr on every run so it
+  lands in job logs beside the output.
+- Licensing: wikiq is GPL-3.0-or-later. See `COPYING`.
+
+### Fixed
+
+- Regex matching (`-RP`/`-CP`) crashed with a `TypeError` on revisions
+  whose text or comment was deleted or suppressed. Deleted content now
+  reports null.
+- Page titles in non-main namespaces were prefixed twice under mwxml
+  0.3.7 and later, producing titles like `Category:Category:Alaska`.
+- The test suite had 16 failing tests, and `pytest test/` silently
+  collected only half of it because the main test file matched neither of
+  pytest's discovery patterns. It is now `test_wikiq.py` and the suite
+  runs green.