wp2txt 2.3.1 → 2.3.3

data/CHANGELOG.md

@@ -5,321 +5,132 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+Entries were rewritten in August 2026 to describe what changed for people using
+wp2txt, rather than how it was implemented. The changes themselves are unaltered.
+
+## [2.3.3] - 2026-09-08
+
+- **Short searches no longer report a false zero**: in Japanese, Chinese, and Korean indexes a phrase of one or two characters could match nothing and come back as `0 matches`, which reads exactly like a term that is genuinely absent from the dump. Such a search now fails with an explicit error instead. If you recorded a zero result for a short term with an earlier version, re-check it
+- **Characters no longer vanish from extracted text**: a multi-byte character that happened to straddle an internal read boundary was dropped, silently and at unpredictable positions. Any non-ASCII text could be affected. Re-extract if you need the output to be exact
+- **Articles whose title contains a colon are no longer skipped**: `Star Trek: Voyager` and titles like it were mistaken for administrative pages and left out of both extraction and indexes. Rebuild with `--build-index -U` and re-extract to pick them up
+- **`query_sql` no longer loses values when column names repeat**: a query selecting two columns under the same name kept only one of them. Names are now made unique, and the `.meta.json` beside a file result records which output name corresponds to which column of your query
+- **Cell clipping counts bytes, not characters**: a long cell was measured in characters, so it could exceed the limit and still be reported as unclipped — 30,000 Japanese characters are 90,000 bytes. The limit is now 65,536 bytes, cut at a character boundary, and the count of clipped cells is accurate
+- **A result of exactly the requested size is no longer marked truncated**: `truncated` came back true when a result happened to fill the limit exactly, with nothing left behind
+- **Output files stay inside the output directory**: a symbolic link within it could redirect a write outside, the `.meta.json` sidecar was not checked at all, and an existing file could slip past `overwrite: false`. Output and sidecar are now reserved together and written to a temporary file first, so a failed or cancelled run leaves your previous output untouched instead of a half-written file in its place
+- **Index builds no longer grow in memory with the size of the dump**: every batch of extracted text was kept until the build finished
+- **A background job that fails to start is now reported as failed**: it stayed `running` indefinitely and blocked every later job until the server was restarted
+- **The gem no longer ships maintainer scripts or images**. If you were running `scripts/fetch_*.rb` from an installed gem, take them from the repository instead
+- **The Docker Hub repository has been removed**: images are published to GitHub Container Registry only. If you still pull from Docker Hub, switch with `docker pull ghcr.io/yohasebe/wp2txt`
+
+## [2.3.2] - 2026-08-13
+
+- **Documentation release — no code changes.** The changelog and guides bundled with the gem and the container image are rewritten to say what each change means for someone using wp2txt, rather than how it was implemented. The guide formerly at `docs/RESEARCH.md` is now [docs/INDEXES.md](docs/INDEXES.md) ("Offline Indexes, Queries, and the MCP Server")
+
 ## [2.3.1] - 2026-08-12
 
-- **Container images no longer carry private files**: the image is built from the working tree with `git init && git add -A`, but the build context excluded `.gitignore`, so files ignored locally were tracked inside the container and shipped in the image — 2.3.0's images contained the maintainer's private research notes under `/wp2txt/research-notes`. The context now keeps `.gitignore` (so the container's file list matches a local `gem build`), explicitly excludes `research-notes` and `.bundle`, and the throwaway `.git` (a blob copy of every added file) is deleted after the build. `rake check_image` builds and inspects an image for private paths, and `rake push` refuses to publish without that check. The published gems were never affected; 2.3.0 images have been withdrawn
-- **Docker Hub retired**: images are published to GitHub Container Registry only (`ghcr.io/yohasebe/wp2txt`). The Docker Hub repository is no longer updated
-- **Fix: multi-line `<ref>` references broken by element splitting — also fixes `--extract-citations` on multi-line cite templates**: `Article#parse` splits elements at newlines, so any reference written across lines (very common for multi-line `{{cite …}}` templates) had its `[ref]` and `[/ref]` markers land in separate elements, invisible to `remove_ref`. The markers and raw reference markup leaked into the output, and — more importantly — `--extract-citations` silently never fired for multi-line cite templates, producing inconsistent output versus their single-line equivalents. `make_reference` now drops empty references outright and flattens the rest onto a single line before element splitting, so multi-line references behave exactly like single-line ones. Ordinary paragraph breaks outside references are unaffected
-- **Fix: `[ref]` markers destroyed by external-link processing**: `process_external_links` stripped the brackets of the `[ref]`/`[/ref]` markers produced by `make_reference` (their contents took the single-word branch), so `remove_ref` could no longer find them and tag names plus reference bodies leaked into extracted text — even with `--ref`, the kept markers came out broken. The markers are now hidden behind placeholders for the duration of the bracket scan and restored afterwards. Downstream corpora no longer contain `ref…/ref` residue, which broke tokenization/sentence splitting and mixed bibliographic text into body prose
-- **Runaway-query hardening**: a `query_sql` child process now sets its own kernel-enforced CPU limit (`RLIMIT_CPU`, the query timeout plus a small grace) in addition to the parent's wall-clock kill. Previously a child orphaned by the parent's death — an interrupted test run, a closed terminal, a crashed server — kept executing forever: sqlite3 holds the GVL inside `sqlite3_step`, so Ruby never reaches a signal-safe point and even SIGTERM is ignored. Observed in the wild as two processes spinning at 99% CPU for over five days. The CPU limit is strictly more permissive than the existing wall-clock deadline, so no query that would otherwise succeed is affected
-- **Test-suite hang guard**: each example now runs under a wall-clock timeout (120s default; `WP2TXT_SPEC_TIMEOUT=0` disables it, the `:no_timeout` tag exempts an example). The suite deliberately exercises runaway queries, so a wedged example must fail rather than spin
+- **Fix: references leaked into extracted text**: reference contents — tag remnants like `ref…/ref` and the bibliographic text inside them — could end up in the extracted body text, breaking sentence splitting and mixing citation details into running prose. In a spot check of five English articles this affected about 70 places per article. References are now removed cleanly (or kept intact as `[ref]…[/ref]` with `--ref`), including references written across several lines and empty ones
+- **Fix: `--extract-citations` ignored multi-line citation templates**: a `{{cite …}}` template split over several lines was left as raw markup instead of being formatted, so citations extracted from such references were silently missing. Single-line and multi-line templates now produce the same result
+- **Long-running queries can no longer outlive their timeout**: an interrupted session could leave a `query_sql` process running at full CPU indefinitely. Queries now stop on their own even if the process that started them goes away. Queries that complete within the time limit are unaffected
+- **Container images move to GitHub Container Registry**: images are published to `ghcr.io/yohasebe/wp2txt` only. The Docker Hub repository is no longer updated — please switch:
 
-## [2.3.0] - 2026-07-24
+      docker pull ghcr.io/yohasebe/wp2txt
 
-- **Documentation split**: README now focuses on text extraction; the research layer (indexes, exhaustive queries, full-text search, langlinks, cross-language SQL, MCP server) is documented in the new [Research Infrastructure Guide](docs/RESEARCH.md), including the complete MCP tool table
-- **Container images on GHCR**: images are now published to `ghcr.io/yohasebe/wp2txt` (Docker Hub `yohasebe/wp2txt` is maintained as a mirror)
+## [2.3.0] - 2026-07-24
 
-- **`extract_corpus` `titles:` argument**: Extract an explicit set of article titles (e.g. a set determined via `query_sql`) — titles are normalized (MediaWiki rules), deduplicated preserving input order, and one redirect hop is resolved; missing titles (including redirects to nowhere) are skipped and reported as `not_found` (count + 20-title sample, also in the `.meta.json` sidecar). Mutually exclusive with the filter arguments (set operations belong in SQL); capped at 10,000 titles. The sidecar records `titles_count` + `titles_sha256` (order-independent) for reproducibility, enumerating the full list when ≤100 titles. Also available via `start_extract_job`
-- **`query_sql` `output_path:` argument**: Write ALL rows of a large result to a JSONL file (with a `.meta.json` sidecar recording the SQL, dump version, attach configuration, row count, and tool version) and return only a summary + 3-row sample — the extract_corpus D4 pattern generalized to SQL. Writes happen in the forked child (so the 30s SIGKILL deadline covers them) to a `.partial` file that the parent atomically renames on success and removes on every failure path (child crash, timeout kill, query error). Hard cap `SQL_FILE_ROW_LIMIT` = 5M rows (`truncated` flag), cells clipped at 64KB (`cells_clipped` count), duplicate column names are made unique (`_2` suffix), `limit` is ignored in this mode, and existing files require `overwrite: true`. The MCP layer's output-dir confinement is now a shared helper (`Wp2txt::OutputPath.confine`) used by extract_corpus, start_extract_job, and query_sql alike
-- **Interlanguage links (`--import-langlinks`)**: Import the official `{lang}wiki-{date}-langlinks.sql.gz` dump into the Tier 1 metadata index as a `langlinks` table (`ll_from` = source page_id, `ll_lang` = target language, `ll_title` = normalized target title). Version pinning is enforced: the langlinks file's dump name must equal the built index's dump version — a mismatch is rejected with no override. Streams the MySQL dump without loading it whole (escape-safe tuple parser, 10k-row transaction batches, indexes created after the load). `--langlinks-langs` restricts imported target languages (e.g. `en,de,fr,zh,ko`); re-import requires `-U` (otherwise a no-op reporting the previous import time). Provenance (source file, size, import time, wp2txt version, language filter, row count) is stamped into the index and reported by `dump_info`; a post-import sanity check samples titles per language and warns when the join rate against a locally installed target edition falls below 90%
-- **Multi-dump ATTACH in `query_sql`**: The MCP `query_sql` tool gains an optional `attach` argument (language codes only, never paths): `attach: ["en"]` read-only ATTACHes that language's locally installed metadata DB as `en_meta` and its FTS DB (when built) as `en_fts`, both sharing the main DB's schema. Codes are validated, only installed indexes are accepted, path resolution is server-side (same-date dump preferred; otherwise the latest build with a `dump_mismatch` note in the response's `attached` metadata). User SQL still cannot contain ATTACH/DETACH — attachments are issued by server code only, via `mode=ro` URIs on a read-only connection. Combined with the langlinks table, this enables single-query cross-language comparisons (e.g. section-structure diffs of article pairs)
+- **Extract an explicit list of articles**: `extract_corpus` accepts `titles:` — hand it the set you arrived at some other way (a `query_sql` result, your own list) instead of describing it with filters again. Titles are normalized the way MediaWiki treats them, duplicates are dropped, and one redirect hop is followed. Titles with no article are skipped and reported back as `not_found`, so a missing article is visible rather than silently absent. Up to 10,000 titles; cannot be combined with the filter arguments. Also available through `start_extract_job`
+- **Write a large SQL result to a file**: `query_sql` accepts `output_path:` — all rows go to a JSONL file and the reply carries a summary plus three sample rows, instead of a row cap. The file is written completely or not at all: an interrupted or failing query leaves no partial output. Up to 5 million rows, cells longer than 64KB are clipped (both are reported back), and an existing file is only replaced with `overwrite: true`. A `.meta.json` beside the output records the SQL, the dump versions involved, and the row count
+- **Interlanguage links (`--import-langlinks`)**: import the official `langlinks` dump into the metadata index to map articles across editions. The langlinks file must carry the same dump date as your index — a mismatch is refused, so links can never be mixed across versions. `--langlinks-langs en,de,fr,zh,ko` limits which target languages are imported; re-importing requires `-U`. After the import, a sample of titles per language is checked against any editions you have installed and the match rate is reported. `dump_info` shows what was imported and when
+- **Query several editions in one statement**: `query_sql` accepts `attach: ["en"]`, which makes that language's index available as `en_meta` (and `en_fts` where a full-text index exists) alongside your own. You pass language codes, not paths, and only editions you have installed are accepted. The reply lists what was attached, including each edition's dump name, and says so when the dates differ. Together with the `langlinks` table this makes cross-edition comparisons a single query
+- **Documentation split**: the README now covers text extraction; the index and MCP features (indexes, offline queries, full-text search, interlanguage links, cross-language SQL, MCP server) moved to a separate guide — now [docs/INDEXES.md](docs/INDEXES.md) — which lists every MCP tool
+- **Container images on GitHub Container Registry**: images are now published to `ghcr.io/yohasebe/wp2txt`
 
 ## [2.2.0] - 2026-07-22
 
-- **Index hardening (design review follow-up)**: `ord` now has identical semantics in `page_sections` and `fts_map` (lead = 0, first heading = 1; schema v2 — rebuild indexes with `--build-index -U`); section headings are normalized identically in both indexes (decorated headings like `== '''X''' ==` now match section filters); indexes record the wp2txt version (and the FTS index a rendering-config digest) so `dump_info` can flag indexes built by code whose text cleaning differs; rebuilds are atomic (built alongside, renamed on completion — a failed rebuild no longer destroys the working index)
-- **MCP safety hardening**: `query_sql` runs in a killable subprocess with a 30s wall-clock cap and returns a query-plan diagnosis on timeout; the keyword screen no longer rejects legitimate values inside string literals; `extract_corpus` output paths are confined to the server output directory and refuse to overwrite existing files without `overwrite: true`; background jobs are serialized (one at a time) and all SQLite connections are closed before forking extraction workers
-- **Cold-start parity for LLM clients**: closes the gaps that previously required internal knowledge to work around. `get_categories` tool; `find_articles`/`extract_corpus` accept `categories` (exact multi-category AND) and `category_match` (substring on category names); `extract_corpus` gains `content: "wikitext"` for structure mining (infoboxes/templates); `query_sql` read-only SQL escape hatch (SELECT/WITH only, keyword-screened, read-only connection, row/cell caps) with `describe_schema`; article titles are normalized like MediaWiki (underscores, capitalization); `get_article` truncates at `max_chars` (default 40k) with an explicit flag; `dump_info` reports `fulltext_current`
-- **Deferred FTS optimize**: `--skip-fts-optimize` skips the single-threaded segment-merge step of the full-text build (the dominant cost on many-core machines: 40-60+ min on a full dump), leaving a fully searchable index; `--fts-optimize` runs the merge later, standalone and idempotent. Optimize state is recorded in index metadata and reported by `dump_info`
-- **Full-text search (Tier 2)**: `--build-index --fulltext` builds a contentless SQLite FTS5 index over cleaned section text (tokenizer auto-selected by language: character-trigram for CJK, unicode61 for space-delimited; `--fts-tokenizer` to override, porter stemming opt-in). Search via `--search` (CLI) or the `search_text` MCP tool: literal phrase or raw FTS5 query modes, composable with category recursion and section filters, `count: "capped"|"exact"` (exact 0 = a verified absence claim for the dump version). Snippets are re-rendered from the dump on demand — the index stores no text, keeping disk cost to the inverted index only
-- **Background extraction jobs**: `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` MCP tools for extractions beyond the 5000-article synchronous cap. Each job runs in its own thread with isolated resources; extraction now streams to disk in batches (memory-safe at any scale)
-- **RAG chunking**: `extract_corpus` accepts `chunk_size` / `chunk_overlap`, emitting one record per chunk with `section_path`, `chunk_index`, and `chunk_count`; chunk boundaries prefer sentence/paragraph breaks
-- **Alias guardrail**: `save_alias_set` re-checks co-occurrence server-side and refuses groups containing frequently-coexisting heading pairs (likely distinct roles, not synonyms) unless `force` is passed — protocol compliance no longer depends on the calling model's discipline
-- **`discover_aliases` MCP prompt**: bundled recipe walking any agent through the alias verification protocol (discover via `section_stats` → verify via `section_cooccurrence` → sample borderline cases → save)
-- **MCP server (`wp2txt-mcp`)**: New binary exposing a local dump to LLM agents via the Model Context Protocol (stdio). Tools: `dump_info`, `get_article`, `get_sections`, `list_headings`, `find_articles`, `category_tree`, `section_stats`, `section_cooccurrence`, `save_alias_set`/`get_alias_set`/`list_alias_sets`, and `extract_corpus` (writes JSONL + reproducibility `.meta.json` sidecar; returns summary and sample only). Requires the `mcp` gem (`gem install mcp`); wp2txt itself does not depend on it
-- **LLM-generated section aliases**: Instead of shipping per-language alias dictionaries, agents discover real heading usage with `section_stats`, verify synonym hypotheses with `section_cooccurrence` (synonymous headings almost never co-occur in one article), and persist named alias sets per dump with `save_alias_set`. Queries reference them via `alias_set`; extraction metadata records the exact set contents used
-- **`Wp2txt::Corpus` facade**: Shared query/extraction layer used by the MCP server; lazy SQLite-backed title lookup avoids loading multi-million-entry indexes into memory
-- **Local metadata index (`--build-index`)**: New offline index built by scanning a multistream dump in parallel. Stores per-article categories, section headings, redirects, and the category hierarchy in SQLite (`~/.wp2txt/cache/*_meta.sqlite3`), keyed to the dump version. No API access required
-- **Offline exhaustive queries (`--find-articles`)**: List articles matching `--in-category` (recursive via `--depth`, powered by dump-derived category hierarchy), `--has-section` (alias-aware), and `--title-match` filters. Supports `--limit` and JSON output (`-j json`); redirects are excluded automatically. Enables queries like "all film articles that have a Plot section" against a version-pinned local dump
+- **Index format change — rebuild existing indexes with `--build-index -U`** (schema v2). Section headings are now recorded consistently, so headings written with decoration (`== '''X''' ==`) match section filters instead of being missed. Each index records the wp2txt version that built it, and `dump_info` warns when an index was built by a version whose text cleaning differs from the one you are running. A rebuild that fails partway no longer destroys the index you already had
+- **Safer defaults for MCP use**: `query_sql` stops after 30 seconds and explains what made the query slow; queries containing words like `Update` inside a quoted value are no longer rejected by mistake. `extract_corpus` can only write under the server's output directory and will not replace an existing file unless you pass `overwrite: true`. Background jobs run one at a time
+- **More ways to select articles from an LLM client**: a `get_categories` tool; `categories` (article must be in all of them) and `category_match` (substring on category names) on `find_articles` and `extract_corpus`; `content: "wikitext"` on `extract_corpus` for reading infoboxes and templates that cleaned text replaces with markers; `query_sql` for read-only SQL when the fixed tools cannot express a question, with `describe_schema` to see the tables. Article titles are matched the way MediaWiki treats them (underscores, capitalization). `get_article` stops at `max_chars` (default 40,000) and says when it truncated; `dump_info` reports whether the full-text index is up to date
+- **Split the full-text build in two**: `--skip-fts-optimize` leaves out the final merge step — the slowest part on a full dump, 40–60 minutes and single-threaded — and still gives you a searchable index. Run `--fts-optimize` later when convenient; it is safe to repeat. `dump_info` shows whether the merge has been done
+- **Full-text search**: `--build-index --fulltext` adds a searchable index over the cleaned article text. The tokenizer is chosen by language (character trigrams for Japanese, Chinese, Korean; word-based elsewhere, `--fts-tokenizer` to override). Search from the CLI with `--search` or from an LLM client with `search_text`, combined with category and section filters, as a literal phrase or a full-text query. Counts can be capped for speed or exact — an exact `0` means the term is absent from that dump. The index stores no article text (snippets are re-read from the dump when needed), so it costs far less disk than a copy of the corpus
+- **Background extraction jobs**: `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` for extractions beyond the 5,000-article limit of a direct call. Extraction writes to disk as it goes, so memory use no longer grows with the size of the job
+- **Chunked output for RAG**: `extract_corpus` accepts `chunk_size` / `chunk_overlap` and emits one record per chunk with its `section_path`, `chunk_index`, and `chunk_count`. Chunks prefer to break at sentence and paragraph boundaries
+- **Alias sets are checked before they are saved**: `save_alias_set` refuses a group whose headings frequently appear together in the same article — a sign they are different sections rather than synonyms — unless you pass `force`
+- **`discover_aliases` prompt**: a bundled walkthrough of finding heading variants (`section_stats` → `section_cooccurrence` → save), so you do not have to remember the steps
+- **MCP server (`wp2txt-mcp`)**: a new command that exposes a local dump to an LLM client over the Model Context Protocol. Tools cover dump identity, single articles and their sections, filtered listings, category trees, heading statistics, alias sets, and extraction to JSONL with a `.meta.json` recording the dump version and query. Requires the `mcp` gem (`gem install mcp`); wp2txt itself does not depend on it
+- **Section aliases come from the dump, not from a bundled dictionary**: find the headings actually used with `section_stats`, check whether two of them mean the same thing with `section_cooccurrence`, and save the group with `save_alias_set`. Queries then refer to it by name via `alias_set`, and extractions record which group was used
+- **Ruby API**: `Wp2txt::Corpus` provides the query and extraction layer used by the MCP server, for embedding wp2txt in your own code. Title lookup reads from SQLite on demand rather than loading a multi-million-entry index into memory
+- **Offline metadata index (`--build-index`)**: builds a local index of a dump — each article's categories, section headings, redirects, and the category hierarchy — by scanning it in parallel. Stored under `~/.wp2txt/cache/` and tied to the dump version. No API access needed
+- **Offline queries over a whole edition (`--find-articles`)**: list articles by `--in-category` (with `--depth` to include subcategories), `--has-section` (alias-aware), and `--title-match`. Supports `--limit` and JSON output (`-j json`); redirects are excluded. This is what makes questions like "all film articles that have a plot section" answerable against a pinned dump
 
 ## [2.1.2] - 2026-07-19
 
-- **Fixed gem file permissions**: The published gem contained files with owner-only (0600) permissions inherited from the build machine, making them unreadable after `sudo gem install`. A `normalize_permissions` task now runs before `rake build`, ensuring all packaged files are world-readable (0644, or 0755 for executables)
+- **Fixed unreadable files after `sudo gem install`**: files in the published gem carried owner-only permissions, so wp2txt failed to run for anyone but the installing user. Packaged files are now readable by everyone (0644, and 0755 for executables). If you hit this, reinstall the gem
 
 ## [2.1.1] - 2026-02-21
 
-- **Bidirectional alias matching**: Section extraction now supports reverse alias lookup - specifying an alias name (e.g., "Synopsis") as target matches the canonical heading ("Plot") and vice versa
-- **Expanded default section aliases**: Increased from 2 to 12 alias groups covering common English Wikipedia sections (Plot, Reception, References, Bibliography, Awards, Legacy, Early life, Career, etc.)
-- **Config forwarding fix**: `--pre`, `--ref`, `--expand-templates`, and `--metadata-only` options now correctly forwarded in `--articles` and `--from-category` modes
+- **Section aliases match in both directions**: asking for "Synopsis" now also finds sections headed "Plot", and the other way round, instead of only matching the name you happened to type
+- **More section aliases out of the box**: 12 groups instead of 2, covering common English Wikipedia sections (Plot, Reception, References, Bibliography, Awards, Legacy, Early life, Career, and others)
+- **Fixed options being ignored**: `--pre`, `--ref`, `--expand-templates`, and `--metadata-only` had no effect when used with `--articles` or `--from-category`. They now apply in those modes too
 
 ## [2.1.0] - 2026-02-19
 
-- **SQLite-based caching infrastructure**: New high-performance caching using SQLite for faster startup and repeated operations:
-  - `GlobalDataCache`: Caches parsed JSON data files (templates, MediaWiki aliases, HTML entities)
-    - Eliminates ~500KB JSON parsing overhead on each startup
-    - Validates cache against source file modification time and size
-    - Location: `~/.wp2txt/cache/global_data.sqlite3`
-  - `CategoryCache`: Caches Wikipedia category hierarchy from API
-    - Stores category members (pages and subcategories) in SQLite tables
-    - Supports recursive tree traversal and bulk page retrieval
-    - Per-language cache files: `~/.wp2txt/cache/categories_en.sqlite3`
-    - Configurable expiry (default: 7 days)
-  - `IndexCache`: Caches parsed multistream index (already existed, now with SQLite3 2.x compatibility)
-    - Reduces index parsing from ~10 minutes to seconds on subsequent runs
-  - All caches use WAL mode for concurrent read access during parallel processing
-
-- **Ractor parallel processing (Ruby 4.0+)**: New `--ractor` option for thread-based parallelism:
-  - Requires Ruby 4.0 or later for stable operation
-  - Uses map-join-value pattern for reliable Ractor orchestration
-  - ~2x speedup compared to sequential processing
-  - Lower memory footprint than process-based parallelism (Parallel gem)
-  - Automatic fallback to sequential processing on Ruby 3.x
-  - Performance: Parallel gem (~3x) remains faster, Ractor (~2x) uses less memory
-
-- **Template expansion**: New `--expand-templates` (`-E`) option expands common templates to readable text:
-  - Date templates: `{{birth date|1990|5|15}}` → "May 15, 1990"
-  - Convert templates: `{{convert|100|km|mi}}` → "100 km (62 mi)"
-  - Coordinate templates: `{{coord|35|41|N|139|41|E}}` → "35°41′N 139°41′E"
-  - Language templates: `{{lang|ja|日本語}}` → "日本語"
-  - Quote templates: `{{blockquote|text}}` → "text"
-  - And 20+ more template types
-  - **Enabled by default** - use `--no-expand-templates` to disable
-  - Parser functions support: `{{#if:}}`, `{{#switch:}}`, `{{#ifeq:}}`, `{{#expr:}}`
-  - Magic words support: `{{PAGENAME}}`, `{{CURRENTYEAR}}`, `{{NAMESPACE}}`
-
-- **Removed legacy test data**: Deleted obsolete static test files:
-  - `data/testdata_en.bz2` (2.8MB, from 2022)
-  - `data/testdata_ja.bz2` (2.6MB, from 2022)
-  - `data/output_samples/` directory (~20MB)
-  - Tests now use live Wikipedia data with caching
-
-- **Incremental dump downloads**: Smart handling of partial dump files when downloading full dumps:
-  - Detects existing partial downloads and offers to resume (download only remaining data)
-  - Validates dump dates - if dates match, can resume; if outdated, offers choices
-  - User options: resume download, download fresh, keep old partial, or use old as-is
-  - Automatic bz2 validation before and after incremental download
-  - Falls back to full download if server doesn't support HTTP Range headers
-
-- **bz2 file validation**: New `Bz2Validator` module detects corrupt or invalid bz2 files before processing:
-  - Validates magic bytes (`BZ`), version byte (`h`), and block size (`1`-`9`)
-  - Optional decompression test to verify file integrity
-  - `StreamProcessor` validates bz2 files by default (configurable via `validate_bz2: false`)
-  - Detailed error types: `not_found`, `too_small`, `invalid_magic`, `invalid_version`, `invalid_block_size`, `decompression_failed`
-
-- **Memory monitoring**: New `MemoryMonitor` module for adaptive resource management:
-  - Cross-platform memory detection (Linux, macOS, Windows)
-  - Adaptive buffer sizing based on available memory
-  - Memory statistics: `current_memory_usage`, `available_memory`, `memory_usage_percent`
-  - Automatic garbage collection when memory is low
-
-- **Parallel article extraction**: `MultistreamReader` now supports parallel processing:
-  - `extract_articles_parallel(titles, num_processes: 4)` - Extract multiple articles in parallel
-  - `each_article_parallel(entries, num_processes: 4)` - Iterate with parallel processing
-  - Automatically groups articles by stream offset to minimize bz2 decompression overhead
-
-- **Performance optimizations**:
-  - Pre-compiled 14 additional regex patterns for text cleanup
-  - Consolidated gsub chains (3 fewer calls per cleanup operation)
-  - Adaptive buffer sizing in `StreamProcessor` based on system memory
-
-- **Cache staleness warnings**: Cache status now shows age and staleness information:
-  - Displays cache date and age (e.g., "2025-01-05 - 4 days ago")
-  - Warns when cache exceeds configured `dump_expiry_days` (default: 30 days)
-  - New `--update-cache` (`-U`) option to force refresh of cached dump files
-  - Users can choose to use stale cache or force update
-
-- **Category-based extraction**: New `--from-category` option extracts all articles from a Wikipedia category:
-  - `wp2txt --lang=ja --from-category="日本の都市" -o ./output` extracts all articles in the category
-  - `--depth` option for subcategory recursion (e.g., `--depth=2` includes 2 levels of subcategories)
-  - `--dry-run` for preview mode (shows article counts without downloading)
-  - `--yes` to skip confirmation prompt for automation
-  - Circular reference prevention for category hierarchies
-  - Rate limiting for Wikipedia API requests
-
-- **Configuration file**: New `--config-init` option creates persistent configuration:
-  - Settings stored in `~/.wp2txt/config.yml`
-  - Configurable: `dump_expiry_days`, `category_expiry_days`, `cache.directory`
-  - Default output format and subcategory depth
-  - CLI options override config file settings
-
-- **Deprecated `--markers=none`**: Complete removal of special content is now deprecated
-  - Removing inline content (e.g., math formulas) makes surrounding text nonsensical
-  - `--markers=none` now shows a warning and behaves like `--markers=all`
-  - Use `--markers=math,code` to show only specific marker types
-
-- **CLI option validation**: Extraction modes are now mutually exclusive with clear error messages:
-  - `--category-only`, `--summary-only`, `--metadata-only` cannot be combined
-  - `--sections` cannot be used with extraction modes
-  - `--section-stats` cannot be combined with extraction modes or `--sections`
-
-- **Network retry with exponential backoff**: HTTP requests now retry on transient errors:
-  - Retries up to 3 times with exponential backoff (2, 4, 8 seconds)
-  - Handles timeouts, connection resets, and DNS failures
-  - CategoryFetcher API requests now log failures instead of silently returning nil
-
-- **Disk full error handling**: OutputWriter now handles `Errno::ENOSPC` gracefully:
-  - Raises `Wp2txt::FileIOError` with descriptive message on disk full or I/O errors
-  - Properly closes file handles before raising
-
-- **File rotation at article boundaries**: OutputWriter `write_from_file` now rotates output files only at blank lines (article boundaries):
-  - Prevents articles from being split across output files
-  - Eliminates UTF-8 character corruption at file boundaries (e.g., 3-byte Japanese characters split mid-byte)
-  - Uses line-by-line reading (`each_line` with `"r:UTF-8"`) instead of fixed-size byte chunks
-  - Verified with full Japanese Wikipedia (1.49M articles) and English Wikipedia (24.2 GB) dumps
-
-- **HTTP timeout consistency**: All HTTP methods in `DumpManager` now use `DEFAULT_HTTP_TIMEOUT`:
-  - Added `open_timeout`/`read_timeout` to `download_incremental`, `get_remote_file_size`, `download_file_with_progress`, `download_file_range`
-  - Previously these methods had no timeout, risking indefinite hangs on network issues
-
-- **Security: Command injection prevention**: All `IO.popen` calls now use array form:
-  - Fixed unsafe string interpolation in `wp2txt.rb`, `stream_processor.rb`, `bz2_validator.rb`, `memory_monitor.rb`
-  - Prevents shell metacharacter interpretation in file paths
-
-- **Security: SSL certificate verification**: Restored proper TLS certificate validation:
-  - Removed `verify_callback` that unconditionally returned `true` (7 locations in `multistream.rb`)
-  - `VERIFY_PEER` now performs actual certificate verification
-
-- **Security: Temp file handling**: `file_mod` now uses `Tempfile` instead of hardcoded `"temp"` filename:
-  - Prevents predictable file names and potential race conditions
-  - Temp files created in same directory as target file
-
-- **CLI option fixes**:
-  - Added missing `--table` option (keep wiki table content)
-  - Added missing `--multiline` option (keep multi-line templates)
-  - Added missing `--pre` option (keep preformatted text blocks)
-  - Fixed `--ref` option not being transferred to processing config
-  - Reference removal is now conditional (respects `--ref` flag)
-
-- **Ractor turbo mode warning**: Shows explicit warning when `--ractor` is used with turbo mode (unsupported combination)
-
-- **Constants extraction**: Replaced magic numbers with named constants:
-  - `DEFAULT_HTTP_TIMEOUT`, `DEFAULT_PROGRESS_INTERVAL`, `INDEX_PROGRESS_THRESHOLD`
-  - `DEFAULT_TOP_N_SECTIONS`, `RESUME_METADATA_MAX_AGE_DAYS`, `MAX_HTTP_RETRIES`
-
-- **Marker classification**: Markers now categorized as inline or block
-  - **Inline markers** (`[MATH]`, `[CODE]`, `[CHEM]`, `[IPA]`): Content that appears mid-sentence; removal would break grammar
-  - **Block markers** (`[TABLE]`, `[CODEBLOCK]`, `[INFOBOX]`, etc.): Standalone content that can be safely removed
-  - New `[CODEBLOCK]` marker for `<syntaxhighlight>`, `<source>`, `<pre>` tags (block-level code)
-  - `[CODE]` marker now only applies to inline `<code>` tags
+- **Faster startup and repeated runs**: parsed data files, the Wikipedia category hierarchy (per language, 7-day expiry), and the multistream dump index are now cached under `~/.wp2txt/cache/`. Re-reading the index of a full dump used to take ~10 minutes; later runs now start in seconds
+- **`--ractor` (Ruby 4.0+)**: thread-based parallel processing — about 2× faster than sequential, with a smaller memory footprint than the default process-based mode (which remains the fastest at ~3×). On Ruby 3.x the option falls back to sequential processing
+- **Templates are expanded into readable text by default**: dates (`{{birth date|1990|5|15}}` → "May 15, 1990"), unit conversions (`{{convert|100|km|mi}}` → "100 km (62 mi)"), coordinates, language tags (`{{lang|ja|日本語}}` → 日本語), quotes, and 20+ more, plus common parser functions (`{{#if:}}`, `{{#switch:}}`) and magic words (`{{PAGENAME}}`). Turn off with `--no-expand-templates`
+- **Interrupted downloads resume**: a partial dump download is detected and, when the server supports it, only the remaining data is fetched. If the dump date has changed you are asked whether to resume, redownload, or keep the old file; files are validated before and after
+- **Corrupt dump files are detected before processing** instead of failing partway through, with the specific reason reported (file truncated, not a bz2 file, failed decompression test)
+- **Output files rotate at article boundaries**: an article is never split across two output files, and multi-byte characters are no longer corrupted at file boundaries. Verified against the full Japanese (1.49M articles) and English (24.2 GB) dumps
+- **Downloads can no longer hang indefinitely**: every network operation now has a timeout (several previously had none)
+- **Transient network errors retry automatically** (3 attempts with 2/4/8-second backoff) instead of failing the run, and failed category API requests are logged instead of silently returning nothing
+- **Running out of disk space raises a clear error** instead of leaving corrupt output behind
+- **Cache age is visible**: cache listings show the date and age of each cached dump and warn when it exceeds `dump_expiry_days` (default 30); `--update-cache` (`-U`) forces a refresh
+- **`--from-category`**: extract every article in a Wikipedia category via the Wikipedia API, with `--depth` for subcategory levels, `--dry-run` to preview article counts before downloading, and `--yes` for unattended runs
+- **`--config-init`**: writes a persistent configuration file to `~/.wp2txt/config.yml` (cache lifetimes, cache directory, output defaults). Command-line options still take precedence
+- **`--markers=none` is deprecated**: removing inline content (a formula mid-sentence) leaves broken text, so the option now warns and behaves like `--markers=all`. Use `--markers=math,code` to keep only specific marker types
+- **Markers are classified as inline or block**: inline markers (`[MATH]`, `[CODE]`, `[CHEM]`, `[IPA]`) stand in for content whose removal would break the sentence; block markers (`[TABLE]`, `[INFOBOX]`, …) replace standalone content. New `[CODEBLOCK]` marker for code blocks; `[CODE]` now covers only inline code
+- **Conflicting options are rejected with a clear message**: `--category-only`, `--summary-only`, and `--metadata-only` are mutually exclusive, and `--sections` / `--section-stats` cannot be combined with them
+- **Option fixes**: `--table`, `--multiline`, and `--pre` were missing from the CLI and are now available; `--ref` was accepted but had no effect — reference removal now respects it
+- `--ractor` combined with turbo mode now warns explicitly (the combination is unsupported)
+- **Security fixes**: file paths are no longer passed through a shell, so crafted path names cannot inject commands; TLS certificate verification is actually performed (it was previously disabled by an accept-everything callback); temporary files use unpredictable names instead of a fixed name in the working directory
+- **Performance**: text cleanup is faster (fewer passes over each article), and buffer sizes adapt to the memory actually available on the machine
+- **Ruby API**: articles can be extracted from a multistream dump in parallel (`extract_articles_parallel`, `each_article_parallel`), grouped by stream to avoid decompressing the same block twice
 
 ## [2.0.0] - 2026-01-08
 
 ### Added
 
-- **Auto-download mode**: New `--lang` option automatically downloads Wikipedia dumps:
-  - `wp2txt --lang=ja -o ./output` downloads and processes Japanese Wikipedia
-  - Downloads cached to `~/.wp2txt/cache/` for reuse
-  - Supports any Wikipedia language code (en, ja, de, fr, zh, etc.)
-
-- **Article extraction**: New `--articles` option extracts specific articles by title:
-  - `wp2txt --lang=en --articles="Tokyo,Kyoto,Osaka" -o ./articles`
-  - Only downloads index + needed data streams (efficient partial download)
-  - O(1) hash lookup for article search
-
-- **Cache management**: New options to manage downloaded dumps:
-  - `--cache-status` - Show cache status for all languages
-  - `--cache-clear` - Clear all cache
-  - `--cache-clear --lang=ja` - Clear cache for specific language
-  - `--cache-dir` - Custom cache directory
-
-- **Content type markers**: New `--markers` option marks special content:
-  - Supported types: `[MATH]`, `[CODE]`, `[CHEM]`, `[TABLE]`, `[SCORE]`, `[TIMELINE]`, `[GRAPH]`, `[IPA]`, `[INFOBOX]`, `[NAVBOX]`, `[GALLERY]`, `[SIDEBAR]`, `[MAPFRAME]`, `[IMAGEMAP]`, `[REFERENCES]`
-  - `--markers=all` (default) - Enable all markers
-  - `--markers=none` - Disable markers (content removed)
-  - `--markers=math,code` - Enable specific markers only
-
-- **Citation extraction**: New `--extract-citations` (`-C`) option for formatted bibliography output:
-  - Extracts author, title, and year from `{{cite book}}`, `{{cite web}}`, `{{Citation}}` templates
-  - Formats citations as "Author. \"Title\". Year."
-  - Available via CLI (`--extract-citations`) and Ruby API (`extract_citations: true`)
-
-- **Multistream support**: New classes for efficient Wikipedia dump processing:
-  - `MultistreamIndex` - Parse multistream index files
-  - `MultistreamReader` - Extract articles from multistream dumps
-  - `DumpManager` - Download and cache dump files
-  - Enables targeted article extraction without downloading full dump
-
-- **Validation framework**: New rake tasks for validating Wikipedia dump processing:
-  - `testdata:prepare[lang,level]` - Download and cache test data
-  - `validate:run[lang,level]` - Run validation on cached data
-  - `validate:full[lang]` - Full dump validation
-
-- **HTML entity management**: Comprehensive entity support from authoritative sources:
-  - 2125 entities from WHATWG HTML specification (`html_entities.json`)
-  - Wikipedia-specific entities (`wikipedia_entities.json`): `&ratio;`, `&dash;`, `&nbso;`
-  - New script `scripts/fetch_html_entities.rb` to update from WHATWG
-  - Replaces hardcoded entity list with data-driven approach
-
-- **MediaWiki data auto-generation**: Magic words and namespace aliases fetched from all Wikipedia APIs:
-  - New script `scripts/fetch_mediawiki_data.rb` queries 350+ Wikipedia language editions
-  - Data stored in `lib/wp2txt/data/mediawiki_aliases.json`
-  - 176 redirect keywords, 231 category aliases, 313 file aliases
-  - Run `ruby scripts/fetch_mediawiki_data.rb` to update
-
-- **JSON/JSONL output format**: New `--format json` option outputs articles as JSONL (one JSON object per line) with `title`, `categories`, `text`, and `redirect` fields. Ideal for data pipelines and machine learning workflows.
-
-- **Streaming processing**: Complete rewrite of the processing architecture:
-  - No longer creates intermediate XML files
-  - Directly streams from bz2 compressed files
-  - Reduced disk I/O and storage requirements
-  - New `StreamProcessor` and `OutputWriter` classes for modular design
-
-- **Regex cache**: Dynamic regex patterns are now cached to avoid repeated compilation
-
-- **Multilingual category support**: Added support for category namespaces in 30+ languages (European, Cyrillic, Asian, Middle Eastern)
-
-- **Multilingual redirect support**: Added support for redirect keywords in 25+ languages
-
-- **Comprehensive test suite**: 395 tests covering:
-  - Unicode handling (CJK, Cyrillic, Arabic, emoji)
-  - Edge cases (deeply nested templates, malformed markup)
-  - Multilingual category and redirect extraction
-  - Text processing utilities
-  - Integration tests with real Wikipedia content
-
-- **SimpleCov integration**: Added code coverage reporting for development
-
-- **Ruby 4.0 compatibility**: Full support for Ruby 4.0
+- **`--lang`**: downloads the Wikipedia dump for any language code and processes it in one step (`wp2txt --lang=ja -o ./output`). Downloads are cached under `~/.wp2txt/cache/` for reuse
+- **`--articles`**: extract specific articles by title (`wp2txt --lang=en --articles="Tokyo,Kyoto,Osaka" -o ./articles`), downloading only the index and the data streams that contain them rather than the whole dump
+- **Cache management**: `--cache-status` (per-language overview), `--cache-clear` (all languages, or one with `--lang`), and `--cache-dir` (custom location)
+- **Content markers (`--markers`)**: special content is replaced by a marker instead of disappearing silently: `[MATH]`, `[CODE]`, `[CHEM]`, `[TABLE]`, `[SCORE]`, `[TIMELINE]`, `[GRAPH]`, `[IPA]`, `[INFOBOX]`, `[NAVBOX]`, `[GALLERY]`, `[SIDEBAR]`, `[MAPFRAME]`, `[IMAGEMAP]`, `[REFERENCES]`. `--markers=all` is the default; `--markers=math,code` keeps only the listed types
+- **Citation extraction (`--extract-citations`, `-C`)**: outputs a formatted bibliography ("Author. \"Title\". Year.") from `{{cite book}}`, `{{cite web}}`, and `{{Citation}}` templates. Also available from the Ruby API (`extract_citations: true`)
+- **JSON/JSONL output (`--format json`)**: one JSON object per line with `title`, `categories`, `text`, and `redirect` fields, for data pipelines and machine-learning workflows
+- **Multistream dumps are handled natively**, which is what makes targeted extraction work without downloading a full dump. Ruby API: `MultistreamIndex`, `MultistreamReader`, `DumpManager`
+- **HTML character entities are converted comprehensively**: 2,125 entities from the WHATWG HTML specification, plus Wikipedia-specific ones (`&ratio;`, `&dash;`, `&nbso;`)
+- **Redirect, category, and file keywords fetched from 350+ Wikipedia language editions** (176 redirect keywords, 231 category aliases, 313 file aliases), replacing a small hardcoded list
+- **Categories are recognized in 30+ languages** (European, Cyrillic, Asian, and Middle Eastern scripts) and **redirects in 25+**
+- **Fully streamed processing**: dumps are processed directly from the compressed file, with no intermediate XML files — far less disk space and I/O than before
+- **Ruby 4.0 support**
 
 ### Changed
 
-- **Performance improvements**:
-  - `format_wiki`: Reduced intermediate string allocations by using `gsub!` for in-place modifications
-  - `cleanup`: Optimized with `gsub!` to reduce memory allocations
-  - `remove_complex`, `make_reference`: Optimized with `gsub!`
-  - Category deduplication: Changed from O(n²) to O(n) by calling `uniq!` once at end instead of every line
-  - `correct_separator`: Uses `tr` instead of `gsub` for single character replacement
-  - `remove_inbetween`: Dynamic regex patterns are now cached
-
-- **BREAKING**: `REMOVE_HR_REGEX` now matches 4 or more hyphens (previously 3+) to align with MediaWiki specification where `----` is the minimum for horizontal rules
-
-- **`chrref_to_utf` function**: Completely rewritten to support all Unicode codepoints (U+0001 to U+10FFFF), including:
-  - Supplementary plane characters (emoji, CJK Extension B, etc.)
-  - Proper handling of invalid codepoints (returns empty string)
-
-- **`convert_characters` function**: Now uses `String#scrub` for safe handling of invalid UTF-8 sequences instead of calling `exit`
-
-- **`command_exist?` function**: Updated to use `IO.popen` instead of `open("| ...")` for Ruby 4.0 compatibility
+- **BREAKING: horizontal rules now require 4 or more hyphens** (previously 3), matching the MediaWiki specification — a line of `---` in an article is now treated as content, not as a rule
+- **Character references convert across the entire Unicode range** (U+0001–U+10FFFF), including emoji and supplementary-plane CJK; invalid codepoints become empty strings instead of garbage characters
+- **Faster text processing**: category deduplication went from quadratic to linear time, and string handling allocates far less memory on large articles
 
 ### Fixed
 
-- **Unicode BMP limitation**: Fixed `chrref_to_utf` to correctly convert character references beyond the Basic Multilingual Plane (U+FFFF). Previously, emoji like `&#x1F600;` would produce invalid characters.
-
-- **Encoding error crash**: Fixed `convert_characters` which previously called `exit` on encoding errors, now gracefully handles invalid byte sequences using `scrub`
-
-- **Horizontal rule detection**: Fixed `REMOVE_HR_REGEX` to correctly match MediaWiki horizontal rules (4+ hyphens)
-
-- **Heading regex**: Fixed `IN_HEADING_REGEX` to allow trailing whitespace after closing equal signs
-
-- **Ruby 4.0 compatibility**: Fixed `open("| which cmd")` pattern which no longer works in Ruby 4.0
+- **Character references beyond U+FFFF now produce the right character**: `&#x1F600;` becomes 😀 instead of an invalid character
+- **Invalid UTF-8 in a dump no longer aborts the run**: bad byte sequences are scrubbed and processing continues (the tool previously exited mid-run)
+- **Headings with trailing whitespace** after the closing `==` are now recognized as headings
 
 ### Deprecated
 
-- **`--convert` / `-c` option**: No longer needed as streaming processing always converts
-- **`--del-interfile` / `-x` option**: No longer needed as intermediate files are no longer created
+- **`--convert` / `-c`**: no longer needed — streaming processing always converts
+- **`--del-interfile` / `-x`**: no longer needed — intermediate files are no longer created
 
 ### Removed
 
-- **Intermediate XML file creation**: The `Splitter` class no longer creates intermediate XML files; processing is now fully streamed
+- **Intermediate XML files**: processing no longer writes intermediate XML next to the output at any point
 
 ### Security
 

data/DEVELOPMENT.md

@@ -383,6 +383,14 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 2. Redirect keywords: `data/language_redirects.json`
 3. Scripts: `scripts/generate_language_data.rb`
 
+## Documentation
+
+User-facing documents (README, README_ja, CHANGELOG, docs/) describe what a user can
+observe and what they need to do — not how the code works internally or why it is
+designed the way it is. Keep implementation detail in this file and in code comments,
+and keep design rationale out of the public repository entirely. The MCP tool table in
+docs/INDEXES.md is checked against the actual server surface by spec/docs_sync_spec.rb.
+
 ## Code Style
 
 - Ruby 2.6+ compatibility
@@ -392,13 +400,21 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 
 ## Docker
 
-Build and push Docker images:
+Images are published by GitHub Actions when a `v*` tag is pushed
+(`.github/workflows/publish-image.yml`), not from a maintainer's machine: a
+local build sends the working tree as its context, so untracked files ride
+along, while a runner starts from a clean checkout.
 
 ```bash
-rake check_image  # Builds the image locally and verifies it carries no private files
-rake push        # Verifies, then builds multi-arch and pushes to GHCR
+rake check_image   # Build locally and run the same gate the workflow runs
 ```
 
+The gate (`scripts/verify_image.rb`) lists what the image holds under `/wp2txt`
+and compares it against what Docker sends as the build context, in both
+directions. Anything unaccounted for fails, so a file the build starts
+producing must be added to `BUILD_ARTIFACTS` deliberately. To rehearse a
+publish, run the workflow from the Actions tab with **push** left off.
+
 ## Release Process
 
 1. Update version in `lib/wp2txt/version.rb`
@@ -406,7 +422,7 @@ rake push        # Verifies, then builds multi-arch and pushes to GHCR
 3. Run full test suite: `bundle exec rspec`
 4. Build gem: `gem build wp2txt.gemspec`
 5. Push to RubyGems: `gem push wp2txt-*.gem`
-6. Push Docker image: `rake push`
+6. Push the `v*` tag — GitHub Actions builds, gates, and publishes the image
 7. Create GitHub release
 
 ## Useful Links

data/DEVELOPMENT_ja.md

@@ -383,6 +383,13 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 2. リダイレクトキーワード: `data/language_redirects.json`
 3. スクリプト: `scripts/generate_language_data.rb`
 
+## ドキュメンテーション
+
+利用者向け文書（README, README_ja, CHANGELOG, docs/）には、利用者が観測できることと
+利用者が取るべき行動だけを書く — コードが内部でどう動くか、なぜそう設計したかは書かない。
+実装の詳細は本ファイルとコードコメントに、設計の理由づけは公開リポジトリの外に置く。
+docs/INDEXES.md の MCP ツール表は spec/docs_sync_spec.rb が実際のサーバー表面と照合する。
+
 ## コードスタイル
 
 - Ruby 2.6+互換性
@@ -395,8 +402,7 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 Dockerイメージのビルドとプッシュ：
 
 ```bash
-rake check_image  # ローカルでイメージをビルドし、私的ファイルの混入がないか検証
-rake push        # 検証したうえでマルチアーキテクチャでビルドしGHCRにプッシュ
+rake check_image   # ローカルでビルドし、CI と同じゲートを走らせる
 ```
 
 ## リリースプロセス
@@ -406,7 +412,7 @@ rake push        # 検証したうえでマルチアーキテクチャでビル
 3. フルテストスイートを実行: `bundle exec rspec`
 4. gemをビルド: `gem build wp2txt.gemspec`
 5. RubyGemsにプッシュ: `gem push wp2txt-*.gem`
-6. Dockerイメージをプッシュ: `rake push`
+6. `v*` タグをプッシュ — GitHub Actions がビルド・検証・公開を行う
 7. GitHubリリースを作成
 
 ## 便利なリンク