wp2txt 2.3.1 → 2.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4062c47480d0801e7a07d8e3bec7c65be8efeeda2eae30d398f9872810d321cd
-  data.tar.gz: 25bb3aa8a4d5baefc4616a8a4b754ec63e92d3e0b86b22ccc0ead2929a39739e
+  metadata.gz: 874d5f0c6463fc612080339cbbcbe560b51aba31a8a5e9478d95719c53744051
+  data.tar.gz: e8377c0df46cd7d35ca8a85cea4f52e6dc4ca57330004ce5fc6f11913aa1abcf
 SHA512:
-  metadata.gz: ff61afbb989613e286af8bea784e4494c8f71b0dd52d24d3d038e72c5d77f9196d899039d9ff8c5e4b16461c044670d8cca6bc3feda55aa8cdbc31cbc30e012b
-  data.tar.gz: ddffdfb2452a3710d527bc7deeae52818cb9c2b4dedd433b2c7ecb06db74e6e389edf127976567a3269583531acf0aabc2ea9058d7422b5b675de9bae2e227fa
+  metadata.gz: ce2ca7660379c37a3f36d6d731aa3a9c1798998c10dfb5ff012503f571653b26aef6324e579c1a8af7f0d2ff52cf90fc8c6a50f44f21d40b73d132277cb50a26
+  data.tar.gz: d8197213c46510742d015dcea77c8ff5c40d48dac7beff8cc409ce1e4ce5f3e138e47d7df9ce675a0614785b21bb266664f8c082c94080c4f3a29c712ffeab07

data/.gitignore

@@ -36,3 +36,4 @@ benchmark_results/
 
 # Developer-specific Ruby version
 .ruby-version
+.private-doc-tokens

data/CHANGELOG.md

@@ -5,321 +5,118 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+Entries were rewritten in August 2026 to describe what changed for people using
+wp2txt, rather than how it was implemented. The changes themselves are unaltered.
+
+## [2.3.2] - 2026-08-13
+
+- **Documentation release — no code changes.** The changelog and guides bundled with the gem and the container image are rewritten to say what each change means for someone using wp2txt, rather than how it was implemented. The guide formerly at `docs/RESEARCH.md` is now [docs/INDEXES.md](docs/INDEXES.md) ("Offline Indexes, Queries, and the MCP Server")
+
 ## [2.3.1] - 2026-08-12
 
-- **Container images no longer carry private files**: the image is built from the working tree with `git init && git add -A`, but the build context excluded `.gitignore`, so files ignored locally were tracked inside the container and shipped in the image — 2.3.0's images contained the maintainer's private research notes under `/wp2txt/research-notes`. The context now keeps `.gitignore` (so the container's file list matches a local `gem build`), explicitly excludes `research-notes` and `.bundle`, and the throwaway `.git` (a blob copy of every added file) is deleted after the build. `rake check_image` builds and inspects an image for private paths, and `rake push` refuses to publish without that check. The published gems were never affected; 2.3.0 images have been withdrawn
-- **Docker Hub retired**: images are published to GitHub Container Registry only (`ghcr.io/yohasebe/wp2txt`). The Docker Hub repository is no longer updated
-- **Fix: multi-line `<ref>` references broken by element splitting — also fixes `--extract-citations` on multi-line cite templates**: `Article#parse` splits elements at newlines, so any reference written across lines (very common for multi-line `{{cite …}}` templates) had its `[ref]` and `[/ref]` markers land in separate elements, invisible to `remove_ref`. The markers and raw reference markup leaked into the output, and — more importantly — `--extract-citations` silently never fired for multi-line cite templates, producing inconsistent output versus their single-line equivalents. `make_reference` now drops empty references outright and flattens the rest onto a single line before element splitting, so multi-line references behave exactly like single-line ones. Ordinary paragraph breaks outside references are unaffected
-- **Fix: `[ref]` markers destroyed by external-link processing**: `process_external_links` stripped the brackets of the `[ref]`/`[/ref]` markers produced by `make_reference` (their contents took the single-word branch), so `remove_ref` could no longer find them and tag names plus reference bodies leaked into extracted text — even with `--ref`, the kept markers came out broken. The markers are now hidden behind placeholders for the duration of the bracket scan and restored afterwards. Downstream corpora no longer contain `ref…/ref` residue, which broke tokenization/sentence splitting and mixed bibliographic text into body prose
-- **Runaway-query hardening**: a `query_sql` child process now sets its own kernel-enforced CPU limit (`RLIMIT_CPU`, the query timeout plus a small grace) in addition to the parent's wall-clock kill. Previously a child orphaned by the parent's death — an interrupted test run, a closed terminal, a crashed server — kept executing forever: sqlite3 holds the GVL inside `sqlite3_step`, so Ruby never reaches a signal-safe point and even SIGTERM is ignored. Observed in the wild as two processes spinning at 99% CPU for over five days. The CPU limit is strictly more permissive than the existing wall-clock deadline, so no query that would otherwise succeed is affected
-- **Test-suite hang guard**: each example now runs under a wall-clock timeout (120s default; `WP2TXT_SPEC_TIMEOUT=0` disables it, the `:no_timeout` tag exempts an example). The suite deliberately exercises runaway queries, so a wedged example must fail rather than spin
+- **Fix: references leaked into extracted text**: reference contents — tag remnants like `ref…/ref` and the bibliographic text inside them — could end up in the extracted body text, breaking sentence splitting and mixing citation details into running prose. In a spot check of five English articles this affected about 70 places per article. References are now removed cleanly (or kept intact as `[ref]…[/ref]` with `--ref`), including references written across several lines and empty ones
+- **Fix: `--extract-citations` ignored multi-line citation templates**: a `{{cite …}}` template split over several lines was left as raw markup instead of being formatted, so citations extracted from such references were silently missing. Single-line and multi-line templates now produce the same result
+- **Long-running queries can no longer outlive their timeout**: an interrupted session could leave a `query_sql` process running at full CPU indefinitely. Queries now stop on their own even if the process that started them goes away. Queries that complete within the time limit are unaffected
+- **Container images move to GitHub Container Registry**: images are published to `ghcr.io/yohasebe/wp2txt` only. The Docker Hub repository is no longer updated — please switch:
 
-## [2.3.0] - 2026-07-24
+      docker pull ghcr.io/yohasebe/wp2txt
 
-- **Documentation split**: README now focuses on text extraction; the research layer (indexes, exhaustive queries, full-text search, langlinks, cross-language SQL, MCP server) is documented in the new [Research Infrastructure Guide](docs/RESEARCH.md), including the complete MCP tool table
-- **Container images on GHCR**: images are now published to `ghcr.io/yohasebe/wp2txt` (Docker Hub `yohasebe/wp2txt` is maintained as a mirror)
+## [2.3.0] - 2026-07-24
 
-- **`extract_corpus` `titles:` argument**: Extract an explicit set of article titles (e.g. a set determined via `query_sql`) — titles are normalized (MediaWiki rules), deduplicated preserving input order, and one redirect hop is resolved; missing titles (including redirects to nowhere) are skipped and reported as `not_found` (count + 20-title sample, also in the `.meta.json` sidecar). Mutually exclusive with the filter arguments (set operations belong in SQL); capped at 10,000 titles. The sidecar records `titles_count` + `titles_sha256` (order-independent) for reproducibility, enumerating the full list when ≤100 titles. Also available via `start_extract_job`
-- **`query_sql` `output_path:` argument**: Write ALL rows of a large result to a JSONL file (with a `.meta.json` sidecar recording the SQL, dump version, attach configuration, row count, and tool version) and return only a summary + 3-row sample — the extract_corpus D4 pattern generalized to SQL. Writes happen in the forked child (so the 30s SIGKILL deadline covers them) to a `.partial` file that the parent atomically renames on success and removes on every failure path (child crash, timeout kill, query error). Hard cap `SQL_FILE_ROW_LIMIT` = 5M rows (`truncated` flag), cells clipped at 64KB (`cells_clipped` count), duplicate column names are made unique (`_2` suffix), `limit` is ignored in this mode, and existing files require `overwrite: true`. The MCP layer's output-dir confinement is now a shared helper (`Wp2txt::OutputPath.confine`) used by extract_corpus, start_extract_job, and query_sql alike
-- **Interlanguage links (`--import-langlinks`)**: Import the official `{lang}wiki-{date}-langlinks.sql.gz` dump into the Tier 1 metadata index as a `langlinks` table (`ll_from` = source page_id, `ll_lang` = target language, `ll_title` = normalized target title). Version pinning is enforced: the langlinks file's dump name must equal the built index's dump version — a mismatch is rejected with no override. Streams the MySQL dump without loading it whole (escape-safe tuple parser, 10k-row transaction batches, indexes created after the load). `--langlinks-langs` restricts imported target languages (e.g. `en,de,fr,zh,ko`); re-import requires `-U` (otherwise a no-op reporting the previous import time). Provenance (source file, size, import time, wp2txt version, language filter, row count) is stamped into the index and reported by `dump_info`; a post-import sanity check samples titles per language and warns when the join rate against a locally installed target edition falls below 90%
-- **Multi-dump ATTACH in `query_sql`**: The MCP `query_sql` tool gains an optional `attach` argument (language codes only, never paths): `attach: ["en"]` read-only ATTACHes that language's locally installed metadata DB as `en_meta` and its FTS DB (when built) as `en_fts`, both sharing the main DB's schema. Codes are validated, only installed indexes are accepted, path resolution is server-side (same-date dump preferred; otherwise the latest build with a `dump_mismatch` note in the response's `attached` metadata). User SQL still cannot contain ATTACH/DETACH — attachments are issued by server code only, via `mode=ro` URIs on a read-only connection. Combined with the langlinks table, this enables single-query cross-language comparisons (e.g. section-structure diffs of article pairs)
+- **Extract an explicit list of articles**: `extract_corpus` accepts `titles:` — hand it the set you arrived at some other way (a `query_sql` result, your own list) instead of describing it with filters again. Titles are normalized the way MediaWiki treats them, duplicates are dropped, and one redirect hop is followed. Titles with no article are skipped and reported back as `not_found`, so a missing article is visible rather than silently absent. Up to 10,000 titles; cannot be combined with the filter arguments. Also available through `start_extract_job`
+- **Write a large SQL result to a file**: `query_sql` accepts `output_path:` — all rows go to a JSONL file and the reply carries a summary plus three sample rows, instead of a row cap. The file is written completely or not at all: an interrupted or failing query leaves no partial output. Up to 5 million rows, cells longer than 64KB are clipped (both are reported back), and an existing file is only replaced with `overwrite: true`. A `.meta.json` beside the output records the SQL, the dump versions involved, and the row count
+- **Interlanguage links (`--import-langlinks`)**: import the official `langlinks` dump into the metadata index to map articles across editions. The langlinks file must carry the same dump date as your index — a mismatch is refused, so links can never be mixed across versions. `--langlinks-langs en,de,fr,zh,ko` limits which target languages are imported; re-importing requires `-U`. After the import, a sample of titles per language is checked against any editions you have installed and the match rate is reported. `dump_info` shows what was imported and when
+- **Query several editions in one statement**: `query_sql` accepts `attach: ["en"]`, which makes that language's index available as `en_meta` (and `en_fts` where a full-text index exists) alongside your own. You pass language codes, not paths, and only editions you have installed are accepted. The reply lists what was attached, including each edition's dump name, and says so when the dates differ. Together with the `langlinks` table this makes cross-edition comparisons a single query
+- **Documentation split**: the README now covers text extraction; the index and MCP features (indexes, offline queries, full-text search, interlanguage links, cross-language SQL, MCP server) moved to a separate guide — now [docs/INDEXES.md](docs/INDEXES.md) — which lists every MCP tool
+- **Container images on GitHub Container Registry**: images are now published to `ghcr.io/yohasebe/wp2txt`
 
 ## [2.2.0] - 2026-07-22
 
-- **Index hardening (design review follow-up)**: `ord` now has identical semantics in `page_sections` and `fts_map` (lead = 0, first heading = 1; schema v2 — rebuild indexes with `--build-index -U`); section headings are normalized identically in both indexes (decorated headings like `== '''X''' ==` now match section filters); indexes record the wp2txt version (and the FTS index a rendering-config digest) so `dump_info` can flag indexes built by code whose text cleaning differs; rebuilds are atomic (built alongside, renamed on completion — a failed rebuild no longer destroys the working index)
-- **MCP safety hardening**: `query_sql` runs in a killable subprocess with a 30s wall-clock cap and returns a query-plan diagnosis on timeout; the keyword screen no longer rejects legitimate values inside string literals; `extract_corpus` output paths are confined to the server output directory and refuse to overwrite existing files without `overwrite: true`; background jobs are serialized (one at a time) and all SQLite connections are closed before forking extraction workers
-- **Cold-start parity for LLM clients**: closes the gaps that previously required internal knowledge to work around. `get_categories` tool; `find_articles`/`extract_corpus` accept `categories` (exact multi-category AND) and `category_match` (substring on category names); `extract_corpus` gains `content: "wikitext"` for structure mining (infoboxes/templates); `query_sql` read-only SQL escape hatch (SELECT/WITH only, keyword-screened, read-only connection, row/cell caps) with `describe_schema`; article titles are normalized like MediaWiki (underscores, capitalization); `get_article` truncates at `max_chars` (default 40k) with an explicit flag; `dump_info` reports `fulltext_current`
-- **Deferred FTS optimize**: `--skip-fts-optimize` skips the single-threaded segment-merge step of the full-text build (the dominant cost on many-core machines: 40-60+ min on a full dump), leaving a fully searchable index; `--fts-optimize` runs the merge later, standalone and idempotent. Optimize state is recorded in index metadata and reported by `dump_info`
-- **Full-text search (Tier 2)**: `--build-index --fulltext` builds a contentless SQLite FTS5 index over cleaned section text (tokenizer auto-selected by language: character-trigram for CJK, unicode61 for space-delimited; `--fts-tokenizer` to override, porter stemming opt-in). Search via `--search` (CLI) or the `search_text` MCP tool: literal phrase or raw FTS5 query modes, composable with category recursion and section filters, `count: "capped"|"exact"` (exact 0 = a verified absence claim for the dump version). Snippets are re-rendered from the dump on demand — the index stores no text, keeping disk cost to the inverted index only
-- **Background extraction jobs**: `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` MCP tools for extractions beyond the 5000-article synchronous cap. Each job runs in its own thread with isolated resources; extraction now streams to disk in batches (memory-safe at any scale)
-- **RAG chunking**: `extract_corpus` accepts `chunk_size` / `chunk_overlap`, emitting one record per chunk with `section_path`, `chunk_index`, and `chunk_count`; chunk boundaries prefer sentence/paragraph breaks
-- **Alias guardrail**: `save_alias_set` re-checks co-occurrence server-side and refuses groups containing frequently-coexisting heading pairs (likely distinct roles, not synonyms) unless `force` is passed — protocol compliance no longer depends on the calling model's discipline
-- **`discover_aliases` MCP prompt**: bundled recipe walking any agent through the alias verification protocol (discover via `section_stats` → verify via `section_cooccurrence` → sample borderline cases → save)
-- **MCP server (`wp2txt-mcp`)**: New binary exposing a local dump to LLM agents via the Model Context Protocol (stdio). Tools: `dump_info`, `get_article`, `get_sections`, `list_headings`, `find_articles`, `category_tree`, `section_stats`, `section_cooccurrence`, `save_alias_set`/`get_alias_set`/`list_alias_sets`, and `extract_corpus` (writes JSONL + reproducibility `.meta.json` sidecar; returns summary and sample only). Requires the `mcp` gem (`gem install mcp`); wp2txt itself does not depend on it
-- **LLM-generated section aliases**: Instead of shipping per-language alias dictionaries, agents discover real heading usage with `section_stats`, verify synonym hypotheses with `section_cooccurrence` (synonymous headings almost never co-occur in one article), and persist named alias sets per dump with `save_alias_set`. Queries reference them via `alias_set`; extraction metadata records the exact set contents used
-- **`Wp2txt::Corpus` facade**: Shared query/extraction layer used by the MCP server; lazy SQLite-backed title lookup avoids loading multi-million-entry indexes into memory
-- **Local metadata index (`--build-index`)**: New offline index built by scanning a multistream dump in parallel. Stores per-article categories, section headings, redirects, and the category hierarchy in SQLite (`~/.wp2txt/cache/*_meta.sqlite3`), keyed to the dump version. No API access required
-- **Offline exhaustive queries (`--find-articles`)**: List articles matching `--in-category` (recursive via `--depth`, powered by dump-derived category hierarchy), `--has-section` (alias-aware), and `--title-match` filters. Supports `--limit` and JSON output (`-j json`); redirects are excluded automatically. Enables queries like "all film articles that have a Plot section" against a version-pinned local dump
+- **Index format change — rebuild existing indexes with `--build-index -U`** (schema v2). Section headings are now recorded consistently, so headings written with decoration (`== '''X''' ==`) match section filters instead of being missed. Each index records the wp2txt version that built it, and `dump_info` warns when an index was built by a version whose text cleaning differs from the one you are running. A rebuild that fails partway no longer destroys the index you already had
+- **Safer defaults for MCP use**: `query_sql` stops after 30 seconds and explains what made the query slow; queries containing words like `Update` inside a quoted value are no longer rejected by mistake. `extract_corpus` can only write under the server's output directory and will not replace an existing file unless you pass `overwrite: true`. Background jobs run one at a time
+- **More ways to select articles from an LLM client**: a `get_categories` tool; `categories` (article must be in all of them) and `category_match` (substring on category names) on `find_articles` and `extract_corpus`; `content: "wikitext"` on `extract_corpus` for reading infoboxes and templates that cleaned text replaces with markers; `query_sql` for read-only SQL when the fixed tools cannot express a question, with `describe_schema` to see the tables. Article titles are matched the way MediaWiki treats them (underscores, capitalization). `get_article` stops at `max_chars` (default 40,000) and says when it truncated; `dump_info` reports whether the full-text index is up to date
+- **Split the full-text build in two**: `--skip-fts-optimize` leaves out the final merge step — the slowest part on a full dump, 40–60 minutes and single-threaded — and still gives you a searchable index. Run `--fts-optimize` later when convenient; it is safe to repeat. `dump_info` shows whether the merge has been done
+- **Full-text search**: `--build-index --fulltext` adds a searchable index over the cleaned article text. The tokenizer is chosen by language (character trigrams for Japanese, Chinese, Korean; word-based elsewhere, `--fts-tokenizer` to override). Search from the CLI with `--search` or from an LLM client with `search_text`, combined with category and section filters, as a literal phrase or a full-text query. Counts can be capped for speed or exact — an exact `0` means the term is absent from that dump. The index stores no article text (snippets are re-read from the dump when needed), so it costs far less disk than a copy of the corpus
+- **Background extraction jobs**: `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` for extractions beyond the 5,000-article limit of a direct call. Extraction writes to disk as it goes, so memory use no longer grows with the size of the job
+- **Chunked output for RAG**: `extract_corpus` accepts `chunk_size` / `chunk_overlap` and emits one record per chunk with its `section_path`, `chunk_index`, and `chunk_count`. Chunks prefer to break at sentence and paragraph boundaries
+- **Alias sets are checked before they are saved**: `save_alias_set` refuses a group whose headings frequently appear together in the same article — a sign they are different sections rather than synonyms — unless you pass `force`
+- **`discover_aliases` prompt**: a bundled walkthrough of finding heading variants (`section_stats` → `section_cooccurrence` → save), so you do not have to remember the steps
+- **MCP server (`wp2txt-mcp`)**: a new command that exposes a local dump to an LLM client over the Model Context Protocol. Tools cover dump identity, single articles and their sections, filtered listings, category trees, heading statistics, alias sets, and extraction to JSONL with a `.meta.json` recording the dump version and query. Requires the `mcp` gem (`gem install mcp`); wp2txt itself does not depend on it
+- **Section aliases come from the dump, not from a bundled dictionary**: find the headings actually used with `section_stats`, check whether two of them mean the same thing with `section_cooccurrence`, and save the group with `save_alias_set`. Queries then refer to it by name via `alias_set`, and extractions record which group was used
+- **Ruby API**: `Wp2txt::Corpus` provides the query and extraction layer used by the MCP server, for embedding wp2txt in your own code. Title lookup reads from SQLite on demand rather than loading a multi-million-entry index into memory
+- **Offline metadata index (`--build-index`)**: builds a local index of a dump — each article's categories, section headings, redirects, and the category hierarchy — by scanning it in parallel. Stored under `~/.wp2txt/cache/` and tied to the dump version. No API access needed
+- **Offline queries over a whole edition (`--find-articles`)**: list articles by `--in-category` (with `--depth` to include subcategories), `--has-section` (alias-aware), and `--title-match`. Supports `--limit` and JSON output (`-j json`); redirects are excluded. This is what makes questions like "all film articles that have a plot section" answerable against a pinned dump
 
 ## [2.1.2] - 2026-07-19
 
-- **Fixed gem file permissions**: The published gem contained files with owner-only (0600) permissions inherited from the build machine, making them unreadable after `sudo gem install`. A `normalize_permissions` task now runs before `rake build`, ensuring all packaged files are world-readable (0644, or 0755 for executables)
+- **Fixed unreadable files after `sudo gem install`**: files in the published gem carried owner-only permissions, so wp2txt failed to run for anyone but the installing user. Packaged files are now readable by everyone (0644, and 0755 for executables). If you hit this, reinstall the gem
 
 ## [2.1.1] - 2026-02-21
 
-- **Bidirectional alias matching**: Section extraction now supports reverse alias lookup - specifying an alias name (e.g., "Synopsis") as target matches the canonical heading ("Plot") and vice versa
-- **Expanded default section aliases**: Increased from 2 to 12 alias groups covering common English Wikipedia sections (Plot, Reception, References, Bibliography, Awards, Legacy, Early life, Career, etc.)
-- **Config forwarding fix**: `--pre`, `--ref`, `--expand-templates`, and `--metadata-only` options now correctly forwarded in `--articles` and `--from-category` modes
+- **Section aliases match in both directions**: asking for "Synopsis" now also finds sections headed "Plot", and the other way round, instead of only matching the name you happened to type
+- **More section aliases out of the box**: 12 groups instead of 2, covering common English Wikipedia sections (Plot, Reception, References, Bibliography, Awards, Legacy, Early life, Career, and others)
+- **Fixed options being ignored**: `--pre`, `--ref`, `--expand-templates`, and `--metadata-only` had no effect when used with `--articles` or `--from-category`. They now apply in those modes too
 
 ## [2.1.0] - 2026-02-19
 
-- **SQLite-based caching infrastructure**: New high-performance caching using SQLite for faster startup and repeated operations:
-  - `GlobalDataCache`: Caches parsed JSON data files (templates, MediaWiki aliases, HTML entities)
-    - Eliminates ~500KB JSON parsing overhead on each startup
-    - Validates cache against source file modification time and size
-    - Location: `~/.wp2txt/cache/global_data.sqlite3`
-  - `CategoryCache`: Caches Wikipedia category hierarchy from API
-    - Stores category members (pages and subcategories) in SQLite tables
-    - Supports recursive tree traversal and bulk page retrieval
-    - Per-language cache files: `~/.wp2txt/cache/categories_en.sqlite3`
-    - Configurable expiry (default: 7 days)
-  - `IndexCache`: Caches parsed multistream index (already existed, now with SQLite3 2.x compatibility)
-    - Reduces index parsing from ~10 minutes to seconds on subsequent runs
-  - All caches use WAL mode for concurrent read access during parallel processing
-
-- **Ractor parallel processing (Ruby 4.0+)**: New `--ractor` option for thread-based parallelism:
-  - Requires Ruby 4.0 or later for stable operation
-  - Uses map-join-value pattern for reliable Ractor orchestration
-  - ~2x speedup compared to sequential processing
-  - Lower memory footprint than process-based parallelism (Parallel gem)
-  - Automatic fallback to sequential processing on Ruby 3.x
-  - Performance: Parallel gem (~3x) remains faster, Ractor (~2x) uses less memory
-
-- **Template expansion**: New `--expand-templates` (`-E`) option expands common templates to readable text:
-  - Date templates: `{{birth date|1990|5|15}}` → "May 15, 1990"
-  - Convert templates: `{{convert|100|km|mi}}` → "100 km (62 mi)"
-  - Coordinate templates: `{{coord|35|41|N|139|41|E}}` → "35°41′N 139°41′E"
-  - Language templates: `{{lang|ja|日本語}}` → "日本語"
-  - Quote templates: `{{blockquote|text}}` → "text"
-  - And 20+ more template types
-  - **Enabled by default** - use `--no-expand-templates` to disable
-  - Parser functions support: `{{#if:}}`, `{{#switch:}}`, `{{#ifeq:}}`, `{{#expr:}}`
-  - Magic words support: `{{PAGENAME}}`, `{{CURRENTYEAR}}`, `{{NAMESPACE}}`
-
-- **Removed legacy test data**: Deleted obsolete static test files:
-  - `data/testdata_en.bz2` (2.8MB, from 2022)
-  - `data/testdata_ja.bz2` (2.6MB, from 2022)
-  - `data/output_samples/` directory (~20MB)
-  - Tests now use live Wikipedia data with caching
-
-- **Incremental dump downloads**: Smart handling of partial dump files when downloading full dumps:
-  - Detects existing partial downloads and offers to resume (download only remaining data)
-  - Validates dump dates - if dates match, can resume; if outdated, offers choices
-  - User options: resume download, download fresh, keep old partial, or use old as-is
-  - Automatic bz2 validation before and after incremental download
-  - Falls back to full download if server doesn't support HTTP Range headers
-
-- **bz2 file validation**: New `Bz2Validator` module detects corrupt or invalid bz2 files before processing:
-  - Validates magic bytes (`BZ`), version byte (`h`), and block size (`1`-`9`)
-  - Optional decompression test to verify file integrity
-  - `StreamProcessor` validates bz2 files by default (configurable via `validate_bz2: false`)
-  - Detailed error types: `not_found`, `too_small`, `invalid_magic`, `invalid_version`, `invalid_block_size`, `decompression_failed`
-
-- **Memory monitoring**: New `MemoryMonitor` module for adaptive resource management:
-  - Cross-platform memory detection (Linux, macOS, Windows)
-  - Adaptive buffer sizing based on available memory
-  - Memory statistics: `current_memory_usage`, `available_memory`, `memory_usage_percent`
-  - Automatic garbage collection when memory is low
-
-- **Parallel article extraction**: `MultistreamReader` now supports parallel processing:
-  - `extract_articles_parallel(titles, num_processes: 4)` - Extract multiple articles in parallel
-  - `each_article_parallel(entries, num_processes: 4)` - Iterate with parallel processing
-  - Automatically groups articles by stream offset to minimize bz2 decompression overhead
-
-- **Performance optimizations**:
-  - Pre-compiled 14 additional regex patterns for text cleanup
-  - Consolidated gsub chains (3 fewer calls per cleanup operation)
-  - Adaptive buffer sizing in `StreamProcessor` based on system memory
-
-- **Cache staleness warnings**: Cache status now shows age and staleness information:
-  - Displays cache date and age (e.g., "2025-01-05 - 4 days ago")
-  - Warns when cache exceeds configured `dump_expiry_days` (default: 30 days)
-  - New `--update-cache` (`-U`) option to force refresh of cached dump files
-  - Users can choose to use stale cache or force update
-
-- **Category-based extraction**: New `--from-category` option extracts all articles from a Wikipedia category:
-  - `wp2txt --lang=ja --from-category="日本の都市" -o ./output` extracts all articles in the category
-  - `--depth` option for subcategory recursion (e.g., `--depth=2` includes 2 levels of subcategories)
-  - `--dry-run` for preview mode (shows article counts without downloading)
-  - `--yes` to skip confirmation prompt for automation
-  - Circular reference prevention for category hierarchies
-  - Rate limiting for Wikipedia API requests
-
-- **Configuration file**: New `--config-init` option creates persistent configuration:
-  - Settings stored in `~/.wp2txt/config.yml`
-  - Configurable: `dump_expiry_days`, `category_expiry_days`, `cache.directory`
-  - Default output format and subcategory depth
-  - CLI options override config file settings
-
-- **Deprecated `--markers=none`**: Complete removal of special content is now deprecated
-  - Removing inline content (e.g., math formulas) makes surrounding text nonsensical
-  - `--markers=none` now shows a warning and behaves like `--markers=all`
-  - Use `--markers=math,code` to show only specific marker types
-
-- **CLI option validation**: Extraction modes are now mutually exclusive with clear error messages:
-  - `--category-only`, `--summary-only`, `--metadata-only` cannot be combined
-  - `--sections` cannot be used with extraction modes
-  - `--section-stats` cannot be combined with extraction modes or `--sections`
-
-- **Network retry with exponential backoff**: HTTP requests now retry on transient errors:
-  - Retries up to 3 times with exponential backoff (2, 4, 8 seconds)
-  - Handles timeouts, connection resets, and DNS failures
-  - CategoryFetcher API requests now log failures instead of silently returning nil
-
-- **Disk full error handling**: OutputWriter now handles `Errno::ENOSPC` gracefully:
-  - Raises `Wp2txt::FileIOError` with descriptive message on disk full or I/O errors
-  - Properly closes file handles before raising
-
-- **File rotation at article boundaries**: OutputWriter `write_from_file` now rotates output files only at blank lines (article boundaries):
-  - Prevents articles from being split across output files
-  - Eliminates UTF-8 character corruption at file boundaries (e.g., 3-byte Japanese characters split mid-byte)
-  - Uses line-by-line reading (`each_line` with `"r:UTF-8"`) instead of fixed-size byte chunks
-  - Verified with full Japanese Wikipedia (1.49M articles) and English Wikipedia (24.2 GB) dumps
-
-- **HTTP timeout consistency**: All HTTP methods in `DumpManager` now use `DEFAULT_HTTP_TIMEOUT`:
-  - Added `open_timeout`/`read_timeout` to `download_incremental`, `get_remote_file_size`, `download_file_with_progress`, `download_file_range`
-  - Previously these methods had no timeout, risking indefinite hangs on network issues
-
-- **Security: Command injection prevention**: All `IO.popen` calls now use array form:
-  - Fixed unsafe string interpolation in `wp2txt.rb`, `stream_processor.rb`, `bz2_validator.rb`, `memory_monitor.rb`
-  - Prevents shell metacharacter interpretation in file paths
-
-- **Security: SSL certificate verification**: Restored proper TLS certificate validation:
-  - Removed `verify_callback` that unconditionally returned `true` (7 locations in `multistream.rb`)
-  - `VERIFY_PEER` now performs actual certificate verification
-
-- **Security: Temp file handling**: `file_mod` now uses `Tempfile` instead of hardcoded `"temp"` filename:
-  - Prevents predictable file names and potential race conditions
-  - Temp files created in same directory as target file
-
-- **CLI option fixes**:
-  - Added missing `--table` option (keep wiki table content)
-  - Added missing `--multiline` option (keep multi-line templates)
-  - Added missing `--pre` option (keep preformatted text blocks)
-  - Fixed `--ref` option not being transferred to processing config
-  - Reference removal is now conditional (respects `--ref` flag)
-
-- **Ractor turbo mode warning**: Shows explicit warning when `--ractor` is used with turbo mode (unsupported combination)
-
-- **Constants extraction**: Replaced magic numbers with named constants:
-  - `DEFAULT_HTTP_TIMEOUT`, `DEFAULT_PROGRESS_INTERVAL`, `INDEX_PROGRESS_THRESHOLD`
-  - `DEFAULT_TOP_N_SECTIONS`, `RESUME_METADATA_MAX_AGE_DAYS`, `MAX_HTTP_RETRIES`
-
-- **Marker classification**: Markers now categorized as inline or block
-  - **Inline markers** (`[MATH]`, `[CODE]`, `[CHEM]`, `[IPA]`): Content that appears mid-sentence; removal would break grammar
-  - **Block markers** (`[TABLE]`, `[CODEBLOCK]`, `[INFOBOX]`, etc.): Standalone content that can be safely removed
-  - New `[CODEBLOCK]` marker for `<syntaxhighlight>`, `<source>`, `<pre>` tags (block-level code)
-  - `[CODE]` marker now only applies to inline `<code>` tags
+- **Faster startup and repeated runs**: parsed data files, the Wikipedia category hierarchy (per language, 7-day expiry), and the multistream dump index are now cached under `~/.wp2txt/cache/`. Re-reading the index of a full dump used to take ~10 minutes; later runs now start in seconds
+- **`--ractor` (Ruby 4.0+)**: thread-based parallel processing — about 2× faster than sequential, with a smaller memory footprint than the default process-based mode (which remains the fastest at ~3×). On Ruby 3.x the option falls back to sequential processing
+- **Templates are expanded into readable text by default**: dates (`{{birth date|1990|5|15}}` → "May 15, 1990"), unit conversions (`{{convert|100|km|mi}}` → "100 km (62 mi)"), coordinates, language tags (`{{lang|ja|日本語}}` → 日本語), quotes, and 20+ more, plus common parser functions (`{{#if:}}`, `{{#switch:}}`) and magic words (`{{PAGENAME}}`). Turn off with `--no-expand-templates`
+- **Interrupted downloads resume**: a partial dump download is detected and, when the server supports it, only the remaining data is fetched. If the dump date has changed you are asked whether to resume, redownload, or keep the old file; files are validated before and after
+- **Corrupt dump files are detected before processing** instead of failing partway through, with the specific reason reported (file truncated, not a bz2 file, failed decompression test)
+- **Output files rotate at article boundaries**: an article is never split across two output files, and multi-byte characters are no longer corrupted at file boundaries. Verified against the full Japanese (1.49M articles) and English (24.2 GB) dumps
+- **Downloads can no longer hang indefinitely**: every network operation now has a timeout (several previously had none)
+- **Transient network errors retry automatically** (3 attempts with 2/4/8-second backoff) instead of failing the run, and failed category API requests are logged instead of silently returning nothing
+- **Running out of disk space raises a clear error** instead of leaving corrupt output behind
+- **Cache age is visible**: cache listings show the date and age of each cached dump and warn when it exceeds `dump_expiry_days` (default 30); `--update-cache` (`-U`) forces a refresh
+- **`--from-category`**: extract every article in a Wikipedia category via the Wikipedia API, with `--depth` for subcategory levels, `--dry-run` to preview article counts before downloading, and `--yes` for unattended runs
+- **`--config-init`**: writes a persistent configuration file to `~/.wp2txt/config.yml` (cache lifetimes, cache directory, output defaults). Command-line options still take precedence
+- **`--markers=none` is deprecated**: removing inline content (a formula mid-sentence) leaves broken text, so the option now warns and behaves like `--markers=all`. Use `--markers=math,code` to keep only specific marker types
+- **Markers are classified as inline or block**: inline markers (`[MATH]`, `[CODE]`, `[CHEM]`, `[IPA]`) stand in for content whose removal would break the sentence; block markers (`[TABLE]`, `[INFOBOX]`, …) replace standalone content. New `[CODEBLOCK]` marker for code blocks; `[CODE]` now covers only inline code
+- **Conflicting options are rejected with a clear message**: `--category-only`, `--summary-only`, and `--metadata-only` are mutually exclusive, and `--sections` / `--section-stats` cannot be combined with them
+- **Option fixes**: `--table`, `--multiline`, and `--pre` were missing from the CLI and are now available; `--ref` was accepted but had no effect — reference removal now respects it
+- `--ractor` combined with turbo mode now warns explicitly (the combination is unsupported)
+- **Security fixes**: file paths are no longer passed through a shell, so crafted path names cannot inject commands; TLS certificate verification is actually performed (it was previously disabled by an accept-everything callback); temporary files use unpredictable names instead of a fixed name in the working directory
+- **Performance**: text cleanup is faster (fewer passes over each article), and buffer sizes adapt to the memory actually available on the machine
+- **Ruby API**: articles can be extracted from a multistream dump in parallel (`extract_articles_parallel`, `each_article_parallel`), grouped by stream to avoid decompressing the same block twice
 
 ## [2.0.0] - 2026-01-08
 
 ### Added
 
-- **Auto-download mode**: New `--lang` option automatically downloads Wikipedia dumps:
-  - `wp2txt --lang=ja -o ./output` downloads and processes Japanese Wikipedia
-  - Downloads cached to `~/.wp2txt/cache/` for reuse
-  - Supports any Wikipedia language code (en, ja, de, fr, zh, etc.)
-
-- **Article extraction**: New `--articles` option extracts specific articles by title:
-  - `wp2txt --lang=en --articles="Tokyo,Kyoto,Osaka" -o ./articles`
-  - Only downloads index + needed data streams (efficient partial download)
-  - O(1) hash lookup for article search
-
-- **Cache management**: New options to manage downloaded dumps:
-  - `--cache-status` - Show cache status for all languages
-  - `--cache-clear` - Clear all cache
-  - `--cache-clear --lang=ja` - Clear cache for specific language
-  - `--cache-dir` - Custom cache directory
-
-- **Content type markers**: New `--markers` option marks special content:
-  - Supported types: `[MATH]`, `[CODE]`, `[CHEM]`, `[TABLE]`, `[SCORE]`, `[TIMELINE]`, `[GRAPH]`, `[IPA]`, `[INFOBOX]`, `[NAVBOX]`, `[GALLERY]`, `[SIDEBAR]`, `[MAPFRAME]`, `[IMAGEMAP]`, `[REFERENCES]`
-  - `--markers=all` (default) - Enable all markers
-  - `--markers=none` - Disable markers (content removed)
-  - `--markers=math,code` - Enable specific markers only
-
-- **Citation extraction**: New `--extract-citations` (`-C`) option for formatted bibliography output:
-  - Extracts author, title, and year from `{{cite book}}`, `{{cite web}}`, `{{Citation}}` templates
-  - Formats citations as "Author. \"Title\". Year."
-  - Available via CLI (`--extract-citations`) and Ruby API (`extract_citations: true`)
-
-- **Multistream support**: New classes for efficient Wikipedia dump processing:
-  - `MultistreamIndex` - Parse multistream index files
-  - `MultistreamReader` - Extract articles from multistream dumps
-  - `DumpManager` - Download and cache dump files
-  - Enables targeted article extraction without downloading full dump
-
-- **Validation framework**: New rake tasks for validating Wikipedia dump processing:
-  - `testdata:prepare[lang,level]` - Download and cache test data
-  - `validate:run[lang,level]` - Run validation on cached data
-  - `validate:full[lang]` - Full dump validation
-
-- **HTML entity management**: Comprehensive entity support from authoritative sources:
-  - 2125 entities from WHATWG HTML specification (`html_entities.json`)
-  - Wikipedia-specific entities (`wikipedia_entities.json`): `&ratio;`, `&dash;`, `&nbso;`
-  - New script `scripts/fetch_html_entities.rb` to update from WHATWG
-  - Replaces hardcoded entity list with data-driven approach
-
-- **MediaWiki data auto-generation**: Magic words and namespace aliases fetched from all Wikipedia APIs:
-  - New script `scripts/fetch_mediawiki_data.rb` queries 350+ Wikipedia language editions
-  - Data stored in `lib/wp2txt/data/mediawiki_aliases.json`
-  - 176 redirect keywords, 231 category aliases, 313 file aliases
-  - Run `ruby scripts/fetch_mediawiki_data.rb` to update
-
-- **JSON/JSONL output format**: New `--format json` option outputs articles as JSONL (one JSON object per line) with `title`, `categories`, `text`, and `redirect` fields. Ideal for data pipelines and machine learning workflows.
-
-- **Streaming processing**: Complete rewrite of the processing architecture:
-  - No longer creates intermediate XML files
-  - Directly streams from bz2 compressed files
-  - Reduced disk I/O and storage requirements
-  - New `StreamProcessor` and `OutputWriter` classes for modular design
-
-- **Regex cache**: Dynamic regex patterns are now cached to avoid repeated compilation
-
-- **Multilingual category support**: Added support for category namespaces in 30+ languages (European, Cyrillic, Asian, Middle Eastern)
-
-- **Multilingual redirect support**: Added support for redirect keywords in 25+ languages
-
-- **Comprehensive test suite**: 395 tests covering:
-  - Unicode handling (CJK, Cyrillic, Arabic, emoji)
-  - Edge cases (deeply nested templates, malformed markup)
-  - Multilingual category and redirect extraction
-  - Text processing utilities
-  - Integration tests with real Wikipedia content
-
-- **SimpleCov integration**: Added code coverage reporting for development
-
-- **Ruby 4.0 compatibility**: Full support for Ruby 4.0
+- **`--lang`**: downloads the Wikipedia dump for any language code and processes it in one step (`wp2txt --lang=ja -o ./output`). Downloads are cached under `~/.wp2txt/cache/` for reuse
+- **`--articles`**: extract specific articles by title (`wp2txt --lang=en --articles="Tokyo,Kyoto,Osaka" -o ./articles`), downloading only the index and the data streams that contain them rather than the whole dump
+- **Cache management**: `--cache-status` (per-language overview), `--cache-clear` (all languages, or one with `--lang`), and `--cache-dir` (custom location)
+- **Content markers (`--markers`)**: special content is replaced by a marker instead of disappearing silently: `[MATH]`, `[CODE]`, `[CHEM]`, `[TABLE]`, `[SCORE]`, `[TIMELINE]`, `[GRAPH]`, `[IPA]`, `[INFOBOX]`, `[NAVBOX]`, `[GALLERY]`, `[SIDEBAR]`, `[MAPFRAME]`, `[IMAGEMAP]`, `[REFERENCES]`. `--markers=all` is the default; `--markers=math,code` keeps only the listed types
+- **Citation extraction (`--extract-citations`, `-C`)**: outputs a formatted bibliography ("Author. \"Title\". Year.") from `{{cite book}}`, `{{cite web}}`, and `{{Citation}}` templates. Also available from the Ruby API (`extract_citations: true`)
+- **JSON/JSONL output (`--format json`)**: one JSON object per line with `title`, `categories`, `text`, and `redirect` fields, for data pipelines and machine-learning workflows
+- **Multistream dumps are handled natively**, which is what makes targeted extraction work without downloading a full dump. Ruby API: `MultistreamIndex`, `MultistreamReader`, `DumpManager`
+- **HTML character entities are converted comprehensively**: 2,125 entities from the WHATWG HTML specification, plus Wikipedia-specific ones (`&ratio;`, `&dash;`, `&nbso;`)
+- **Redirect, category, and file keywords fetched from 350+ Wikipedia language editions** (176 redirect keywords, 231 category aliases, 313 file aliases), replacing a small hardcoded list
+- **Categories are recognized in 30+ languages** (European, Cyrillic, Asian, and Middle Eastern scripts) and **redirects in 25+**
+- **Fully streamed processing**: dumps are processed directly from the compressed file, with no intermediate XML files — far less disk space and I/O than before
+- **Ruby 4.0 support**
 
 ### Changed
 
-- **Performance improvements**:
-  - `format_wiki`: Reduced intermediate string allocations by using `gsub!` for in-place modifications
-  - `cleanup`: Optimized with `gsub!` to reduce memory allocations
-  - `remove_complex`, `make_reference`: Optimized with `gsub!`
-  - Category deduplication: Changed from O(n²) to O(n) by calling `uniq!` once at end instead of every line
-  - `correct_separator`: Uses `tr` instead of `gsub` for single character replacement
-  - `remove_inbetween`: Dynamic regex patterns are now cached
-
-- **BREAKING**: `REMOVE_HR_REGEX` now matches 4 or more hyphens (previously 3+) to align with MediaWiki specification where `----` is the minimum for horizontal rules
-
-- **`chrref_to_utf` function**: Completely rewritten to support all Unicode codepoints (U+0001 to U+10FFFF), including:
-  - Supplementary plane characters (emoji, CJK Extension B, etc.)
-  - Proper handling of invalid codepoints (returns empty string)
-
-- **`convert_characters` function**: Now uses `String#scrub` for safe handling of invalid UTF-8 sequences instead of calling `exit`
-
-- **`command_exist?` function**: Updated to use `IO.popen` instead of `open("| ...")` for Ruby 4.0 compatibility
+- **BREAKING: horizontal rules now require 4 or more hyphens** (previously 3), matching the MediaWiki specification — a line of `---` in an article is now treated as content, not as a rule
+- **Character references convert across the entire Unicode range** (U+0001–U+10FFFF), including emoji and supplementary-plane CJK; invalid codepoints become empty strings instead of garbage characters
+- **Faster text processing**: category deduplication went from quadratic to linear time, and string handling allocates far less memory on large articles
 
 ### Fixed
 
-- **Unicode BMP limitation**: Fixed `chrref_to_utf` to correctly convert character references beyond the Basic Multilingual Plane (U+FFFF). Previously, emoji like `&#x1F600;` would produce invalid characters.
-
-- **Encoding error crash**: Fixed `convert_characters` which previously called `exit` on encoding errors, now gracefully handles invalid byte sequences using `scrub`
-
-- **Horizontal rule detection**: Fixed `REMOVE_HR_REGEX` to correctly match MediaWiki horizontal rules (4+ hyphens)
-
-- **Heading regex**: Fixed `IN_HEADING_REGEX` to allow trailing whitespace after closing equal signs
-
-- **Ruby 4.0 compatibility**: Fixed `open("| which cmd")` pattern which no longer works in Ruby 4.0
+- **Character references beyond U+FFFF now produce the right character**: `&#x1F600;` becomes 😀 instead of an invalid character
+- **Invalid UTF-8 in a dump no longer aborts the run**: bad byte sequences are scrubbed and processing continues (the tool previously exited mid-run)
+- **Headings with trailing whitespace** after the closing `==` are now recognized as headings
 
 ### Deprecated
 
-- **`--convert` / `-c` option**: No longer needed as streaming processing always converts
-- **`--del-interfile` / `-x` option**: No longer needed as intermediate files are no longer created
+- **`--convert` / `-c`**: no longer needed — streaming processing always converts
+- **`--del-interfile` / `-x`**: no longer needed — intermediate files are no longer created
 
 ### Removed
 
-- **Intermediate XML file creation**: The `Splitter` class no longer creates intermediate XML files; processing is now fully streamed
+- **Intermediate XML files**: processing no longer writes intermediate XML next to the output at any point
 
 ### Security
 

data/DEVELOPMENT.md

@@ -383,6 +383,14 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 2. Redirect keywords: `data/language_redirects.json`
 3. Scripts: `scripts/generate_language_data.rb`
 
+## Documentation
+
+User-facing documents (README, README_ja, CHANGELOG, docs/) describe what a user can
+observe and what they need to do — not how the code works internally or why it is
+designed the way it is. Keep implementation detail in this file and in code comments,
+and keep design rationale out of the public repository entirely. The MCP tool table in
+docs/INDEXES.md is checked against the actual server surface by spec/docs_sync_spec.rb.
+
 ## Code Style
 
 - Ruby 2.6+ compatibility

data/DEVELOPMENT_ja.md

@@ -383,6 +383,13 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 2. リダイレクトキーワード: `data/language_redirects.json`
 3. スクリプト: `scripts/generate_language_data.rb`
 
+## ドキュメンテーション
+
+利用者向け文書（README, README_ja, CHANGELOG, docs/）には、利用者が観測できることと
+利用者が取るべき行動だけを書く — コードが内部でどう動くか、なぜそう設計したかは書かない。
+実装の詳細は本ファイルとコードコメントに、設計の理由づけは公開リポジトリの外に置く。
+docs/INDEXES.md の MCP ツール表は spec/docs_sync_spec.rb が実際のサーバー表面と照合する。
+
 ## コードスタイル
 
 - Ruby 2.6+互換性

data/README.md

@@ -255,12 +255,12 @@ By default, citation templates are removed. Use `--extract-citations` to extract
 
 Supported: `{{cite book}}`, `{{cite web}}`, `{{cite news}}`, `{{cite journal}}`, `{{Citation}}`, etc.
 
-## Research Infrastructure (Indexes, Exhaustive Queries, MCP)
+## Offline Indexes, Queries, and the MCP Server
 
-Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned research
-database**: SQLite indexes over categories, section headings, redirects, and (optionally)
-the full article text, plus interlanguage links for cross-edition comparison — all
-queryable offline, exhaustively, and exposed to LLM agents via an MCP server.
+Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned database**:
+SQLite indexes over categories, section headings, redirects, and (optionally) the full
+article text, plus interlanguage links for cross-edition comparison — all queryable
+offline, and usable from an LLM client via an MCP server.
 
 ```console
 $ wp2txt --build-index --fulltext --lang=ja      # build the indexes
@@ -270,13 +270,12 @@ $ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
 $ wp2txt-mcp --lang=ja                           # stdio MCP server for LLM agents
 ```
 
-Unlike web/API access, these queries scan every article (a `0 matches` result is a
-verifiable absence claim for that dump version) and are reproducible: extractions record
-the dump version and query in a `.meta.json` sidecar.
+These queries scan every article — `0 matches` means the term is absent from that dump
+version — and extractions record the dump version and query in a `.meta.json` sidecar,
+so results can be reproduced later.
 
-**→ See the [Research Infrastructure Guide](docs/RESEARCH.md)** for index building,
-exhaustive queries, full-text search, interlanguage links, cross-language SQL, the full
-MCP tool list, and design principles.
+**→ See [docs/INDEXES.md](docs/INDEXES.md)** for index building, offline queries,
+full-text search, interlanguage links, cross-language SQL, and the full MCP tool list.
 
 ## Command Line Options
 
@@ -354,11 +353,11 @@ MCP tool list, and design principles.
 ### Research infrastructure options
 
     --build-index                    Build the metadata index (add --fulltext for FTS)
-    --find-articles / --search       Exhaustive offline queries (see docs/RESEARCH.md)
+    --find-articles / --search       Offline queries over a whole edition (see docs/INDEXES.md)
     --import-langlinks               Import interlanguage links (version-matched)
     --fts-optimize                   Optimize an existing full-text index
 
-See the [Research Infrastructure Guide](docs/RESEARCH.md) for details.
+See [docs/INDEXES.md](docs/INDEXES.md) for details.
 
 ## Configuration File
 

data/README_ja.md

@@ -333,9 +333,9 @@ defaults:
 
 コマンドラインオプションは設定ファイルの設定を上書きします。
 
-## 研究基盤（索引・悉皆クエリ・MCP）
+## オフライン索引・クエリ・MCPサーバー
 
-テキスト抽出に加えて、wp2txtはダンプを**ローカルな版固定研究データベース**に変換できます。
+テキスト抽出に加えて、wp2txtはダンプを**ローカルな版固定データベース**に変換できます。
 カテゴリ・節見出し・リダイレクト・（オプションで）記事全文のSQLite索引、言語版横断比較のための
 言語間リンク（langlinks）— すべてオフラインで悉皆的にクエリでき、MCPサーバー経由で
 LLMエージェントにも公開できます。
@@ -348,12 +348,12 @@ $ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
 $ wp2txt-mcp --lang=ja                           # LLMエージェント向けMCPサーバー
 ```
 
-web/API アクセスと異なり、これらのクエリは全記事を走査します（`0件` は当該ダンプ版に対する
-検証可能な不在の言明になります）。抽出結果には dump 版とクエリを記録した `.meta.json`
-サイドカーが付き、再現可能です。
+これらのクエリは全記事を走査します（`0件` は当該ダンプ版にその語が存在しないことを
+意味します）。抽出結果には dump 版とクエリを記録した `.meta.json` サイドカーが付き、
+あとから再現できます。
 
-**→ 詳細は [Research Infrastructure Guide](docs/RESEARCH.md)（英語）を参照**：
-索引構築、悉皆クエリ、全文検索、言語間リンク、言語版横断SQL、MCPツール一覧、設計原則。
+**→ 詳細は [docs/INDEXES.md](docs/INDEXES.md)（英語）を参照**：
+索引構築、オフラインクエリ、全文検索、言語間リンク、言語版横断SQL、MCPツール一覧。
 
 ## パフォーマンス
 

data/Rakefile

@@ -34,7 +34,7 @@ Rake::Task["build"].enhance([:normalize_permissions])
 
 # Paths that must never reach a published image. The image is built from the
 # working tree, so anything ignored locally (private notes, scratch files)
-# would otherwise ride along; 2.3.0's images shipped research-notes/ this way.
+# would otherwise ride along.
 IMAGE_FORBIDDEN_PATHS = %w[/wp2txt/research-notes /wp2txt/tmp /wp2txt/.git /wp2txt/CLAUDE.md /wp2txt/.claude].freeze
 
 desc "Verify a built image contains no private material (run before pushing)"

data/docs/{RESEARCH.md → INDEXES.md}

@@ -1,40 +1,31 @@
-# wp2txt Research Infrastructure Guide
+# Offline Indexes, Queries, and the MCP Server
 
-This guide covers the research-oriented layer of wp2txt: local indexes over Wikipedia
-dumps, exhaustive offline queries, full-text search, cross-language SQL, and the MCP
-server that exposes all of this to LLM agents.
+This guide covers wp2txt's index-based features: local indexes over Wikipedia dumps,
+offline queries across a whole edition, full-text search, interlanguage links,
+cross-language SQL, and the MCP server for connecting an LLM client.
 
 For plain-text extraction (the classic wp2txt), see the [README](../README.md).
 
-## Concept
-
-Web search and the Wikipedia API operate on ranked, paginated, ever-changing data. They
-can show that something *exists*, but they cannot make **exhaustive** claims ("342 of the
-11,486 film articles with a plot section mention X — and none of the others do"), and
-their answers change from day to day.
-
-wp2txt takes the opposite approach: build local indexes over an official dump file, so that
-every query is
-
-- **exhaustive** — it scans every article, not search-ranked results;
-- **version-pinned** — results are tied to one dump (e.g. `jawiki-20260701`) and
-  reproducible later;
-- **agent-operable** — the MCP server exposes the whole layer to LLM agents, with
-  guardrails and provenance records designed for autonomous use.
+Everything below runs against a downloaded dump file: queries cover every article of the
+edition rather than a page of search results, results are tied to one dump (e.g.
+`jawiki-20260701`) and can be reproduced later, and nothing goes over the network once
+the dump is downloaded. Typical uses: "which of the 1.5M articles have a plot section",
+"how many film articles mention X, and which ones don't", "how do the section structures
+of the same article differ between the English and Japanese editions".
 
 ## 1. Building the indexes
 
 ```console
-# Tier 1: metadata index (categories, section headings, redirects, category hierarchy)
+# Metadata index (categories, section headings, redirects, category hierarchy)
 $ wp2txt --build-index --lang=ja
 
-# Tier 1 + Tier 2: add an FTS5 full-text index over the cleaned article text
+# Metadata index + FTS5 full-text index over the cleaned article text
 $ wp2txt --build-index --fulltext --lang=ja
 ```
 
 The dump is downloaded automatically if needed and everything is cached under
 `~/.wp2txt/cache/`. Ballpark figures (Apple Silicon laptop): Japanese Wikipedia ~15 min /
-~1.7 GB for the metadata index, ~1.5 h / ~12 GB with full text; English roughly 4× the
+~1.7 GB for the metadata index, ~1.5 h / ~12 GB with full text; English roughly 2.5–3× the
 time, ~10 GB / ~12 GB respectively.
 
 The full-text tokenizer is selected per language: character trigrams for Japanese,
@@ -58,8 +49,8 @@ $ wp2txt --find-articles --in-category "Films" -D 2 -j json --limit 100 --lang=e
 $ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
 ```
 
-Search totals are exhaustive counts, so `0 matches` is a verifiable **absence claim** for
-that dump version — something ranked web search cannot provide.
+Search totals count every match in the edition, so `0 matches` means the term is absent
+from that dump version — a result you can state and re-verify later.
 
 ## 3. Interlanguage links (langlinks)
 
@@ -114,7 +105,7 @@ $ claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yo
 
 | Tool | Purpose |
 |------|---------|
-| `dump_info` | Dump identity, index tiers, corpus statistics, langlinks provenance |
+| `dump_info` | Dump identity, installed indexes, corpus statistics, langlinks provenance |
 | `get_article` / `get_sections` / `list_headings` / `get_categories` | Single-article access (redirect-aware) |
 | `find_articles` | Exhaustive filtered listing (category recursion, category AND / pattern match, section headings, title match) |
 | `category_tree` / `section_stats` | Scope exploration and heading-frequency discovery |
@@ -126,17 +117,19 @@ $ claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yo
 | `extract_corpus` | Filtered or explicit-title extraction to JSONL + reproducibility sidecar; optional RAG chunking |
 | `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` | Background jobs for large extractions |
 
-### Design principles
+### What happens when your assistant uses these tools
 
-- **Division of labor**: the tool does mechanical, exhaustive narrowing and counting;
-  semantic judgment is left to the LLM. The LLM never has to count.
-- **Context economy**: large results go to disk; the model receives a summary plus a
-  3-record sample, never the full corpus.
-- **Reproducibility**: every extraction and file-writing query records the dump version,
-  the query, and any alias sets in a `.meta.json` sidecar.
-- **Guardrails**: SQL is screened and executed read-only in a killable subprocess;
-  alias sets are re-verified server-side before saving; output paths are confined to the
-  server's output directory.
+- **Filtering and counting run over the whole dump**, and your assistant reads the result
+  rather than tallying articles itself.
+- **Large results are written to a file**; the reply carries a summary and a short sample.
+  Your corpus lands on disk intact instead of being paraphrased through the chat.
+- **Extractions are traceable.** Extractions and file-writing queries leave a `.meta.json`
+  next to the output recording the dump version, the query, and any alias sets used, so
+  you can reproduce or cite the result later.
+- **The tools cannot change your data.** Queries run read-only and are stopped after 30
+  seconds, saved alias sets are re-checked before being stored, and files can only be
+  written under the server's output directory — worth knowing if you plan to let an
+  assistant work unattended.
 
 ## 5. Cross-language SQL
 
@@ -157,12 +150,12 @@ query_sql(
 ```
 
 Attached databases appear as `{lang}_meta` (and `{lang}_fts` when that language has a
-full-text tier) and share the main database's schema. Language codes are validated and
-resolved server-side; user SQL can never contain ATTACH itself. The response records what
-was attached (dump names included), and flags date mismatches between editions.
+full-text index) and share the main database's schema. You pass language codes, not paths,
+and the SQL you write cannot attach anything itself. The response lists what was attached,
+including each edition's dump name, and flags it when the dates differ.
 
-To the best of our knowledge no other system offers version-pinned, cross-edition SQL over
-both metadata **and** article text, fully offline.
+This runs entirely offline against pinned dump versions, so a cross-edition comparison can
+be re-run later and produce the same numbers.
 
 ## 6. Large results, explicit sets, and reproducibility
 
@@ -182,26 +175,34 @@ extract_corpus(titles: ["東京物語", "羅生門", ...], content: "summary",
   redirect hop, and reports unmatched titles in `not_found` — closing the loop
   *SQL decides the set → the tool materializes it → the LLM reads it*.
 
-## 7. The alias discovery loop
-
-Section headings vary ("Plot" vs "Synopsis"; 「あらすじ」 vs 「ストーリー」). wp2txt ships
-no per-language dictionaries. Instead, agents discover aliases from the dump itself:
-
-1. `section_stats` — find the actual headings used in a scope
-2. LLM proposes synonym groups
-3. `section_cooccurrence` — verify mechanically (true synonyms almost never co-occur in
-   the same article; a high co-occurrence ratio is evidence *against* the hypothesis)
-4. `save_alias_set` — persist the verified groups, re-checked server-side, and recorded
-   in every extraction that uses them
-
-The bundled `discover_aliases` MCP prompt walks any agent through this protocol.
-
-## 8. Honest limitations
-
-- Queries operate on the **cleaned-text space**: content replaced by markers
-  (`[MATH]`, `[CODE]`, `[TABLE]`, …) is not searchable.
-- Trigram languages (ja/zh/ko) cannot match queries shorter than 3 characters;
-  word-based languages have no stemming (`run` ≠ `running`). Both are deliberate:
-  exact counts and absence claims require predictable matching.
-- Categories and links come from the dump itself, as written by editors — they inherit
-  Wikipedia's own inconsistencies, which is precisely what makes them worth studying.
+## 7. Section alias sets
+
+Section headings vary by article and by language ("Plot" vs "Synopsis"; 「あらすじ」 vs
+「ストーリー」), and wp2txt ships no per-language dictionaries. Three tools manage named
+groups of equivalent headings instead:
+
+- `section_stats` lists the headings actually used in a scope, with counts.
+- `section_cooccurrence` reports how often two headings appear in the same article.
+  Headings that mean the same thing rarely co-occur, so a high ratio is evidence that
+  they are *different* sections (「概要」 and 「あらすじ」 co-occur often — not synonyms).
+- `save_alias_set` stores a named group. The group is re-checked against co-occurrence
+  before saving; a failing group is not stored and the call returns `saved: false`
+  rather than an error. Pass `force` to override, and use `list_alias_sets` to see
+  what is stored.
+
+Queries then accept `alias_set: "name"` in place of a heading list, and extractions
+record the group's exact contents in their `.meta.json`. The bundled `discover_aliases`
+prompt walks an LLM client through building and saving a set.
+
+## 8. Limitations to keep in mind
+
+- Searches run over the **cleaned text**: content that extraction replaces with a marker
+  (`[MATH]`, `[CODE]`, `[TABLE]`, …) cannot be matched. A count is a count of the cleaned
+  text, not of the raw wikitext.
+- Japanese, Chinese, and Korean indexes cannot match queries shorter than 3 characters.
+  Word-based languages match exact forms only — `run` does not find `running`. Plan your
+  search terms accordingly, especially when you intend to report a zero result.
+- Categories and interlanguage links are read from the dump as editors wrote them.
+  Categories added by a template rather than written in the article text are not visible
+  to wp2txt, which can make a category look much smaller than it is on the website —
+  check against the article text if a count looks wrong.

data/lib/wp2txt/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Wp2txt
-  VERSION = "2.3.1"
+  VERSION = "2.3.2"
 end

data/spec/docs_sync_spec.rb

@@ -5,7 +5,7 @@ require "spec_helper"
 # The MCP tool surface and its public documentation drift apart easily
 # (the 2.2.0 README shipped with a tool table missing four tools). This spec
 # pins them together: every tool defined in bin/wp2txt-mcp must appear in the
-# docs/RESEARCH.md tool table, and the table must not list phantom tools.
+# docs/INDEXES.md tool table, and the table must not list phantom tools.
 RSpec.describe "documentation surface sync" do
   repo_root = File.expand_path("..", __dir__)
 
@@ -15,9 +15,9 @@ RSpec.describe "documentation surface sync" do
   end
 
   define_method(:documented_tools) do
-    doc = File.read(File.join(repo_root, "docs", "RESEARCH.md"))
+    doc = File.read(File.join(repo_root, "docs", "INDEXES.md"))
     table = doc[/^### Tools\n(.*?)\n\n/m, 1]
-    raise "Tools table not found in docs/RESEARCH.md" unless table
+    raise "Tools table not found in docs/INDEXES.md" unless table
 
     # Tool names live in the first column only (the purpose column may
     # backtick argument names like `attach`)
@@ -30,10 +30,32 @@ RSpec.describe "documentation surface sync" do
     expect(defined_tools.size).to be >= 15
   end
 
-  it "documents every MCP tool in docs/RESEARCH.md, with no phantom entries" do
+  it "documents every MCP tool in docs/INDEXES.md, with no phantom entries" do
     missing = defined_tools - documented_tools
     phantom = documented_tools - defined_tools
-    expect(missing).to be_empty, "tools not documented in docs/RESEARCH.md: #{missing.join(', ')}"
+    expect(missing).to be_empty, "tools not documented in docs/INDEXES.md: #{missing.join(', ')}"
     expect(phantom).to be_empty, "documented tools that do not exist: #{phantom.join(', ')}"
   end
+
+  # Tripwire: tracked files must not contain tokens listed in .private-doc-tokens,
+  # an untracked, machine-local file (one substring per line; # starts a comment).
+  # The file exists only on machines that maintain such a list; everywhere else
+  # (CI, other contributors) this example skips — loudly, so a silently dead
+  # check cannot be mistaken for a passing one.
+  it "keeps machine-local private tokens out of tracked files" do
+    token_file = File.join(repo_root, ".private-doc-tokens")
+    skip "SKIPPED: no .private-doc-tokens on this machine — tripwire not checked" unless File.exist?(token_file)
+
+    tokens = File.readlines(token_file, encoding: "UTF-8")
+                 .map(&:strip).reject { |t| t.empty? || t.start_with?("#") }
+    tracked = `git -C #{repo_root} ls-files -z`.split("\x0")
+    hits = tracked.flat_map do |f|
+      path = File.join(repo_root, f)
+      next [] unless File.file?(path)
+
+      content = File.read(path, encoding: "BINARY")
+      tokens.filter_map { |t| "#{f}: #{t}" if content.include?(t.b) }
+    end
+    expect(hits).to be_empty, "private tokens found in tracked files:\n  #{hits.join("\n  ")}"
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: wp2txt
 version: !ruby/object:Gem::Version
-  version: 2.3.1
+  version: 2.3.2
 platform: ruby
 authors:
 - Yoichiro Hasebe
@@ -217,7 +217,7 @@ files:
 - Rakefile
 - bin/wp2txt
 - bin/wp2txt-mcp
-- docs/RESEARCH.md
+- docs/INDEXES.md
 - image/wp2txt-logo.svg
 - image/wp2txt.svg
 - lib/wp2txt.rb