wp2txt 2.3.0 → 2.3.2

data/README.md

@@ -84,7 +84,7 @@ docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 
 The `wp2txt` command is available inside the container. Use `/data` for input/output files.
 
-Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`); Docker Hub (`yohasebe/wp2txt`) is maintained as a mirror.
+Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`). The former Docker Hub repository (`yohasebe/wp2txt`) is no longer updated as of 2.3.1.
 
 **MCP server (no Ruby required on the host):**
 
@@ -255,12 +255,12 @@ By default, citation templates are removed. Use `--extract-citations` to extract
 
 Supported: `{{cite book}}`, `{{cite web}}`, `{{cite news}}`, `{{cite journal}}`, `{{Citation}}`, etc.
 
-## Research Infrastructure (Indexes, Exhaustive Queries, MCP)
+## Offline Indexes, Queries, and the MCP Server
 
-Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned research
-database**: SQLite indexes over categories, section headings, redirects, and (optionally)
-the full article text, plus interlanguage links for cross-edition comparison — all
-queryable offline, exhaustively, and exposed to LLM agents via an MCP server.
+Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned database**:
+SQLite indexes over categories, section headings, redirects, and (optionally) the full
+article text, plus interlanguage links for cross-edition comparison — all queryable
+offline, and usable from an LLM client via an MCP server.
 
 ```console
 $ wp2txt --build-index --fulltext --lang=ja      # build the indexes
@@ -270,13 +270,12 @@ $ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
 $ wp2txt-mcp --lang=ja                           # stdio MCP server for LLM agents
 ```
 
-Unlike web/API access, these queries scan every article (a `0 matches` result is a
-verifiable absence claim for that dump version) and are reproducible: extractions record
-the dump version and query in a `.meta.json` sidecar.
+These queries scan every article — `0 matches` means the term is absent from that dump
+version — and extractions record the dump version and query in a `.meta.json` sidecar,
+so results can be reproduced later.
 
-**→ See the [Research Infrastructure Guide](docs/RESEARCH.md)** for index building,
-exhaustive queries, full-text search, interlanguage links, cross-language SQL, the full
-MCP tool list, and design principles.
+**→ See [docs/INDEXES.md](docs/INDEXES.md)** for index building, offline queries,
+full-text search, interlanguage links, cross-language SQL, and the full MCP tool list.
 
 ## Command Line Options
 
@@ -354,11 +353,11 @@ MCP tool list, and design principles.
 ### Research infrastructure options
 
     --build-index                    Build the metadata index (add --fulltext for FTS)
-    --find-articles / --search       Exhaustive offline queries (see docs/RESEARCH.md)
+    --find-articles / --search       Offline queries over a whole edition (see docs/INDEXES.md)
     --import-langlinks               Import interlanguage links (version-matched)
     --fts-optimize                   Optimize an existing full-text index
 
-See the [Research Infrastructure Guide](docs/RESEARCH.md) for details.
+See [docs/INDEXES.md](docs/INDEXES.md) for details.
 
 ## Configuration File
 

data/README_ja.md

@@ -34,6 +34,8 @@ WP2TXTはWikipediaダンプファイルからプレーンテキストとカテ
 - **多言語対応** - 350以上のWikipedia言語でカテゴリ・リダイレクトを検出
 - **ストリーミング処理** - 中間ファイルなしで大規模ダンプを処理
 - **JSON出力** - データパイプライン向けの機械可読JSONL形式
+- **オフライン研究索引** - カテゴリ・節見出し・全文に対する悉皆的・版固定クエリ、言語間リンクによる言語版横断SQL
+- **MCPサーバー** - ローカルダンプをLLMエージェント（Claude、ChatGPT、Gemini、ローカルモデル）に公開
 
 ## ユースケース
 
@@ -43,6 +45,8 @@ wp2txtは以下の用途に適しています：
 - トピック領域を横断した比較言語研究
 - NLPタスク向けのメタデータ付きWikipediaテキスト抽出
 - 並行カテゴリ構造を利用した対照言語研究
+- 版固定のRAG知識ベース・LLM評価データセット
+- web検索では原理的に不可能な悉皆的言明（「このカテゴリのどの記事にもXへの言及がない」）
 
 ## データアクセス
 
@@ -71,11 +75,13 @@ Windows：[Bzip2 for Windows](http://gnuwin32.sourceforge.net/packages/bzip2.htm
 ### Docker（代替方法）
 
 ```shell
-docker run -it -v /path/to/localdata:/data yohasebe/wp2txt
+docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 ```
 
 `wp2txt`コマンドはコンテナ内で使用可能です。入出力には`/data`ディレクトリを使用してください。
 
+イメージはGitHub Container Registry（`ghcr.io/yohasebe/wp2txt`）で公開されています。旧 Docker Hub リポジトリ（`yohasebe/wp2txt`）は 2.3.1 以降更新されません。
+
 ## 基本的な使い方
 
 ### 自動ダウンロードと処理（推奨）
@@ -327,6 +333,28 @@ defaults:
 
 コマンドラインオプションは設定ファイルの設定を上書きします。
 
+## オフライン索引・クエリ・MCPサーバー
+
+テキスト抽出に加えて、wp2txtはダンプを**ローカルな版固定データベース**に変換できます。
+カテゴリ・節見出し・リダイレクト・（オプションで）記事全文のSQLite索引、言語版横断比較のための
+言語間リンク（langlinks）— すべてオフラインで悉皆的にクエリでき、MCPサーバー経由で
+LLMエージェントにも公開できます。
+
+```console
+$ wp2txt --build-index --fulltext --lang=ja      # 索引の構築
+$ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
+$ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
+$ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
+$ wp2txt-mcp --lang=ja                           # LLMエージェント向けMCPサーバー
+```
+
+これらのクエリは全記事を走査します（`0件` は当該ダンプ版にその語が存在しないことを
+意味します）。抽出結果には dump 版とクエリを記録した `.meta.json` サイドカーが付き、
+あとから再現できます。
+
+**→ 詳細は [docs/INDEXES.md](docs/INDEXES.md)（英語）を参照**：
+索引構築、オフラインクエリ、全文検索、言語間リンク、言語版横断SQL、MCPツール一覧。
+
 ## パフォーマンス
 
 MacBook Air M4でのベンチマーク結果（7並列プロセス、ターボモード、ダウンロード時間除く）:
@@ -347,6 +375,10 @@ MacBook Air M4でのベンチマーク結果（7並列プロセス、ターボ
 
 詳細なリリースノートは[CHANGELOG.md](CHANGELOG.md)を参照してください。
 
+**v2.3.0（2026年7月）**: 言語間リンク取り込み、言語版横断SQL（multi-dump ATTACH）、明示タイトル集合の抽出、SQL結果のファイル出力と再現性サイドカー、GHCRイメージ公開。
+
+**v2.2.0（2026年7月）**: オフラインメタデータ索引、FTS5全文検索、MCPサーバー、query_sql、抽出ジョブ。
+
 **v2.1.0（2026年2月）**: SQLiteキャッシュ、Ractor並列処理（Ruby 4.0+）、テンプレート展開、コンテンツマーカー、Dockerイメージ更新。
 
 **v2.0.0（2026年1月）**: 自動ダウンロードモード、カテゴリベース抽出、タイトル指定抽出、JSON出力、ストリーミング処理、Ruby 4.0サポート。

data/Rakefile

@@ -32,16 +32,39 @@ Rake::Task["build"].enhance([:normalize_permissions])
 # Docker
 # =============================================================================
 
-desc "Push Docker images"
-task :push do
+# Paths that must never reach a published image. The image is built from the
+# working tree, so anything ignored locally (private notes, scratch files)
+# would otherwise ride along.
+IMAGE_FORBIDDEN_PATHS = %w[/wp2txt/research-notes /wp2txt/tmp /wp2txt/.git /wp2txt/CLAUDE.md /wp2txt/.claude].freeze
+
+desc "Verify a built image contains no private material (run before pushing)"
+task :verify_image, [:tag] do |_t, args|
+  tag = args[:tag] || "wp2txt-verify:local"
+  checks = IMAGE_FORBIDDEN_PATHS.map { |p| "test -e #{p} && echo LEAK:#{p}" }.join("; ")
+  out = `docker run --rm #{tag} sh -c '#{checks}; true' 2>&1`
+  leaks = out.lines.grep(/^LEAK:/).map(&:strip)
+  abort "Image #{tag} contains private paths:\n  #{leaks.join("\n  ")}" unless leaks.empty?
+
+  puts "OK: #{tag} contains none of #{IMAGE_FORBIDDEN_PATHS.join(', ')}"
+end
+
+desc "Build the image locally and verify it, without pushing"
+task :check_image do
+  sh "docker build -t wp2txt-verify:local ."
+  Rake::Task[:verify_image].invoke
+end
+
+desc "Build and push Docker images to GHCR (verifies a local build first)"
+task push: :check_image do
+  # Docker Hub was retired after 2.3.0; GHCR (ghcr.io/yohasebe/wp2txt) is the
+  # only registry. Requires `docker buildx use multiarch` and a ghcr.io login.
   sh <<-SCRIPT.strip_heredoc, { verbose: false }
     /bin/bash -xeu <<'BASH'
-      # docker buildx create --name mybuilder
-      # docker buildx use mybuilder
+      # docker buildx create --name multiarch
+      # docker buildx use multiarch
       # docker buildx inspect --bootstrap
       docker buildx build --platform linux/amd64,linux/arm64 \
         -t ghcr.io/yohasebe/wp2txt:#{Wp2txt::VERSION} -t ghcr.io/yohasebe/wp2txt:latest \
-        -t yohasebe/wp2txt:#{Wp2txt::VERSION} -t yohasebe/wp2txt:latest \
         . --push
     BASH
   SCRIPT

data/bin/wp2txt-mcp

@@ -254,7 +254,7 @@ end
 
 server.define_tool(
   name: "save_alias_set",
-  description: "Persist a named set of section-heading alias groups for this dump (e.g. [[\"あらすじ\",\"ストーリー\",\"物語\"]]). A server-side guardrail re-checks co-occurrence: a pair that appears in the same article more than 20% of the time (when both headings occur in at least 100 articles) is treated as distinct sections, not synonyms, and blocks the save with saved:false and the offending pairs listed — fix the groups, or pass force:true only if you verified them another way (e.g., by reading section contents with get_sections). The response echoes the saved groups and the exact criteria used. Saved sets are referenced via alias_set in find_articles / get_sections / extract_corpus and recorded in extraction metadata for reproducibility.",
+  description: "Persist a named set of section-heading alias groups for this dump (e.g. [[\"あらすじ\",\"ストーリー\",\"物語\"]]). A server-side guardrail re-checks co-occurrence: a pair that appears in the same article more than 20% of the time (when both headings occur in at least 100 articles) is treated as distinct sections, not synonyms, and blocks the save with saved:false and the offending pairs listed — fix the groups, or pass force:true only if you verified them another way (e.g., by reading section contents with get_sections). The response echoes the saved groups and the exact criteria used. Saved sets are referenced via alias_set in find_articles / get_sections / extract_corpus and recorded in extraction metadata for reproducibility. A refused save is signaled by saved:false in the RESULT (not an exception); verify persistence with list_alias_sets.",
   input_schema: {
     properties: {
       name: { type: "string" },

data/docs/{RESEARCH.md → INDEXES.md}

@@ -1,40 +1,31 @@
-# wp2txt Research Infrastructure Guide
+# Offline Indexes, Queries, and the MCP Server
 
-This guide covers the research-oriented layer of wp2txt: local indexes over Wikipedia
-dumps, exhaustive offline queries, full-text search, cross-language SQL, and the MCP
-server that exposes all of this to LLM agents.
+This guide covers wp2txt's index-based features: local indexes over Wikipedia dumps,
+offline queries across a whole edition, full-text search, interlanguage links,
+cross-language SQL, and the MCP server for connecting an LLM client.
 
 For plain-text extraction (the classic wp2txt), see the [README](../README.md).
 
-## Concept
-
-Web search and the Wikipedia API operate on ranked, paginated, ever-changing data. They
-can show that something *exists*, but they cannot make **exhaustive** claims ("342 of the
-11,486 film articles with a plot section mention X — and none of the others do"), and
-their answers change from day to day.
-
-wp2txt takes the opposite approach: build local indexes over an official dump file, so that
-every query is
-
-- **exhaustive** — it scans every article, not search-ranked results;
-- **version-pinned** — results are tied to one dump (e.g. `jawiki-20260701`) and
-  reproducible later;
-- **agent-operable** — the MCP server exposes the whole layer to LLM agents, with
-  guardrails and provenance records designed for autonomous use.
+Everything below runs against a downloaded dump file: queries cover every article of the
+edition rather than a page of search results, results are tied to one dump (e.g.
+`jawiki-20260701`) and can be reproduced later, and nothing goes over the network once
+the dump is downloaded. Typical uses: "which of the 1.5M articles have a plot section",
+"how many film articles mention X, and which ones don't", "how do the section structures
+of the same article differ between the English and Japanese editions".
 
 ## 1. Building the indexes
 
 ```console
-# Tier 1: metadata index (categories, section headings, redirects, category hierarchy)
+# Metadata index (categories, section headings, redirects, category hierarchy)
 $ wp2txt --build-index --lang=ja
 
-# Tier 1 + Tier 2: add an FTS5 full-text index over the cleaned article text
+# Metadata index + FTS5 full-text index over the cleaned article text
 $ wp2txt --build-index --fulltext --lang=ja
 ```
 
 The dump is downloaded automatically if needed and everything is cached under
 `~/.wp2txt/cache/`. Ballpark figures (Apple Silicon laptop): Japanese Wikipedia ~15 min /
-~1.7 GB for the metadata index, ~1.5 h / ~12 GB with full text; English roughly 4× the
+~1.7 GB for the metadata index, ~1.5 h / ~12 GB with full text; English roughly 2.5–3× the
 time, ~10 GB / ~12 GB respectively.
 
 The full-text tokenizer is selected per language: character trigrams for Japanese,
@@ -58,8 +49,8 @@ $ wp2txt --find-articles --in-category "Films" -D 2 -j json --limit 100 --lang=e
 $ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
 ```
 
-Search totals are exhaustive counts, so `0 matches` is a verifiable **absence claim** for
-that dump version — something ranked web search cannot provide.
+Search totals count every match in the edition, so `0 matches` means the term is absent
+from that dump version — a result you can state and re-verify later.
 
 ## 3. Interlanguage links (langlinks)
 
@@ -114,7 +105,7 @@ $ claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yo
 
 | Tool | Purpose |
 |------|---------|
-| `dump_info` | Dump identity, index tiers, corpus statistics, langlinks provenance |
+| `dump_info` | Dump identity, installed indexes, corpus statistics, langlinks provenance |
 | `get_article` / `get_sections` / `list_headings` / `get_categories` | Single-article access (redirect-aware) |
 | `find_articles` | Exhaustive filtered listing (category recursion, category AND / pattern match, section headings, title match) |
 | `category_tree` / `section_stats` | Scope exploration and heading-frequency discovery |
@@ -126,17 +117,19 @@ $ claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yo
 | `extract_corpus` | Filtered or explicit-title extraction to JSONL + reproducibility sidecar; optional RAG chunking |
 | `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` | Background jobs for large extractions |
 
-### Design principles
+### What happens when your assistant uses these tools
 
-- **Division of labor**: the tool does mechanical, exhaustive narrowing and counting;
-  semantic judgment is left to the LLM. The LLM never has to count.
-- **Context economy**: large results go to disk; the model receives a summary plus a
-  3-record sample, never the full corpus.
-- **Reproducibility**: every extraction and file-writing query records the dump version,
-  the query, and any alias sets in a `.meta.json` sidecar.
-- **Guardrails**: SQL is screened and executed read-only in a killable subprocess;
-  alias sets are re-verified server-side before saving; output paths are confined to the
-  server's output directory.
+- **Filtering and counting run over the whole dump**, and your assistant reads the result
+  rather than tallying articles itself.
+- **Large results are written to a file**; the reply carries a summary and a short sample.
+  Your corpus lands on disk intact instead of being paraphrased through the chat.
+- **Extractions are traceable.** Extractions and file-writing queries leave a `.meta.json`
+  next to the output recording the dump version, the query, and any alias sets used, so
+  you can reproduce or cite the result later.
+- **The tools cannot change your data.** Queries run read-only and are stopped after 30
+  seconds, saved alias sets are re-checked before being stored, and files can only be
+  written under the server's output directory — worth knowing if you plan to let an
+  assistant work unattended.
 
 ## 5. Cross-language SQL
 
@@ -157,12 +150,12 @@ query_sql(
 ```
 
 Attached databases appear as `{lang}_meta` (and `{lang}_fts` when that language has a
-full-text tier) and share the main database's schema. Language codes are validated and
-resolved server-side; user SQL can never contain ATTACH itself. The response records what
-was attached (dump names included), and flags date mismatches between editions.
+full-text index) and share the main database's schema. You pass language codes, not paths,
+and the SQL you write cannot attach anything itself. The response lists what was attached,
+including each edition's dump name, and flags it when the dates differ.
 
-To the best of our knowledge no other system offers version-pinned, cross-edition SQL over
-both metadata **and** article text, fully offline.
+This runs entirely offline against pinned dump versions, so a cross-edition comparison can
+be re-run later and produce the same numbers.
 
 ## 6. Large results, explicit sets, and reproducibility
 
@@ -182,26 +175,34 @@ extract_corpus(titles: ["東京物語", "羅生門", ...], content: "summary",
   redirect hop, and reports unmatched titles in `not_found` — closing the loop
   *SQL decides the set → the tool materializes it → the LLM reads it*.
 
-## 7. The alias discovery loop
-
-Section headings vary ("Plot" vs "Synopsis"; 「あらすじ」 vs 「ストーリー」). wp2txt ships
-no per-language dictionaries. Instead, agents discover aliases from the dump itself:
-
-1. `section_stats` — find the actual headings used in a scope
-2. LLM proposes synonym groups
-3. `section_cooccurrence` — verify mechanically (true synonyms almost never co-occur in
-   the same article; a high co-occurrence ratio is evidence *against* the hypothesis)
-4. `save_alias_set` — persist the verified groups, re-checked server-side, and recorded
-   in every extraction that uses them
-
-The bundled `discover_aliases` MCP prompt walks any agent through this protocol.
-
-## 8. Honest limitations
-
-- Queries operate on the **cleaned-text space**: content replaced by markers
-  (`[MATH]`, `[CODE]`, `[TABLE]`, …) is not searchable.
-- Trigram languages (ja/zh/ko) cannot match queries shorter than 3 characters;
-  word-based languages have no stemming (`run` ≠ `running`). Both are deliberate:
-  exact counts and absence claims require predictable matching.
-- Categories and links come from the dump itself, as written by editors — they inherit
-  Wikipedia's own inconsistencies, which is precisely what makes them worth studying.
+## 7. Section alias sets
+
+Section headings vary by article and by language ("Plot" vs "Synopsis"; 「あらすじ」 vs
+「ストーリー」), and wp2txt ships no per-language dictionaries. Three tools manage named
+groups of equivalent headings instead:
+
+- `section_stats` lists the headings actually used in a scope, with counts.
+- `section_cooccurrence` reports how often two headings appear in the same article.
+  Headings that mean the same thing rarely co-occur, so a high ratio is evidence that
+  they are *different* sections (「概要」 and 「あらすじ」 co-occur often — not synonyms).
+- `save_alias_set` stores a named group. The group is re-checked against co-occurrence
+  before saving; a failing group is not stored and the call returns `saved: false`
+  rather than an error. Pass `force` to override, and use `list_alias_sets` to see
+  what is stored.
+
+Queries then accept `alias_set: "name"` in place of a heading list, and extractions
+record the group's exact contents in their `.meta.json`. The bundled `discover_aliases`
+prompt walks an LLM client through building and saving a set.
+
+## 8. Limitations to keep in mind
+
+- Searches run over the **cleaned text**: content that extraction replaces with a marker
+  (`[MATH]`, `[CODE]`, `[TABLE]`, …) cannot be matched. A count is a count of the cleaned
+  text, not of the raw wikitext.
+- Japanese, Chinese, and Korean indexes cannot match queries shorter than 3 characters.
+  Word-based languages match exact forms only — `run` does not find `running`. Plan your
+  search terms accordingly, especially when you intend to report a zero result.
+- Categories and interlanguage links are read from the dump as editors wrote them.
+  Categories added by a template rather than written in the article text are not visible
+  to wp2txt, which can make a category look much smaller than it is on the website —
+  check against the article text if a count looks wrong.

data/lib/wp2txt/corpus.rb

@@ -482,6 +482,31 @@ module Wp2txt
     SQL_ROW_LIMIT = 200
     SQL_CELL_LIMIT = 2000
     SQL_TIMEOUT_SECONDS = 30
+
+    # Grace added to the child's own CPU limit: the parent's IO.select deadline
+    # should normally fire first; this is the fallback for when it cannot.
+    SQL_CHILD_CPU_GRACE = 5
+
+    # Self-imposed deadline for a query child, applied inside the fork.
+    #
+    # The parent kills the child on timeout, but a child ORPHANED by the
+    # parent's death (interrupted test run, closed terminal, crashed server)
+    # would otherwise spin forever: sqlite3 holds the GVL inside sqlite3_step,
+    # so Ruby's deferred signal handling never reaches a safe point and even
+    # SIGTERM is ignored — only SIGKILL or the kernel can stop it. A CPU
+    # rlimit is enforced by the kernel regardless of the GVL (SIGXCPU at the
+    # soft limit, SIGKILL at the hard one), so the child always dies on its own.
+    def self.apply_child_cpu_limit(timeout)
+      return unless Process.respond_to?(:setrlimit) && defined?(Process::RLIMIT_CPU)
+
+      seconds = timeout.to_f.ceil
+      Process.setrlimit(Process::RLIMIT_CPU,
+                        seconds + SQL_CHILD_CPU_GRACE,
+                        seconds + (SQL_CHILD_CPU_GRACE * 2))
+    rescue StandardError
+      # A platform without CPU rlimits keeps the previous behaviour (parent-only kill)
+      nil
+    end
     SQL_FORBIDDEN = /\b(ATTACH|DETACH|PRAGMA|INSERT|UPDATE|DELETE|DROP|CREATE|ALTER|REPLACE|VACUUM|REINDEX)\b/i
 
     # File-output mode (query_sql output_path:): hard row cap and per-cell
@@ -656,6 +681,7 @@ module Wp2txt
       fts_path = fts.db_path
       reader_io, writer_io = IO.pipe
       pid = Process.fork do
+        self.class.apply_child_cpu_limit(timeout)
         reader_io.close
         outcome = begin
           db = build_readonly_connection(attach_fts: attach_fts, fts_path: fts_path, attachments: attachments)
@@ -797,6 +823,7 @@ module Wp2txt
       fts_path = fts.db_path
       reader_io, writer_io = IO.pipe
       pid = Process.fork do
+        self.class.apply_child_cpu_limit(timeout)
         reader_io.close
         outcome = begin
           db = build_readonly_connection(attach_fts: attach_fts, fts_path: fts_path, attachments: attachments)

data/lib/wp2txt/regex.rb

@@ -210,6 +210,13 @@ module Wp2txt
   REMOVE_HR_REGEX = Regexp.new('^\s*\-{4,}\s*$')
   MAKE_REFERENCE_REGEX_A = Regexp.new('<br ?\/>', Regexp::IGNORECASE)
   MAKE_REFERENCE_REGEX_B = Regexp.new('<ref[^>]*\/>', Regexp::IGNORECASE)
+  # A reference carrying no content adds nothing in either mode
+  MAKE_REFERENCE_REGEX_EMPTY = Regexp.new('<ref[^>]*>\s*<\/ref>', Regexp::MULTILINE | Regexp::IGNORECASE)
+  # One whole reference span, used to normalize newlines inside it.
+  # The span must not contain another opening <ref: an unclosed <ref> would
+  # otherwise pair with a later </ref> and swallow the paragraphs in between.
+  MAKE_REFERENCE_REGEX_SPAN = Regexp.new('<ref[^>]*>(?:(?!<ref[\s>]).)*?<\/ref>',
+                                         Regexp::MULTILINE | Regexp::IGNORECASE)
   MAKE_REFERENCE_REGEX_C = Regexp.new('<ref[^>]*>', Regexp::IGNORECASE)
   MAKE_REFERENCE_REGEX_D = Regexp.new('<\/ref>', Regexp::IGNORECASE)
   FORMAT_REF_REGEX = Regexp.new('\[ref\](.*?)\[\/ref\]', Regexp::MULTILINE)

data/lib/wp2txt/text_processing.rb

@@ -308,6 +308,14 @@ module Wp2txt
     result = +str.to_s
     result.gsub!(MAKE_REFERENCE_REGEX_A, "\n")
     result.gsub!(MAKE_REFERENCE_REGEX_B, "")
+    # Element splitting (Article#parse) breaks paragraphs at newlines, so a
+    # reference written across lines would land in separate elements and its
+    # [ref]/[/ref] pair would never be visible to remove_ref at the same time.
+    # Drop empty references outright, and flatten the rest onto one line so
+    # multi-line references behave exactly like single-line ones (this is also
+    # what makes --extract-citations work on multi-line cite templates).
+    result.gsub!(MAKE_REFERENCE_REGEX_EMPTY, "")
+    result.gsub!(MAKE_REFERENCE_REGEX_SPAN) { |span| span.gsub(/\s*\n\s*/, " ") }
     result.gsub!(MAKE_REFERENCE_REGEX_C, "[ref]")
     result.gsub!(MAKE_REFERENCE_REGEX_D, "[/ref]")
     result

data/lib/wp2txt/utils.rb

@@ -415,11 +415,24 @@ module Wp2txt
     result.strip
   end
 
+  # Reference markers are already in [ref] form when external links are
+  # processed, and the scanner would strip their brackets (making remove_ref
+  # fail). Hide them behind placeholders for the duration of the scan.
+  # NOTE: do not special-case them inside the block instead — returning
+  # "[ref]" from the block makes process_nested_single_pass re-detect the same
+  # spot forever, burning MAX_NESTING_ITERATIONS and leaving the whole string
+  # unprocessed (measured: 478x slower, no links processed).
+  REF_OPEN_PLACEHOLDER  = "«REFOPEN»"
+  REF_CLOSE_PLACEHOLDER = "«REFCLOSE»"
+
   def process_external_links(str)
     # Early exit if no external links present
     return str unless str.include?("[")
 
-    process_nested_single_pass(str, "[", "]") do |contents|
+    protected_str = str.gsub("[ref]", REF_OPEN_PLACEHOLDER)
+                       .gsub("[/ref]", REF_CLOSE_PLACEHOLDER)
+
+    processed = process_nested_single_pass(protected_str, "[", "]") do |contents|
       if /\A\s.+\s\z/ =~ contents
         " (#{contents.strip}) "
       else
@@ -432,6 +445,9 @@ module Wp2txt
         end
       end
     end
+
+    processed.gsub(REF_OPEN_PLACEHOLDER, "[ref]")
+             .gsub(REF_CLOSE_PLACEHOLDER, "[/ref]")
   end
 
   #################### template processing ####################

data/lib/wp2txt/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Wp2txt
-  VERSION = "2.3.0"
+  VERSION = "2.3.2"
 end

data/spec/corpus_spec.rb

@@ -199,6 +199,31 @@ RSpec.describe Wp2txt::Corpus do
       expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - start).to be < 5
     end
 
+    it "gives the query child its own kernel-enforced CPU limit" do
+      # A child orphaned by the parent's death (interrupted test run, closed
+      # terminal) must still die on its own: sqlite3 holds the GVL inside
+      # sqlite3_step, so signals are never processed and only the kernel can
+      # stop it. Verified here by reading the limit the child actually gets.
+      skip "no CPU rlimits on this platform" unless Process.respond_to?(:setrlimit)
+
+      reader, writer = IO.pipe
+      pid = Process.fork do
+        reader.close
+        Wp2txt::Corpus.apply_child_cpu_limit(30)
+        writer.puts(Process.getrlimit(Process::RLIMIT_CPU).inspect)
+        writer.close
+        exit!(0)
+      end
+      writer.close
+      limits = reader.read
+      reader.close
+      Process.waitpid(pid)
+
+      # 30s query timeout => soft 35s / hard 40s of CPU time. Both exceed the
+      # parent's own wall-clock deadline, so no legitimate query is affected.
+      expect(limits.strip).to eq("[35, 40]")
+    end
+
     it "keeps serving queries after a timeout" do
       expect do
         @corpus.query_sql("WITH RECURSIVE c(x) AS (SELECT 1 UNION ALL SELECT x + 1 FROM c) SELECT COUNT(*) FROM c", timeout: 1)

data/spec/docs_sync_spec.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require "spec_helper"
+
+# The MCP tool surface and its public documentation drift apart easily
+# (the 2.2.0 README shipped with a tool table missing four tools). This spec
+# pins them together: every tool defined in bin/wp2txt-mcp must appear in the
+# docs/INDEXES.md tool table, and the table must not list phantom tools.
+RSpec.describe "documentation surface sync" do
+  repo_root = File.expand_path("..", __dir__)
+
+  define_method(:defined_tools) do
+    src = File.read(File.join(repo_root, "bin", "wp2txt-mcp"))
+    src.scan(/server\.define_tool\(\s*name:\s*"([a-z_]+)"/).flatten
+  end
+
+  define_method(:documented_tools) do
+    doc = File.read(File.join(repo_root, "docs", "INDEXES.md"))
+    table = doc[/^### Tools\n(.*?)\n\n/m, 1]
+    raise "Tools table not found in docs/INDEXES.md" unless table
+
+    # Tool names live in the first column only (the purpose column may
+    # backtick argument names like `attach`)
+    table.lines.filter_map { |line| line.split("|")[1] }
+         .flat_map { |cell| cell.scan(/`([a-z_]+)`/).flatten }
+         .uniq
+  end
+
+  it "defines a non-trivial number of MCP tools" do
+    expect(defined_tools.size).to be >= 15
+  end
+
+  it "documents every MCP tool in docs/INDEXES.md, with no phantom entries" do
+    missing = defined_tools - documented_tools
+    phantom = documented_tools - defined_tools
+    expect(missing).to be_empty, "tools not documented in docs/INDEXES.md: #{missing.join(', ')}"
+    expect(phantom).to be_empty, "documented tools that do not exist: #{phantom.join(', ')}"
+  end
+
+  # Tripwire: tracked files must not contain tokens listed in .private-doc-tokens,
+  # an untracked, machine-local file (one substring per line; # starts a comment).
+  # The file exists only on machines that maintain such a list; everywhere else
+  # (CI, other contributors) this example skips — loudly, so a silently dead
+  # check cannot be mistaken for a passing one.
+  it "keeps machine-local private tokens out of tracked files" do
+    token_file = File.join(repo_root, ".private-doc-tokens")
+    skip "SKIPPED: no .private-doc-tokens on this machine — tripwire not checked" unless File.exist?(token_file)
+
+    tokens = File.readlines(token_file, encoding: "UTF-8")
+                 .map(&:strip).reject { |t| t.empty? || t.start_with?("#") }
+    tracked = `git -C #{repo_root} ls-files -z`.split("\x0")
+    hits = tracked.flat_map do |f|
+      path = File.join(repo_root, f)
+      next [] unless File.file?(path)
+
+      content = File.read(path, encoding: "BINARY")
+      tokens.filter_map { |t| "#{f}: #{t}" if content.include?(t.b) }
+    end
+    expect(hits).to be_empty, "private tokens found in tracked files:\n  #{hits.join("\n  ")}"
+  end
+end

data/spec/spec_helper.rb

@@ -59,6 +59,28 @@ RSpec.configure do |config|
     mocks.verify_partial_doubles = true
   end
 
+  # Per-example wall-clock guard. The suite deliberately exercises runaway
+  # queries and forked workers; an interrupted or wedged example must not turn
+  # into a process spinning at 100% CPU for days (observed 2026-07-24).
+  # Slowest legitimate example is ~14s, so 120s is pure headroom. Override with
+  # WP2TXT_SPEC_TIMEOUT=0 to disable, or tag an example :no_timeout.
+  spec_timeout = (ENV["WP2TXT_SPEC_TIMEOUT"] || 120).to_i
+  if spec_timeout.positive?
+    require "timeout"
+    config.around(:each) do |example|
+      if example.metadata[:no_timeout]
+        example.run
+      else
+        begin
+          Timeout.timeout(spec_timeout) { example.run }
+        rescue Timeout::Error
+          raise "example exceeded the #{spec_timeout}s spec timeout (possible hang; " \
+                "set WP2TXT_SPEC_TIMEOUT=0 to disable this guard)"
+        end
+      end
+    end
+  end
+
   config.shared_context_metadata_behavior = :apply_to_host_groups
   config.filter_run_when_matching :focus
   config.example_status_persistence_file_path = "spec/examples.txt"