wp2txt 2.3.0 → 2.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 076aa3a7681c0e0e6e13948e5c66f6f9e15a0307507a944a78f29c0e7856eea0
-  data.tar.gz: f8f2d3a4efc82d1922f9ca2764b23bb348aa924e752439805ab61683ff66a737
+  metadata.gz: 4062c47480d0801e7a07d8e3bec7c65be8efeeda2eae30d398f9872810d321cd
+  data.tar.gz: 25bb3aa8a4d5baefc4616a8a4b754ec63e92d3e0b86b22ccc0ead2929a39739e
 SHA512:
-  metadata.gz: ae377878c20dde5a376d96efec19f84a74a20e66dd8f45ff5d41c438f519ced7cf5d4e6a6102ceee5c4b59333f0b1ca42167cf5e178abf803de868855b4efa6c
-  data.tar.gz: c23cb4598a7afeda092d75688be12e9e777874a4b9da289aeb1d97123b5cc64b3b56337722290fcc4352fde9031a856d4ba81393055ef8596fe8818d51e258ac
+  metadata.gz: ff61afbb989613e286af8bea784e4494c8f71b0dd52d24d3d038e72c5d77f9196d899039d9ff8c5e4b16461c044670d8cca6bc3feda55aa8cdbc31cbc30e012b
+  data.tar.gz: ddffdfb2452a3710d527bc7deeae52818cb9c2b4dedd433b2c7ecb06db74e6e389edf127976567a3269583531acf0aabc2ea9058d7422b5b675de9bae2e227fa

data/.dockerignore

@@ -6,11 +6,12 @@ pkg
 spec
 coverage
 tmp
+.bundle
+research-notes
 benchmark_results
 data/output_samples
 scripts
 .dockerignore
-.gitignore
 .solargraph.yml
 .rubocop.yml
 Gemfile.lock

data/CHANGELOG.md

@@ -5,6 +5,15 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.3.1] - 2026-08-12
+
+- **Container images no longer carry private files**: the image is built from the working tree with `git init && git add -A`, but the build context excluded `.gitignore`, so files ignored locally were tracked inside the container and shipped in the image — 2.3.0's images contained the maintainer's private research notes under `/wp2txt/research-notes`. The context now keeps `.gitignore` (so the container's file list matches a local `gem build`), explicitly excludes `research-notes` and `.bundle`, and the throwaway `.git` (a blob copy of every added file) is deleted after the build. `rake check_image` builds and inspects an image for private paths, and `rake push` refuses to publish without that check. The published gems were never affected; 2.3.0 images have been withdrawn
+- **Docker Hub retired**: images are published to GitHub Container Registry only (`ghcr.io/yohasebe/wp2txt`). The Docker Hub repository is no longer updated
+- **Fix: multi-line `<ref>` references broken by element splitting — also fixes `--extract-citations` on multi-line cite templates**: `Article#parse` splits elements at newlines, so any reference written across lines (very common for multi-line `{{cite …}}` templates) had its `[ref]` and `[/ref]` markers land in separate elements, invisible to `remove_ref`. The markers and raw reference markup leaked into the output, and — more importantly — `--extract-citations` silently never fired for multi-line cite templates, producing inconsistent output versus their single-line equivalents. `make_reference` now drops empty references outright and flattens the rest onto a single line before element splitting, so multi-line references behave exactly like single-line ones. Ordinary paragraph breaks outside references are unaffected
+- **Fix: `[ref]` markers destroyed by external-link processing**: `process_external_links` stripped the brackets of the `[ref]`/`[/ref]` markers produced by `make_reference` (their contents took the single-word branch), so `remove_ref` could no longer find them and tag names plus reference bodies leaked into extracted text — even with `--ref`, the kept markers came out broken. The markers are now hidden behind placeholders for the duration of the bracket scan and restored afterwards. Downstream corpora no longer contain `ref…/ref` residue, which broke tokenization/sentence splitting and mixed bibliographic text into body prose
+- **Runaway-query hardening**: a `query_sql` child process now sets its own kernel-enforced CPU limit (`RLIMIT_CPU`, the query timeout plus a small grace) in addition to the parent's wall-clock kill. Previously a child orphaned by the parent's death — an interrupted test run, a closed terminal, a crashed server — kept executing forever: sqlite3 holds the GVL inside `sqlite3_step`, so Ruby never reaches a signal-safe point and even SIGTERM is ignored. Observed in the wild as two processes spinning at 99% CPU for over five days. The CPU limit is strictly more permissive than the existing wall-clock deadline, so no query that would otherwise succeed is affected
+- **Test-suite hang guard**: each example now runs under a wall-clock timeout (120s default; `WP2TXT_SPEC_TIMEOUT=0` disables it, the `:no_timeout` tag exempts an example). The suite deliberately exercises runaway queries, so a wedged example must fail rather than spin
+
 ## [2.3.0] - 2026-07-24
 
 - **Documentation split**: README now focuses on text extraction; the research layer (indexes, exhaustive queries, full-text search, langlinks, cross-language SQL, MCP server) is documented in the new [Research Infrastructure Guide](docs/RESEARCH.md), including the complete MCP tool table

data/DEVELOPMENT.md

@@ -395,7 +395,8 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 Build and push Docker images:
 
 ```bash
-rake push  # Builds multi-arch and pushes to Docker Hub
+rake check_image  # Builds the image locally and verifies it carries no private files
+rake push        # Verifies, then builds multi-arch and pushes to GHCR
 ```
 
 ## Release Process

data/DEVELOPMENT_ja.md

@@ -395,7 +395,8 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 Dockerイメージのビルドとプッシュ：
 
 ```bash
-rake push  # マルチアーキテクチャでビルドしDocker Hubにプッシュ
+rake check_image  # ローカルでイメージをビルドし、私的ファイルの混入がないか検証
+rake push        # 検証したうえでマルチアーキテクチャでビルドしGHCRにプッシュ
 ```
 
 ## リリースプロセス

data/Dockerfile

@@ -7,7 +7,12 @@ WORKDIR /wp2txt
 COPY . ./
 RUN rm -f Gemfile.lock
 
-# Install dependencies (git is required by gemspec's `git ls-files`)
+# Install dependencies (git is required by gemspec's `git ls-files`).
+# The repository's .gitignore is copied in deliberately: `git add -A` must
+# honour it so the file list here matches a local `gem build` — without it,
+# ignored material (private notes, scratch files) would land in the image.
+# The throwaway .git is removed afterwards: it holds a blob copy of every
+# added file and is dead weight in the published image.
 RUN apk update && \
     apk upgrade && \
     apk add --no-cache \
@@ -17,6 +22,7 @@ RUN apk update && \
       build-base curl-dev wget && \
     git init && git add -A && \
     bundle install -j4 && \
+    rm -rf /wp2txt/.git && \
     apk del .build-packages
 
 # lbzip2 is not available as an Alpine package; build from source

data/README.md

@@ -84,7 +84,7 @@ docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 
 The `wp2txt` command is available inside the container. Use `/data` for input/output files.
 
-Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`); Docker Hub (`yohasebe/wp2txt`) is maintained as a mirror.
+Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`). The former Docker Hub repository (`yohasebe/wp2txt`) is no longer updated as of 2.3.1.
 
 **MCP server (no Ruby required on the host):**
 

data/README_ja.md

@@ -34,6 +34,8 @@ WP2TXTはWikipediaダンプファイルからプレーンテキストとカテ
 - **多言語対応** - 350以上のWikipedia言語でカテゴリ・リダイレクトを検出
 - **ストリーミング処理** - 中間ファイルなしで大規模ダンプを処理
 - **JSON出力** - データパイプライン向けの機械可読JSONL形式
+- **オフライン研究索引** - カテゴリ・節見出し・全文に対する悉皆的・版固定クエリ、言語間リンクによる言語版横断SQL
+- **MCPサーバー** - ローカルダンプをLLMエージェント（Claude、ChatGPT、Gemini、ローカルモデル）に公開
 
 ## ユースケース
 
@@ -43,6 +45,8 @@ wp2txtは以下の用途に適しています：
 - トピック領域を横断した比較言語研究
 - NLPタスク向けのメタデータ付きWikipediaテキスト抽出
 - 並行カテゴリ構造を利用した対照言語研究
+- 版固定のRAG知識ベース・LLM評価データセット
+- web検索では原理的に不可能な悉皆的言明（「このカテゴリのどの記事にもXへの言及がない」）
 
 ## データアクセス
 
@@ -71,11 +75,13 @@ Windows：[Bzip2 for Windows](http://gnuwin32.sourceforge.net/packages/bzip2.htm
 ### Docker（代替方法）
 
 ```shell
-docker run -it -v /path/to/localdata:/data yohasebe/wp2txt
+docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 ```
 
 `wp2txt`コマンドはコンテナ内で使用可能です。入出力には`/data`ディレクトリを使用してください。
 
+イメージはGitHub Container Registry（`ghcr.io/yohasebe/wp2txt`）で公開されています。旧 Docker Hub リポジトリ（`yohasebe/wp2txt`）は 2.3.1 以降更新されません。
+
 ## 基本的な使い方
 
 ### 自動ダウンロードと処理（推奨）
@@ -327,6 +333,28 @@ defaults:
 
 コマンドラインオプションは設定ファイルの設定を上書きします。
 
+## 研究基盤（索引・悉皆クエリ・MCP）
+
+テキスト抽出に加えて、wp2txtはダンプを**ローカルな版固定研究データベース**に変換できます。
+カテゴリ・節見出し・リダイレクト・（オプションで）記事全文のSQLite索引、言語版横断比較のための
+言語間リンク（langlinks）— すべてオフラインで悉皆的にクエリでき、MCPサーバー経由で
+LLMエージェントにも公開できます。
+
+```console
+$ wp2txt --build-index --fulltext --lang=ja      # 索引の構築
+$ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
+$ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
+$ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
+$ wp2txt-mcp --lang=ja                           # LLMエージェント向けMCPサーバー
+```
+
+web/API アクセスと異なり、これらのクエリは全記事を走査します（`0件` は当該ダンプ版に対する
+検証可能な不在の言明になります）。抽出結果には dump 版とクエリを記録した `.meta.json`
+サイドカーが付き、再現可能です。
+
+**→ 詳細は [Research Infrastructure Guide](docs/RESEARCH.md)（英語）を参照**：
+索引構築、悉皆クエリ、全文検索、言語間リンク、言語版横断SQL、MCPツール一覧、設計原則。
+
 ## パフォーマンス
 
 MacBook Air M4でのベンチマーク結果（7並列プロセス、ターボモード、ダウンロード時間除く）:
@@ -347,6 +375,10 @@ MacBook Air M4でのベンチマーク結果（7並列プロセス、ターボ
 
 詳細なリリースノートは[CHANGELOG.md](CHANGELOG.md)を参照してください。
 
+**v2.3.0（2026年7月）**: 言語間リンク取り込み、言語版横断SQL（multi-dump ATTACH）、明示タイトル集合の抽出、SQL結果のファイル出力と再現性サイドカー、GHCRイメージ公開。
+
+**v2.2.0（2026年7月）**: オフラインメタデータ索引、FTS5全文検索、MCPサーバー、query_sql、抽出ジョブ。
+
 **v2.1.0（2026年2月）**: SQLiteキャッシュ、Ractor並列処理（Ruby 4.0+）、テンプレート展開、コンテンツマーカー、Dockerイメージ更新。
 
 **v2.0.0（2026年1月）**: 自動ダウンロードモード、カテゴリベース抽出、タイトル指定抽出、JSON出力、ストリーミング処理、Ruby 4.0サポート。

data/Rakefile

@@ -32,16 +32,39 @@ Rake::Task["build"].enhance([:normalize_permissions])
 # Docker
 # =============================================================================
 
-desc "Push Docker images"
-task :push do
+# Paths that must never reach a published image. The image is built from the
+# working tree, so anything ignored locally (private notes, scratch files)
+# would otherwise ride along; 2.3.0's images shipped research-notes/ this way.
+IMAGE_FORBIDDEN_PATHS = %w[/wp2txt/research-notes /wp2txt/tmp /wp2txt/.git /wp2txt/CLAUDE.md /wp2txt/.claude].freeze
+
+desc "Verify a built image contains no private material (run before pushing)"
+task :verify_image, [:tag] do |_t, args|
+  tag = args[:tag] || "wp2txt-verify:local"
+  checks = IMAGE_FORBIDDEN_PATHS.map { |p| "test -e #{p} && echo LEAK:#{p}" }.join("; ")
+  out = `docker run --rm #{tag} sh -c '#{checks}; true' 2>&1`
+  leaks = out.lines.grep(/^LEAK:/).map(&:strip)
+  abort "Image #{tag} contains private paths:\n  #{leaks.join("\n  ")}" unless leaks.empty?
+
+  puts "OK: #{tag} contains none of #{IMAGE_FORBIDDEN_PATHS.join(', ')}"
+end
+
+desc "Build the image locally and verify it, without pushing"
+task :check_image do
+  sh "docker build -t wp2txt-verify:local ."
+  Rake::Task[:verify_image].invoke
+end
+
+desc "Build and push Docker images to GHCR (verifies a local build first)"
+task push: :check_image do
+  # Docker Hub was retired after 2.3.0; GHCR (ghcr.io/yohasebe/wp2txt) is the
+  # only registry. Requires `docker buildx use multiarch` and a ghcr.io login.
   sh <<-SCRIPT.strip_heredoc, { verbose: false }
     /bin/bash -xeu <<'BASH'
-      # docker buildx create --name mybuilder
-      # docker buildx use mybuilder
+      # docker buildx create --name multiarch
+      # docker buildx use multiarch
       # docker buildx inspect --bootstrap
       docker buildx build --platform linux/amd64,linux/arm64 \
         -t ghcr.io/yohasebe/wp2txt:#{Wp2txt::VERSION} -t ghcr.io/yohasebe/wp2txt:latest \
-        -t yohasebe/wp2txt:#{Wp2txt::VERSION} -t yohasebe/wp2txt:latest \
         . --push
     BASH
   SCRIPT

data/bin/wp2txt-mcp

@@ -254,7 +254,7 @@ end
 
 server.define_tool(
   name: "save_alias_set",
-  description: "Persist a named set of section-heading alias groups for this dump (e.g. [[\"あらすじ\",\"ストーリー\",\"物語\"]]). A server-side guardrail re-checks co-occurrence: a pair that appears in the same article more than 20% of the time (when both headings occur in at least 100 articles) is treated as distinct sections, not synonyms, and blocks the save with saved:false and the offending pairs listed — fix the groups, or pass force:true only if you verified them another way (e.g., by reading section contents with get_sections). The response echoes the saved groups and the exact criteria used. Saved sets are referenced via alias_set in find_articles / get_sections / extract_corpus and recorded in extraction metadata for reproducibility.",
+  description: "Persist a named set of section-heading alias groups for this dump (e.g. [[\"あらすじ\",\"ストーリー\",\"物語\"]]). A server-side guardrail re-checks co-occurrence: a pair that appears in the same article more than 20% of the time (when both headings occur in at least 100 articles) is treated as distinct sections, not synonyms, and blocks the save with saved:false and the offending pairs listed — fix the groups, or pass force:true only if you verified them another way (e.g., by reading section contents with get_sections). The response echoes the saved groups and the exact criteria used. Saved sets are referenced via alias_set in find_articles / get_sections / extract_corpus and recorded in extraction metadata for reproducibility. A refused save is signaled by saved:false in the RESULT (not an exception); verify persistence with list_alias_sets.",
   input_schema: {
     properties: {
       name: { type: "string" },

data/lib/wp2txt/corpus.rb

@@ -482,6 +482,31 @@ module Wp2txt
     SQL_ROW_LIMIT = 200
     SQL_CELL_LIMIT = 2000
     SQL_TIMEOUT_SECONDS = 30
+
+    # Grace added to the child's own CPU limit: the parent's IO.select deadline
+    # should normally fire first; this is the fallback for when it cannot.
+    SQL_CHILD_CPU_GRACE = 5
+
+    # Self-imposed deadline for a query child, applied inside the fork.
+    #
+    # The parent kills the child on timeout, but a child ORPHANED by the
+    # parent's death (interrupted test run, closed terminal, crashed server)
+    # would otherwise spin forever: sqlite3 holds the GVL inside sqlite3_step,
+    # so Ruby's deferred signal handling never reaches a safe point and even
+    # SIGTERM is ignored — only SIGKILL or the kernel can stop it. A CPU
+    # rlimit is enforced by the kernel regardless of the GVL (SIGXCPU at the
+    # soft limit, SIGKILL at the hard one), so the child always dies on its own.
+    def self.apply_child_cpu_limit(timeout)
+      return unless Process.respond_to?(:setrlimit) && defined?(Process::RLIMIT_CPU)
+
+      seconds = timeout.to_f.ceil
+      Process.setrlimit(Process::RLIMIT_CPU,
+                        seconds + SQL_CHILD_CPU_GRACE,
+                        seconds + (SQL_CHILD_CPU_GRACE * 2))
+    rescue StandardError
+      # A platform without CPU rlimits keeps the previous behaviour (parent-only kill)
+      nil
+    end
     SQL_FORBIDDEN = /\b(ATTACH|DETACH|PRAGMA|INSERT|UPDATE|DELETE|DROP|CREATE|ALTER|REPLACE|VACUUM|REINDEX)\b/i
 
     # File-output mode (query_sql output_path:): hard row cap and per-cell
@@ -656,6 +681,7 @@ module Wp2txt
       fts_path = fts.db_path
       reader_io, writer_io = IO.pipe
       pid = Process.fork do
+        self.class.apply_child_cpu_limit(timeout)
         reader_io.close
         outcome = begin
           db = build_readonly_connection(attach_fts: attach_fts, fts_path: fts_path, attachments: attachments)
@@ -797,6 +823,7 @@ module Wp2txt
       fts_path = fts.db_path
       reader_io, writer_io = IO.pipe
       pid = Process.fork do
+        self.class.apply_child_cpu_limit(timeout)
         reader_io.close
         outcome = begin
           db = build_readonly_connection(attach_fts: attach_fts, fts_path: fts_path, attachments: attachments)

data/lib/wp2txt/regex.rb

@@ -210,6 +210,13 @@ module Wp2txt
   REMOVE_HR_REGEX = Regexp.new('^\s*\-{4,}\s*$')
   MAKE_REFERENCE_REGEX_A = Regexp.new('<br ?\/>', Regexp::IGNORECASE)
   MAKE_REFERENCE_REGEX_B = Regexp.new('<ref[^>]*\/>', Regexp::IGNORECASE)
+  # A reference carrying no content adds nothing in either mode
+  MAKE_REFERENCE_REGEX_EMPTY = Regexp.new('<ref[^>]*>\s*<\/ref>', Regexp::MULTILINE | Regexp::IGNORECASE)
+  # One whole reference span, used to normalize newlines inside it.
+  # The span must not contain another opening <ref: an unclosed <ref> would
+  # otherwise pair with a later </ref> and swallow the paragraphs in between.
+  MAKE_REFERENCE_REGEX_SPAN = Regexp.new('<ref[^>]*>(?:(?!<ref[\s>]).)*?<\/ref>',
+                                         Regexp::MULTILINE | Regexp::IGNORECASE)
   MAKE_REFERENCE_REGEX_C = Regexp.new('<ref[^>]*>', Regexp::IGNORECASE)
   MAKE_REFERENCE_REGEX_D = Regexp.new('<\/ref>', Regexp::IGNORECASE)
   FORMAT_REF_REGEX = Regexp.new('\[ref\](.*?)\[\/ref\]', Regexp::MULTILINE)

data/lib/wp2txt/text_processing.rb

@@ -308,6 +308,14 @@ module Wp2txt
     result = +str.to_s
     result.gsub!(MAKE_REFERENCE_REGEX_A, "\n")
     result.gsub!(MAKE_REFERENCE_REGEX_B, "")
+    # Element splitting (Article#parse) breaks paragraphs at newlines, so a
+    # reference written across lines would land in separate elements and its
+    # [ref]/[/ref] pair would never be visible to remove_ref at the same time.
+    # Drop empty references outright, and flatten the rest onto one line so
+    # multi-line references behave exactly like single-line ones (this is also
+    # what makes --extract-citations work on multi-line cite templates).
+    result.gsub!(MAKE_REFERENCE_REGEX_EMPTY, "")
+    result.gsub!(MAKE_REFERENCE_REGEX_SPAN) { |span| span.gsub(/\s*\n\s*/, " ") }
     result.gsub!(MAKE_REFERENCE_REGEX_C, "[ref]")
     result.gsub!(MAKE_REFERENCE_REGEX_D, "[/ref]")
     result

data/lib/wp2txt/utils.rb

@@ -415,11 +415,24 @@ module Wp2txt
     result.strip
   end
 
+  # Reference markers are already in [ref] form when external links are
+  # processed, and the scanner would strip their brackets (making remove_ref
+  # fail). Hide them behind placeholders for the duration of the scan.
+  # NOTE: do not special-case them inside the block instead — returning
+  # "[ref]" from the block makes process_nested_single_pass re-detect the same
+  # spot forever, burning MAX_NESTING_ITERATIONS and leaving the whole string
+  # unprocessed (measured: 478x slower, no links processed).
+  REF_OPEN_PLACEHOLDER  = "«REFOPEN»"
+  REF_CLOSE_PLACEHOLDER = "«REFCLOSE»"
+
   def process_external_links(str)
     # Early exit if no external links present
     return str unless str.include?("[")
 
-    process_nested_single_pass(str, "[", "]") do |contents|
+    protected_str = str.gsub("[ref]", REF_OPEN_PLACEHOLDER)
+                       .gsub("[/ref]", REF_CLOSE_PLACEHOLDER)
+
+    processed = process_nested_single_pass(protected_str, "[", "]") do |contents|
       if /\A\s.+\s\z/ =~ contents
         " (#{contents.strip}) "
       else
@@ -432,6 +445,9 @@ module Wp2txt
         end
       end
     end
+
+    processed.gsub(REF_OPEN_PLACEHOLDER, "[ref]")
+             .gsub(REF_CLOSE_PLACEHOLDER, "[/ref]")
   end
 
   #################### template processing ####################

data/lib/wp2txt/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Wp2txt
-  VERSION = "2.3.0"
+  VERSION = "2.3.1"
 end

data/spec/corpus_spec.rb

@@ -199,6 +199,31 @@ RSpec.describe Wp2txt::Corpus do
       expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - start).to be < 5
     end
 
+    it "gives the query child its own kernel-enforced CPU limit" do
+      # A child orphaned by the parent's death (interrupted test run, closed
+      # terminal) must still die on its own: sqlite3 holds the GVL inside
+      # sqlite3_step, so signals are never processed and only the kernel can
+      # stop it. Verified here by reading the limit the child actually gets.
+      skip "no CPU rlimits on this platform" unless Process.respond_to?(:setrlimit)
+
+      reader, writer = IO.pipe
+      pid = Process.fork do
+        reader.close
+        Wp2txt::Corpus.apply_child_cpu_limit(30)
+        writer.puts(Process.getrlimit(Process::RLIMIT_CPU).inspect)
+        writer.close
+        exit!(0)
+      end
+      writer.close
+      limits = reader.read
+      reader.close
+      Process.waitpid(pid)
+
+      # 30s query timeout => soft 35s / hard 40s of CPU time. Both exceed the
+      # parent's own wall-clock deadline, so no legitimate query is affected.
+      expect(limits.strip).to eq("[35, 40]")
+    end
+
     it "keeps serving queries after a timeout" do
       expect do
         @corpus.query_sql("WITH RECURSIVE c(x) AS (SELECT 1 UNION ALL SELECT x + 1 FROM c) SELECT COUNT(*) FROM c", timeout: 1)

data/spec/docs_sync_spec.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+require "spec_helper"
+
+# The MCP tool surface and its public documentation drift apart easily
+# (the 2.2.0 README shipped with a tool table missing four tools). This spec
+# pins them together: every tool defined in bin/wp2txt-mcp must appear in the
+# docs/RESEARCH.md tool table, and the table must not list phantom tools.
+RSpec.describe "documentation surface sync" do
+  repo_root = File.expand_path("..", __dir__)
+
+  define_method(:defined_tools) do
+    src = File.read(File.join(repo_root, "bin", "wp2txt-mcp"))
+    src.scan(/server\.define_tool\(\s*name:\s*"([a-z_]+)"/).flatten
+  end
+
+  define_method(:documented_tools) do
+    doc = File.read(File.join(repo_root, "docs", "RESEARCH.md"))
+    table = doc[/^### Tools\n(.*?)\n\n/m, 1]
+    raise "Tools table not found in docs/RESEARCH.md" unless table
+
+    # Tool names live in the first column only (the purpose column may
+    # backtick argument names like `attach`)
+    table.lines.filter_map { |line| line.split("|")[1] }
+         .flat_map { |cell| cell.scan(/`([a-z_]+)`/).flatten }
+         .uniq
+  end
+
+  it "defines a non-trivial number of MCP tools" do
+    expect(defined_tools.size).to be >= 15
+  end
+
+  it "documents every MCP tool in docs/RESEARCH.md, with no phantom entries" do
+    missing = defined_tools - documented_tools
+    phantom = documented_tools - defined_tools
+    expect(missing).to be_empty, "tools not documented in docs/RESEARCH.md: #{missing.join(', ')}"
+    expect(phantom).to be_empty, "documented tools that do not exist: #{phantom.join(', ')}"
+  end
+end

data/spec/spec_helper.rb

@@ -59,6 +59,28 @@ RSpec.configure do |config|
     mocks.verify_partial_doubles = true
   end
 
+  # Per-example wall-clock guard. The suite deliberately exercises runaway
+  # queries and forked workers; an interrupted or wedged example must not turn
+  # into a process spinning at 100% CPU for days (observed 2026-07-24).
+  # Slowest legitimate example is ~14s, so 120s is pure headroom. Override with
+  # WP2TXT_SPEC_TIMEOUT=0 to disable, or tag an example :no_timeout.
+  spec_timeout = (ENV["WP2TXT_SPEC_TIMEOUT"] || 120).to_i
+  if spec_timeout.positive?
+    require "timeout"
+    config.around(:each) do |example|
+      if example.metadata[:no_timeout]
+        example.run
+      else
+        begin
+          Timeout.timeout(spec_timeout) { example.run }
+        rescue Timeout::Error
+          raise "example exceeded the #{spec_timeout}s spec timeout (possible hang; " \
+                "set WP2TXT_SPEC_TIMEOUT=0 to disable this guard)"
+        end
+      end
+    end
+  end
+
   config.shared_context_metadata_behavior = :apply_to_host_groups
   config.filter_run_when_matching :focus
   config.example_status_persistence_file_path = "spec/examples.txt"

data/spec/utils_spec.rb

@@ -59,6 +59,135 @@ RSpec.describe "Wp2txt Utils" do
     end
   end
 
+  # Regression: process_external_links used to strip the brackets of the
+  # [ref]/[/ref] markers (contents "ref" / "/ref" take the parts.size == 1
+  # branch), so remove_ref could no longer find them and the tag names plus
+  # reference body leaked into the extracted text. These tests exercise the
+  # composed make_reference -> format_wiki path, since testing remove_ref in
+  # isolation passes even with the bug.
+  describe "reference markers through format_wiki" do
+    it "removes <ref>...</ref> including the body" do
+      input = "Paris was founded.<ref>Patrick Boucheron, France in the World (2019) pp 81-86.</ref> The city grew."
+      result = format_wiki(make_reference(input))
+      expect(result).to eq "Paris was founded. The city grew."
+    end
+
+    it "removes named <ref name=\"x\">...</ref> including the body" do
+      input = "Fine dining.<ref name=\"lemonde\">Le Monde, 2 February 2015</ref> Paris has."
+      result = format_wiki(make_reference(input))
+      expect(result).to eq "Fine dining. Paris has."
+    end
+
+    it "removes self-closing <ref name=\"x\"/>" do
+      input = "An asteroid,<ref name=\"x\"/> and a building."
+      result = format_wiki(make_reference(input))
+      expect(result).to eq "An asteroid, and a building."
+    end
+
+    it "removes consecutive <ref>A</ref><ref>B</ref> without joining tag names" do
+      input = "Text<ref>A</ref><ref>B</ref> more."
+      result = format_wiki(make_reference(input))
+      expect(result).to eq "Text more."
+    end
+
+    it "keeps [ref]...[/ref] markers when config[:ref] is true" do
+      input = "Paris was founded.<ref>Patrick Boucheron, France in the World (2019) pp 81-86.</ref> The city grew."
+      result = format_wiki(make_reference(input), ref: true)
+      expect(result).to eq "Paris was founded.[ref]Patrick Boucheron, France in the World (2019) pp 81-86.[/ref] The city grew."
+    end
+
+    # Guards against the rejected fix of returning "[ref]" from the scanner
+    # block: that makes process_nested_single_pass re-detect the same spot
+    # until MAX_NESTING_ITERATIONS, leaving external links unprocessed.
+    it "still processes external links outside [ref] markers" do
+      input = "Claim.<ref>See [http://example.com the site] for detail.</ref> Next [http://foo.com Foo] end."
+      expect(format_wiki(make_reference(input))).to eq "Claim. Next Foo end."
+      expect(format_wiki(make_reference(input), ref: true)).to eq "Claim.[ref]See the site for detail.[/ref] Next Foo end."
+    end
+  end
+
+  # Regression: element splitting (Article#parse) breaks paragraphs at
+  # newlines, so a reference written across lines landed in separate elements
+  # with [ref] and [/ref] never visible to remove_ref at the same time. This
+  # also meant --extract-citations never fired for multi-line cite templates.
+  # These tests go through Article.new -> format_wiki per element, since
+  # passing a string to format_wiki directly skips element splitting and does
+  # not reproduce the bug.
+  describe "multi-line references through element splitting" do
+    def render_elements(wikitext, config = {})
+      Wp2txt::Article.new(wikitext).elements.map { |e| format_wiki(e[1], config) }.join
+    end
+
+    it "removes an empty reference spanning a blank line" do
+      result = render_elements("...in the Super League.<ref>\n\n</ref> In 2006, Catalans Dragons became...")
+      expect(result).not_to include("[ref]")
+      expect(result).not_to include("[/ref]")
+      expect(result).to include("Super League. In 2006,")
+    end
+
+    it "removes a named empty reference spanning a blank line" do
+      result = render_elements("Claim.<ref name=\"x\">\n\n</ref> Next.")
+      expect(result).not_to include("[ref]")
+      expect(result).not_to include("[/ref]")
+      expect(result).to include("Claim. Next.")
+    end
+
+    it "removes a non-empty reference spanning a blank line" do
+      result = render_elements("Claim.<ref>Author\n\nPublisher</ref> Next.")
+      expect(result).not_to include("[ref]")
+      expect(result).not_to include("[/ref]")
+      expect(result).to include("Claim. Next.")
+    end
+
+    it "removes a multi-line cite template reference (with and without leading space)" do
+      ["Claim.<ref> {{cite book\n |title=Foo\n |year=2019}}</ref> Next.",
+       "Claim.<ref>{{cite book\n|title=Foo\n|year=2019}}</ref> Next."].each do |input|
+        result = render_elements(input)
+        expect(result).not_to include("[ref]")
+        expect(result).not_to include("[/ref]")
+        expect(result).not_to include("cite book")
+        expect(result).to include("Claim. Next.")
+      end
+    end
+
+    # The core of this fix: before, only the multi-line form leaked raw
+    # markup, so --extract-citations silently did nothing for it.
+    it "extracts citations identically from single-line and multi-line cite templates" do
+      config = { extract_citations: true, ref: true }
+      single = render_elements("Claim.<ref>{{cite book|title=Foo|year=2019}}</ref> Next.", config)
+      multi  = render_elements("Claim.<ref>{{cite book\n|title=Foo\n|year=2019}}</ref> Next.", config)
+      expect(multi).to eq single
+      expect(single).to include("[ref]\"Foo\". 2019.[/ref]")
+    end
+
+    # Control: newlines outside references must still split paragraphs.
+    it "still splits ordinary paragraph boundaries at blank lines" do
+      elements = Wp2txt::Article.new("First para.\n\nSecond para.").elements
+      paragraphs = elements.select { |e| e[0] == :mw_paragraph }
+      expect(paragraphs.size).to eq 2
+      expect(paragraphs[0][1]).to include("First para.")
+      expect(paragraphs[1][1]).to include("Second para.")
+    end
+
+    # An unclosed <ref> must not pair with a later </ref> across paragraphs;
+    # the body text stays (a floating [ref] is the pre-existing behavior for
+    # this malformed markup).
+    it "does not swallow paragraphs when a <ref> is left unclosed" do
+      wikitext = "First para with <ref>unclosed reference.\n\nSecond para here.\n\n" \
+                 "Third para with <ref>closed</ref> end."
+      result = render_elements(wikitext)
+      expect(result).to include("Second para here.")
+      expect(result).to include("First para with")
+    end
+
+    it "handles a multi-line reference with a group attribute" do
+      result = render_elements("Claim.<ref group=\"note\">Author\nTitle 2019</ref> Next.")
+      expect(result).not_to include("[ref]")
+      expect(result).not_to include("[/ref]")
+      expect(result).to include("Claim. Next.")
+    end
+  end
+
   describe "remove_table" do
     it "removes table formated parts" do
       str_before = "{| ... \n{| ... \n ...|}\n ...|}"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: wp2txt
 version: !ruby/object:Gem::Version
-  version: 2.3.0
+  version: 2.3.1
 platform: ruby
 authors:
 - Yoichiro Hasebe
@@ -276,6 +276,7 @@ files:
 - spec/config_spec.rb
 - spec/constants_spec.rb
 - spec/corpus_spec.rb
+- spec/docs_sync_spec.rb
 - spec/file_utils_spec.rb
 - spec/fixtures/samples.rb
 - spec/formatter_sections_spec.rb
@@ -341,6 +342,7 @@ test_files:
 - spec/config_spec.rb
 - spec/constants_spec.rb
 - spec/corpus_spec.rb
+- spec/docs_sync_spec.rb
 - spec/file_utils_spec.rb
 - spec/fixtures/samples.rb
 - spec/formatter_sections_spec.rb