wp2txt 2.2.0 → 2.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5e54ea4f824be0f060036923759b1c6755b20ee1e721245db58a23665b9efaab
-  data.tar.gz: ca0a5b1ae0efae6eb4c485a6319569ea9995c662054e7e697a70429632613a25
+  metadata.gz: 4062c47480d0801e7a07d8e3bec7c65be8efeeda2eae30d398f9872810d321cd
+  data.tar.gz: 25bb3aa8a4d5baefc4616a8a4b754ec63e92d3e0b86b22ccc0ead2929a39739e
 SHA512:
-  metadata.gz: 34b61e5ee404431e521e77bc1223b9b7e1e0c9f2082ae4898e6e4b9fdee9de593c9ca45d661723f8d14f5ebffd62360dfb27d3a7b0d87d2a37691bb127c9ab6f
-  data.tar.gz: ca002ba2fc0c5a9bc8b2ae55dbc736f154e54452af681aad6f1b9ed30ca616535db94ee8cccb069defa56aacc4feb96b306d6fdb1a0102ee20611c98a213d5d0
+  metadata.gz: ff61afbb989613e286af8bea784e4494c8f71b0dd52d24d3d038e72c5d77f9196d899039d9ff8c5e4b16461c044670d8cca6bc3feda55aa8cdbc31cbc30e012b
+  data.tar.gz: ddffdfb2452a3710d527bc7deeae52818cb9c2b4dedd433b2c7ecb06db74e6e389edf127976567a3269583531acf0aabc2ea9058d7422b5b675de9bae2e227fa

data/.dockerignore

@@ -6,11 +6,12 @@ pkg
 spec
 coverage
 tmp
+.bundle
+research-notes
 benchmark_results
 data/output_samples
 scripts
 .dockerignore
-.gitignore
 .solargraph.yml
 .rubocop.yml
 Gemfile.lock

data/.gitignore

@@ -23,6 +23,9 @@ tags
 # Claude Code development files
 CLAUDE.md
 
+# Private design/research notes (not for publication)
+research-notes/
+
 # Error logs
 error_log.txt
 spec/examples.txt

data/CHANGELOG.md

@@ -5,6 +5,25 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.3.1] - 2026-08-12
+
+- **Container images no longer carry private files**: the image is built from the working tree with `git init && git add -A`, but the build context excluded `.gitignore`, so files ignored locally were tracked inside the container and shipped in the image — 2.3.0's images contained the maintainer's private research notes under `/wp2txt/research-notes`. The context now keeps `.gitignore` (so the container's file list matches a local `gem build`), explicitly excludes `research-notes` and `.bundle`, and the throwaway `.git` (a blob copy of every added file) is deleted after the build. `rake check_image` builds and inspects an image for private paths, and `rake push` refuses to publish without that check. The published gems were never affected; 2.3.0 images have been withdrawn
+- **Docker Hub retired**: images are published to GitHub Container Registry only (`ghcr.io/yohasebe/wp2txt`). The Docker Hub repository is no longer updated
+- **Fix: multi-line `<ref>` references broken by element splitting — also fixes `--extract-citations` on multi-line cite templates**: `Article#parse` splits elements at newlines, so any reference written across lines (very common for multi-line `{{cite …}}` templates) had its `[ref]` and `[/ref]` markers land in separate elements, invisible to `remove_ref`. The markers and raw reference markup leaked into the output, and — more importantly — `--extract-citations` silently never fired for multi-line cite templates, producing inconsistent output versus their single-line equivalents. `make_reference` now drops empty references outright and flattens the rest onto a single line before element splitting, so multi-line references behave exactly like single-line ones. Ordinary paragraph breaks outside references are unaffected
+- **Fix: `[ref]` markers destroyed by external-link processing**: `process_external_links` stripped the brackets of the `[ref]`/`[/ref]` markers produced by `make_reference` (their contents took the single-word branch), so `remove_ref` could no longer find them and tag names plus reference bodies leaked into extracted text — even with `--ref`, the kept markers came out broken. The markers are now hidden behind placeholders for the duration of the bracket scan and restored afterwards. Downstream corpora no longer contain `ref…/ref` residue, which broke tokenization/sentence splitting and mixed bibliographic text into body prose
+- **Runaway-query hardening**: a `query_sql` child process now sets its own kernel-enforced CPU limit (`RLIMIT_CPU`, the query timeout plus a small grace) in addition to the parent's wall-clock kill. Previously a child orphaned by the parent's death — an interrupted test run, a closed terminal, a crashed server — kept executing forever: sqlite3 holds the GVL inside `sqlite3_step`, so Ruby never reaches a signal-safe point and even SIGTERM is ignored. Observed in the wild as two processes spinning at 99% CPU for over five days. The CPU limit is strictly more permissive than the existing wall-clock deadline, so no query that would otherwise succeed is affected
+- **Test-suite hang guard**: each example now runs under a wall-clock timeout (120s default; `WP2TXT_SPEC_TIMEOUT=0` disables it, the `:no_timeout` tag exempts an example). The suite deliberately exercises runaway queries, so a wedged example must fail rather than spin
+
+## [2.3.0] - 2026-07-24
+
+- **Documentation split**: README now focuses on text extraction; the research layer (indexes, exhaustive queries, full-text search, langlinks, cross-language SQL, MCP server) is documented in the new [Research Infrastructure Guide](docs/RESEARCH.md), including the complete MCP tool table
+- **Container images on GHCR**: images are now published to `ghcr.io/yohasebe/wp2txt` (Docker Hub `yohasebe/wp2txt` is maintained as a mirror)
+
+- **`extract_corpus` `titles:` argument**: Extract an explicit set of article titles (e.g. a set determined via `query_sql`) — titles are normalized (MediaWiki rules), deduplicated preserving input order, and one redirect hop is resolved; missing titles (including redirects to nowhere) are skipped and reported as `not_found` (count + 20-title sample, also in the `.meta.json` sidecar). Mutually exclusive with the filter arguments (set operations belong in SQL); capped at 10,000 titles. The sidecar records `titles_count` + `titles_sha256` (order-independent) for reproducibility, enumerating the full list when ≤100 titles. Also available via `start_extract_job`
+- **`query_sql` `output_path:` argument**: Write ALL rows of a large result to a JSONL file (with a `.meta.json` sidecar recording the SQL, dump version, attach configuration, row count, and tool version) and return only a summary + 3-row sample — the extract_corpus D4 pattern generalized to SQL. Writes happen in the forked child (so the 30s SIGKILL deadline covers them) to a `.partial` file that the parent atomically renames on success and removes on every failure path (child crash, timeout kill, query error). Hard cap `SQL_FILE_ROW_LIMIT` = 5M rows (`truncated` flag), cells clipped at 64KB (`cells_clipped` count), duplicate column names are made unique (`_2` suffix), `limit` is ignored in this mode, and existing files require `overwrite: true`. The MCP layer's output-dir confinement is now a shared helper (`Wp2txt::OutputPath.confine`) used by extract_corpus, start_extract_job, and query_sql alike
+- **Interlanguage links (`--import-langlinks`)**: Import the official `{lang}wiki-{date}-langlinks.sql.gz` dump into the Tier 1 metadata index as a `langlinks` table (`ll_from` = source page_id, `ll_lang` = target language, `ll_title` = normalized target title). Version pinning is enforced: the langlinks file's dump name must equal the built index's dump version — a mismatch is rejected with no override. Streams the MySQL dump without loading it whole (escape-safe tuple parser, 10k-row transaction batches, indexes created after the load). `--langlinks-langs` restricts imported target languages (e.g. `en,de,fr,zh,ko`); re-import requires `-U` (otherwise a no-op reporting the previous import time). Provenance (source file, size, import time, wp2txt version, language filter, row count) is stamped into the index and reported by `dump_info`; a post-import sanity check samples titles per language and warns when the join rate against a locally installed target edition falls below 90%
+- **Multi-dump ATTACH in `query_sql`**: The MCP `query_sql` tool gains an optional `attach` argument (language codes only, never paths): `attach: ["en"]` read-only ATTACHes that language's locally installed metadata DB as `en_meta` and its FTS DB (when built) as `en_fts`, both sharing the main DB's schema. Codes are validated, only installed indexes are accepted, path resolution is server-side (same-date dump preferred; otherwise the latest build with a `dump_mismatch` note in the response's `attached` metadata). User SQL still cannot contain ATTACH/DETACH — attachments are issued by server code only, via `mode=ro` URIs on a read-only connection. Combined with the langlinks table, this enables single-query cross-language comparisons (e.g. section-structure diffs of article pairs)
+
 ## [2.2.0] - 2026-07-22
 
 - **Index hardening (design review follow-up)**: `ord` now has identical semantics in `page_sections` and `fts_map` (lead = 0, first heading = 1; schema v2 — rebuild indexes with `--build-index -U`); section headings are normalized identically in both indexes (decorated headings like `== '''X''' ==` now match section filters); indexes record the wp2txt version (and the FTS index a rendering-config digest) so `dump_info` can flag indexes built by code whose text cleaning differs; rebuilds are atomic (built alongside, renamed on completion — a failed rebuild no longer destroys the working index)

data/DEVELOPMENT.md

@@ -395,7 +395,8 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 Build and push Docker images:
 
 ```bash
-rake push  # Builds multi-arch and pushes to Docker Hub
+rake check_image  # Builds the image locally and verifies it carries no private files
+rake push        # Verifies, then builds multi-arch and pushes to GHCR
 ```
 
 ## Release Process

data/DEVELOPMENT_ja.md

@@ -395,7 +395,8 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
 Dockerイメージのビルドとプッシュ：
 
 ```bash
-rake push  # マルチアーキテクチャでビルドしDocker Hubにプッシュ
+rake check_image  # ローカルでイメージをビルドし、私的ファイルの混入がないか検証
+rake push        # 検証したうえでマルチアーキテクチャでビルドしGHCRにプッシュ
 ```
 
 ## リリースプロセス

data/Dockerfile

@@ -1,10 +1,18 @@
 FROM ruby:4.0.1-alpine3.23
 
+# Link the GHCR package to the source repository
+LABEL org.opencontainers.image.source="https://github.com/yohasebe/wp2txt"
+
 WORKDIR /wp2txt
 COPY . ./
 RUN rm -f Gemfile.lock
 
-# Install dependencies (git is required by gemspec's `git ls-files`)
+# Install dependencies (git is required by gemspec's `git ls-files`).
+# The repository's .gitignore is copied in deliberately: `git add -A` must
+# honour it so the file list here matches a local `gem build` — without it,
+# ignored material (private notes, scratch files) would land in the image.
+# The throwaway .git is removed afterwards: it holds a blob copy of every
+# added file and is dead weight in the published image.
 RUN apk update && \
     apk upgrade && \
     apk add --no-cache \
@@ -14,6 +22,7 @@ RUN apk update && \
       build-base curl-dev wget && \
     git init && git add -A && \
     bundle install -j4 && \
+    rm -rf /wp2txt/.git && \
     apk del .build-packages
 
 # lbzip2 is not available as an Alpine package; build from source

data/README.md

@@ -34,7 +34,7 @@ WP2TXT extracts plain text and category information from Wikipedia dump files. I
 - **Multilingual support** - Category and redirect detection for 350+ Wikipedia languages
 - **Streaming processing** - Process large dumps without intermediate files
 - **JSON output** - Machine-readable JSONL format for data pipelines
-- **Offline metadata index** - Exhaustive, version-pinned queries over categories and section headings, no API required
+- **Offline research indexes** - Exhaustive, version-pinned queries over categories, section headings, and full text; interlanguage links for cross-edition SQL
 - **MCP server** - Expose a local dump to LLM agents (Claude, ChatGPT, Gemini, local models) for grounded, reproducible corpus work
 
 ## Use Cases
@@ -79,19 +79,21 @@ The Docker image bundles everything — Ruby, decompression tools, and the MCP s
 **Interactive / batch CLI:**
 
 ```shell
-docker run -it -v /path/to/localdata:/data yohasebe/wp2txt
+docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 ```
 
 The `wp2txt` command is available inside the container. Use `/data` for input/output files.
 
+Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`). The former Docker Hub repository (`yohasebe/wp2txt`) is no longer updated as of 2.3.1.
+
 **MCP server (no Ruby required on the host):**
 
 ```shell
 # Build the indexes once (cached in a named volume so they persist)
-docker run -it -v wp2txt:/root/.wp2txt yohasebe/wp2txt wp2txt --build-index --fulltext -L ja
+docker run -it -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt --build-index --fulltext -L ja
 
 # Register with an MCP client, e.g. Claude Code
-claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt yohasebe/wp2txt wp2txt-mcp -L ja
+claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt-mcp -L ja
 ```
 
 Note: use `-i` (not `-it`) when running the MCP server — a TTY would corrupt the JSON-RPC stream. The named volume (`wp2txt:`) holds downloaded dumps and indexes; without it they are lost when the container exits.
@@ -253,63 +255,28 @@ By default, citation templates are removed. Use `--extract-citations` to extract
 
 Supported: `{{cite book}}`, `{{cite web}}`, `{{cite news}}`, `{{cite journal}}`, `{{Citation}}`, etc.
 
-## Offline Metadata Index
-
-Build a local SQLite index of a dump — per-article categories, section headings, redirects, and the category hierarchy — extracted from the dump itself (no API access, no rate limits, version-pinned for reproducibility):
-
-    $ wp2txt --build-index --lang=ja     # downloads dump if needed; ja: ~15 min, ~1.7 GB index
-
-Then run exhaustive queries entirely offline:
-
-    # All film articles (recursing 3 levels of subcategories) that have a plot section
-    $ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
-
-    # Machine-readable output with total count
-    $ wp2txt --find-articles --in-category "Films" -D 2 -j json --limit 100 --lang=en
-
-Unlike web/API access, these queries are *exhaustive* (they scan every article, not search-ranked results) and *reproducible* (pinned to a specific dump version). Answering "which of the 1.5M articles have X" takes milliseconds once the index is built.
-
-### Full-Text Search
-
-Add `--fulltext` to also build an FTS5 index over the article text itself (tokenizer auto-selected: character-trigram for Japanese/Chinese/Korean, word-based for space-delimited languages):
-
-    $ wp2txt --build-index --fulltext --lang=ja
-    $ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
-
-Search hits report the article, section, and a snippet; totals are exhaustive counts, so `0 matches` is a verifiable absence claim for that dump version. The index is contentless (stores only the inverted index; snippets are re-rendered from the dump), keeping disk overhead moderate.
+## Research Infrastructure (Indexes, Exhaustive Queries, MCP)
 
-## MCP Server (LLM Integration)
+Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned research
+database**: SQLite indexes over categories, section headings, redirects, and (optionally)
+the full article text, plus interlanguage links for cross-edition comparison — all
+queryable offline, exhaustively, and exposed to LLM agents via an MCP server.
 
-`wp2txt-mcp` exposes a local dump to LLM agents (Claude, ChatGPT, Gemini, local models — any MCP-capable client) via the Model Context Protocol:
-
-    $ gem install mcp                    # optional dependency, needed only for the server
-    $ wp2txt --build-index --lang=ja     # prerequisite
-    $ wp2txt-mcp --lang=ja               # stdio MCP server
-
-Example client configuration (e.g., Claude Desktop / Claude Code):
-
-```json
-{
-  "mcpServers": {
-    "wp2txt": { "command": "wp2txt-mcp", "args": ["--lang", "ja"] }
-  }
-}
+```console
+$ wp2txt --build-index --fulltext --lang=ja      # build the indexes
+$ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
+$ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
+$ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
+$ wp2txt-mcp --lang=ja                           # stdio MCP server for LLM agents
 ```
 
-Tools provided:
-
-| Tool | Purpose |
-|------|---------|
-| `dump_info` | Dump version, index tiers, corpus statistics |
-| `get_article` / `get_sections` / `list_headings` | Single-article access (redirect-aware) |
-| `find_articles` | Exhaustive filtered listing (category recursion, section headings, title match) |
-| `category_tree` / `section_stats` | Scope exploration and heading frequency discovery |
-| `section_cooccurrence` | Verify section-alias hypotheses (synonyms rarely co-occur in one article) |
-| `save_alias_set` etc. | Persist verified per-dump alias groups (server-side co-occurrence guardrail) |
-| `extract_corpus` | Filtered extraction to JSONL + reproducibility sidecar; optional RAG chunking |
-| `start_extract_job` / `job_status` / `cancel_job` | Background jobs for large extractions |
+Unlike web/API access, these queries scan every article (a `0 matches` result is a
+verifiable absence claim for that dump version) and are reproducible: extractions record
+the dump version and query in a `.meta.json` sidecar.
 
-Design principles: large results go to disk (the model receives a summary plus a 3-record sample, never the full corpus); every output records the dump version and query in a `.meta.json` sidecar; section aliases are discovered from real dump statistics by the LLM and mechanically verified — no hand-maintained per-language dictionaries. The bundled `discover_aliases` prompt walks any agent through the verification protocol.
+**→ See the [Research Infrastructure Guide](docs/RESEARCH.md)** for index building,
+exhaustive queries, full-text search, interlanguage links, cross-language SQL, the full
+MCP tool list, and design principles.
 
 ## Command Line Options
 
@@ -384,6 +351,15 @@ Design principles: large results go to disk (the model receives a summary plus a
       -v, --version                    Print version
       -h, --help                       Show help
 
+### Research infrastructure options
+
+    --build-index                    Build the metadata index (add --fulltext for FTS)
+    --find-articles / --search       Exhaustive offline queries (see docs/RESEARCH.md)
+    --import-langlinks               Import interlanguage links (version-matched)
+    --fts-optimize                   Optimize an existing full-text index
+
+See the [Research Infrastructure Guide](docs/RESEARCH.md) for details.
+
 ## Configuration File
 
 Create persistent settings with:
@@ -425,6 +401,10 @@ Turbo mode (default) splits bz2 into XML chunks first, then processes in paralle
 
 See [CHANGELOG.md](CHANGELOG.md) for detailed release notes.
 
+**v2.3.0 (July 2026)**: Interlanguage links import, cross-language SQL (multi-dump ATTACH), explicit-title extraction, SQL results to file with reproducibility sidecars, GHCR image publishing.
+
+**v2.2.0 (July 2026)**: Offline metadata index, FTS5 full-text search, MCP server, query_sql escape hatch, extraction jobs.
+
 **v2.1.0 (February 2026)**: SQLite caching, Ractor parallelism (Ruby 4.0+), template expansion, content markers, Docker image update.
 
 **v2.0.0 (January 2026)**: Auto-download mode, category-based extraction, article extraction by title, JSON output, streaming processing, Ruby 4.0 support.

data/README_ja.md

@@ -34,6 +34,8 @@ WP2TXTはWikipediaダンプファイルからプレーンテキストとカテ
 - **多言語対応** - 350以上のWikipedia言語でカテゴリ・リダイレクトを検出
 - **ストリーミング処理** - 中間ファイルなしで大規模ダンプを処理
 - **JSON出力** - データパイプライン向けの機械可読JSONL形式
+- **オフライン研究索引** - カテゴリ・節見出し・全文に対する悉皆的・版固定クエリ、言語間リンクによる言語版横断SQL
+- **MCPサーバー** - ローカルダンプをLLMエージェント（Claude、ChatGPT、Gemini、ローカルモデル）に公開
 
 ## ユースケース
 
@@ -43,6 +45,8 @@ wp2txtは以下の用途に適しています：
 - トピック領域を横断した比較言語研究
 - NLPタスク向けのメタデータ付きWikipediaテキスト抽出
 - 並行カテゴリ構造を利用した対照言語研究
+- 版固定のRAG知識ベース・LLM評価データセット
+- web検索では原理的に不可能な悉皆的言明（「このカテゴリのどの記事にもXへの言及がない」）
 
 ## データアクセス
 
@@ -71,11 +75,13 @@ Windows：[Bzip2 for Windows](http://gnuwin32.sourceforge.net/packages/bzip2.htm
 ### Docker（代替方法）
 
 ```shell
-docker run -it -v /path/to/localdata:/data yohasebe/wp2txt
+docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 ```
 
 `wp2txt`コマンドはコンテナ内で使用可能です。入出力には`/data`ディレクトリを使用してください。
 
+イメージはGitHub Container Registry（`ghcr.io/yohasebe/wp2txt`）で公開されています。旧 Docker Hub リポジトリ（`yohasebe/wp2txt`）は 2.3.1 以降更新されません。
+
 ## 基本的な使い方
 
 ### 自動ダウンロードと処理（推奨）
@@ -327,6 +333,28 @@ defaults:
 
 コマンドラインオプションは設定ファイルの設定を上書きします。
 
+## 研究基盤（索引・悉皆クエリ・MCP）
+
+テキスト抽出に加えて、wp2txtはダンプを**ローカルな版固定研究データベース**に変換できます。
+カテゴリ・節見出し・リダイレクト・（オプションで）記事全文のSQLite索引、言語版横断比較のための
+言語間リンク（langlinks）— すべてオフラインで悉皆的にクエリでき、MCPサーバー経由で
+LLMエージェントにも公開できます。
+
+```console
+$ wp2txt --build-index --fulltext --lang=ja      # 索引の構築
+$ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
+$ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
+$ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
+$ wp2txt-mcp --lang=ja                           # LLMエージェント向けMCPサーバー
+```
+
+web/API アクセスと異なり、これらのクエリは全記事を走査します（`0件` は当該ダンプ版に対する
+検証可能な不在の言明になります）。抽出結果には dump 版とクエリを記録した `.meta.json`
+サイドカーが付き、再現可能です。
+
+**→ 詳細は [Research Infrastructure Guide](docs/RESEARCH.md)（英語）を参照**：
+索引構築、悉皆クエリ、全文検索、言語間リンク、言語版横断SQL、MCPツール一覧、設計原則。
+
 ## パフォーマンス
 
 MacBook Air M4でのベンチマーク結果（7並列プロセス、ターボモード、ダウンロード時間除く）:
@@ -347,6 +375,10 @@ MacBook Air M4でのベンチマーク結果（7並列プロセス、ターボ
 
 詳細なリリースノートは[CHANGELOG.md](CHANGELOG.md)を参照してください。
 
+**v2.3.0（2026年7月）**: 言語間リンク取り込み、言語版横断SQL（multi-dump ATTACH）、明示タイトル集合の抽出、SQL結果のファイル出力と再現性サイドカー、GHCRイメージ公開。
+
+**v2.2.0（2026年7月）**: オフラインメタデータ索引、FTS5全文検索、MCPサーバー、query_sql、抽出ジョブ。
+
 **v2.1.0（2026年2月）**: SQLiteキャッシュ、Ractor並列処理（Ruby 4.0+）、テンプレート展開、コンテンツマーカー、Dockerイメージ更新。
 
 **v2.0.0（2026年1月）**: 自動ダウンロードモード、カテゴリベース抽出、タイトル指定抽出、JSON出力、ストリーミング処理、Ruby 4.0サポート。

data/Rakefile

@@ -32,14 +32,40 @@ Rake::Task["build"].enhance([:normalize_permissions])
 # Docker
 # =============================================================================
 
-desc "Push Docker images"
-task :push do
+# Paths that must never reach a published image. The image is built from the
+# working tree, so anything ignored locally (private notes, scratch files)
+# would otherwise ride along; 2.3.0's images shipped research-notes/ this way.
+IMAGE_FORBIDDEN_PATHS = %w[/wp2txt/research-notes /wp2txt/tmp /wp2txt/.git /wp2txt/CLAUDE.md /wp2txt/.claude].freeze
+
+desc "Verify a built image contains no private material (run before pushing)"
+task :verify_image, [:tag] do |_t, args|
+  tag = args[:tag] || "wp2txt-verify:local"
+  checks = IMAGE_FORBIDDEN_PATHS.map { |p| "test -e #{p} && echo LEAK:#{p}" }.join("; ")
+  out = `docker run --rm #{tag} sh -c '#{checks}; true' 2>&1`
+  leaks = out.lines.grep(/^LEAK:/).map(&:strip)
+  abort "Image #{tag} contains private paths:\n  #{leaks.join("\n  ")}" unless leaks.empty?
+
+  puts "OK: #{tag} contains none of #{IMAGE_FORBIDDEN_PATHS.join(', ')}"
+end
+
+desc "Build the image locally and verify it, without pushing"
+task :check_image do
+  sh "docker build -t wp2txt-verify:local ."
+  Rake::Task[:verify_image].invoke
+end
+
+desc "Build and push Docker images to GHCR (verifies a local build first)"
+task push: :check_image do
+  # Docker Hub was retired after 2.3.0; GHCR (ghcr.io/yohasebe/wp2txt) is the
+  # only registry. Requires `docker buildx use multiarch` and a ghcr.io login.
   sh <<-SCRIPT.strip_heredoc, { verbose: false }
     /bin/bash -xeu <<'BASH'
-      # docker buildx create --name mybuilder
-      # docker buildx use mybuilder
+      # docker buildx create --name multiarch
+      # docker buildx use multiarch
       # docker buildx inspect --bootstrap
-      docker buildx build --platform linux/amd64,linux/arm64 -t yohasebe/wp2txt:#{Wp2txt::VERSION} -t yohasebe/wp2txt:latest . --push
+      docker buildx build --platform linux/amd64,linux/arm64 \
+        -t ghcr.io/yohasebe/wp2txt:#{Wp2txt::VERSION} -t ghcr.io/yohasebe/wp2txt:latest \
+        . --push
     BASH
   SCRIPT
 end

data/bin/wp2txt

@@ -711,6 +711,7 @@ class WpApp
     return run_find_articles(opts) if opts[:find_articles]
     return run_search(opts) if opts[:search]
     return run_fts_optimize(opts) if opts[:fts_optimize]
+    return run_import_langlinks(opts) if opts[:import_langlinks]
 
     # Determine input source
     if opts[:from_category] && opts[:lang]

data/bin/wp2txt-mcp

@@ -24,6 +24,7 @@ require "optimist"
 require "json"
 require_relative "../lib/wp2txt/corpus"
 require_relative "../lib/wp2txt/corpus_jobs"
+require_relative "../lib/wp2txt/output_path"
 require_relative "../lib/wp2txt/version"
 
 opts = Optimist.options do
@@ -159,16 +160,23 @@ end
 
 server.define_tool(
   name: "query_sql",
-  description: "Read-only SQL (SELECT/WITH only) against the index databases — the escape hatch for questions the fixed tools cannot express (multi-way intersections, aggregations, joins). Call describe_schema first to see the tables. Main DB: pages(page_id, title, namespace, redirect_to, text_length), page_categories(page_id, category), page_sections(page_id, heading, level, ord), category_hierarchy(child, parent), alias_sets. When the fulltext tier exists, the FTS DB is attached as `fts` (fts.fts_sections MATCH ..., fts.fts_map). Filter articles with namespace = 0 AND redirect_to IS NULL. Rows are capped by `limit` (default 200, max 1000) with truncated flag; long cells are clipped. Queries are killed after 30 seconds (the error explains the likely cause); the tables are large (millions of rows), so constrain joins with indexed keys (page_id) and add WHERE filters.",
+  description: "Read-only SQL (SELECT/WITH only) against the index databases — the escape hatch for questions the fixed tools cannot express (multi-way intersections, aggregations, joins). Call describe_schema first to see the tables. Main DB: pages(page_id, title, namespace, redirect_to, text_length), page_categories(page_id, category), page_sections(page_id, heading, level, ord), category_hierarchy(child, parent), alias_sets; a langlinks table (ll_from = source page_id, ll_lang, ll_title) is also present when interlanguage links were imported via wp2txt --import-langlinks. When the fulltext tier exists, the FTS DB is attached as `fts` (fts.fts_sections MATCH ..., fts.fts_map). Cross-language: pass attach: [\"en\",\"de\"] to read-only ATTACH other locally installed languages' indexes, referenced as {lang}_meta.pages / {lang}_meta.page_sections / ... and {lang}_fts when that language has a fulltext tier — attached DBs share the main DB's schema. Hyphenated language codes map to underscored aliases (e.g. zh-yue → zh_yue_meta). Only installed languages are accepted (the main language itself cannot be attached); the response's `attached` field lists what was attached (dump_name, fts, dump_mismatch when the dates differ). Join langlinks (ll_from = page_id) to map articles across languages. Filter articles with namespace = 0 AND redirect_to IS NULL. Rows are capped by `limit` (default 200, max 1000) with truncated flag; long cells are clipped. output_path: write ALL rows to a JSONL file (with a .meta.json recording the SQL, dump versions, and attach configuration) and return only a summary + 3-row sample — use this whenever the result is large; `limit` is ignored in this mode. Queries are killed after 30 seconds (the error explains the likely cause); the tables are large (millions of rows), so constrain joins with indexed keys (page_id) and add WHERE filters.",
   input_schema: {
     properties: {
       sql: { type: "string" },
-      limit: { type: "integer", minimum: 1, maximum: 1000 }
+      limit: { type: "integer", minimum: 1, maximum: 1000 },
+      attach: { type: "array", items: { type: "string" },
+                description: "Language codes of other locally installed dumps to ATTACH read-only as {lang}_meta / {lang}_fts (default: none)" },
+      output_path: { type: "string", description: "JSONL output path for ALL rows; relative paths resolve under the server's output dir" },
+      overwrite: { type: "boolean", description: "Replace an existing output file (default: refuse)" }
     },
     required: ["sql"]
   }
-) do |sql:, limit: 200, server_context:|
-  respond { corpus.query_sql(sql, limit: limit) }
+) do |sql:, limit: 200, attach: [], output_path: nil, overwrite: false, server_context:|
+  respond do
+    path = output_path && Wp2txt::OutputPath.confine(output_path, OUTPUT_DIR, overwrite: overwrite)
+    corpus.query_sql(sql, limit: limit, attach: attach, output_path: path, overwrite: overwrite)
+  end
 end
 
 server.define_tool(
@@ -246,7 +254,7 @@ end
 
 server.define_tool(
   name: "save_alias_set",
-  description: "Persist a named set of section-heading alias groups for this dump (e.g. [[\"あらすじ\",\"ストーリー\",\"物語\"]]). A server-side guardrail re-checks co-occurrence: a pair that appears in the same article more than 20% of the time (when both headings occur in at least 100 articles) is treated as distinct sections, not synonyms, and blocks the save with saved:false and the offending pairs listed — fix the groups, or pass force:true only if you verified them another way (e.g., by reading section contents with get_sections). The response echoes the saved groups and the exact criteria used. Saved sets are referenced via alias_set in find_articles / get_sections / extract_corpus and recorded in extraction metadata for reproducibility.",
+  description: "Persist a named set of section-heading alias groups for this dump (e.g. [[\"あらすじ\",\"ストーリー\",\"物語\"]]). A server-side guardrail re-checks co-occurrence: a pair that appears in the same article more than 20% of the time (when both headings occur in at least 100 articles) is treated as distinct sections, not synonyms, and blocks the save with saved:false and the offending pairs listed — fix the groups, or pass force:true only if you verified them another way (e.g., by reading section contents with get_sections). The response echoes the saved groups and the exact criteria used. Saved sets are referenced via alias_set in find_articles / get_sections / extract_corpus and recorded in extraction metadata for reproducibility. A refused save is signaled by saved:false in the RESULT (not an exception); verify persistence with list_alias_sets.",
   input_schema: {
     properties: {
       name: { type: "string" },
@@ -279,6 +287,8 @@ EXTRACT_SCHEMA = {
   properties: {
     output_path: { type: "string", description: "JSONL output path; relative paths resolve under the server's output dir" },
     content: { type: "string", enum: ["sections", "summary", "full", "wikitext"] },
+    titles: { type: "array", items: { type: "string" }, maxItems: 10_000,
+              description: "Explicit article titles to extract (mutually exclusive with the filter arguments)" },
     sections: { type: "array", items: { type: "string" } },
     alias_set: { type: "string" },
     category: { type: "string" },
@@ -294,23 +304,12 @@ EXTRACT_SCHEMA = {
   required: ["output_path", "content"]
 }.freeze
 
-def extract_params(output_path:, content:, sections: nil, alias_set: nil, category: nil,
+def extract_params(output_path:, content:, titles: nil, sections: nil, alias_set: nil, category: nil,
                    depth: 0, categories: nil, category_match: nil,
                    title_match: nil, limit: 0, chunk_size: nil, chunk_overlap: 0,
                    overwrite: false)
-  # Confine writes to the server's output directory (an agent mixing up paths
-  # must not be able to clobber arbitrary user files), and refuse to silently
-  # replace existing outputs
-  path = File.expand_path(output_path, OUTPUT_DIR)
-  base = File.expand_path(OUTPUT_DIR)
-  unless path == base || path.start_with?(base + File::SEPARATOR)
-    raise ArgumentError, "output_path must stay within the server output directory (#{base})"
-  end
-  if File.exist?(path) && !overwrite
-    raise ArgumentError, "output file already exists: #{path} (pass overwrite: true to replace it)"
-  end
-
-  { output_path: path, content: content,
+  { output_path: Wp2txt::OutputPath.confine(output_path, OUTPUT_DIR, overwrite: overwrite),
+    content: content, titles: titles,
     sections: sections, alias_set: alias_set, category: category, depth: depth,
     categories: categories, category_match: category_match,
     title_match: title_match, limit: limit, chunk_size: chunk_size, chunk_overlap: chunk_overlap }
@@ -318,7 +317,7 @@ end
 
 server.define_tool(
   name: "extract_corpus",
-  description: "Extract matching articles to a JSONL file on disk (with a .meta.json sidecar recording dump version and query for reproducibility). Returns a summary and the first 3 records only — the full corpus stays on disk; process it with your own tools. content: 'sections' (named sections only), 'summary' (lead text), 'full' (whole cleaned article), or 'wikitext' (raw markup, for mining infoboxes/templates that cleaning replaces with markers). Filters are the same as find_articles (category/depth, categories, category_match, title_match). Optional chunk_size/chunk_overlap produce one RAG-ready record per chunk with section_path and chunk_index (not with wikitext). Synchronous, capped at 5000 articles (result flags truncation) — use start_extract_job for larger extractions. Dry-run the filters with find_articles first.",
+  description: "Extract matching articles to a JSONL file on disk (with a .meta.json sidecar recording dump version and query for reproducibility). Returns a summary and the first 3 records only — the full corpus stays on disk; process it with your own tools. content: 'sections' (named sections only), 'summary' (lead text), 'full' (whole cleaned article), or 'wikitext' (raw markup, for mining infoboxes/templates that cleaning replaces with markers). Filters are the same as find_articles (category/depth, categories, category_match, title_match). titles: extract an explicit list of articles (e.g. a set determined via query_sql); mutually exclusive with the filter arguments; redirects are resolved; missing titles are reported in not_found. Optional chunk_size/chunk_overlap produce one RAG-ready record per chunk with section_path and chunk_index (not with wikitext). Synchronous, capped at 5000 articles (result flags truncation) — use start_extract_job for larger extractions. Dry-run the filters with find_articles first.",
   input_schema: EXTRACT_SCHEMA
 ) do |server_context:, **args|
   respond { corpus.extract_corpus(**extract_params(**args)) }