wp2txt 2.1.2 → 2.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 03b8bbbab05c9ab7f56ed3f89c00ae14c3259fbb7b00d54010b04a0decb79f16
-  data.tar.gz: dd92bacf8e6f17e9d6d5ad33bb3e9c5777cb59a53f4f150136a2143dd84f5b1d
+  metadata.gz: 076aa3a7681c0e0e6e13948e5c66f6f9e15a0307507a944a78f29c0e7856eea0
+  data.tar.gz: f8f2d3a4efc82d1922f9ca2764b23bb348aa924e752439805ab61683ff66a737
 SHA512:
-  metadata.gz: b4d95651f10fd30d3c0d23d2846c60b891716b2638a62151712a28bf6ea1f2a6e0da18abaa5b48553f6d736ba86bfeb8fd978ee1e07facfc45c56256734b239d
-  data.tar.gz: e10dcbcba41561a26df4f5740b6517bc2253fe6226eb68d588c17c9bf00f2ec6b9e41ba6eceeff63be0fc742856eb20c385c290a32f5ccc8d364b110fff7fcdb
+  metadata.gz: ae377878c20dde5a376d96efec19f84a74a20e66dd8f45ff5d41c438f519ced7cf5d4e6a6102ceee5c4b59333f0b1ca42167cf5e178abf803de868855b4efa6c
+  data.tar.gz: c23cb4598a7afeda092d75688be12e9e777874a4b9da289aeb1d97123b5cc64b3b56337722290fcc4352fde9031a856d4ba81393055ef8596fe8818d51e258ac

data/.gitignore

@@ -23,6 +23,9 @@ tags
 # Claude Code development files
 CLAUDE.md
 
+# Private design/research notes (not for publication)
+research-notes/
+
 # Error logs
 error_log.txt
 spec/examples.txt

data/CHANGELOG.md

@@ -5,6 +5,33 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.3.0] - 2026-07-24
+
+- **Documentation split**: README now focuses on text extraction; the research layer (indexes, exhaustive queries, full-text search, langlinks, cross-language SQL, MCP server) is documented in the new [Research Infrastructure Guide](docs/RESEARCH.md), including the complete MCP tool table
+- **Container images on GHCR**: images are now published to `ghcr.io/yohasebe/wp2txt` (Docker Hub `yohasebe/wp2txt` is maintained as a mirror)
+
+- **`extract_corpus` `titles:` argument**: Extract an explicit set of article titles (e.g. a set determined via `query_sql`) — titles are normalized (MediaWiki rules), deduplicated preserving input order, and one redirect hop is resolved; missing titles (including redirects to nowhere) are skipped and reported as `not_found` (count + 20-title sample, also in the `.meta.json` sidecar). Mutually exclusive with the filter arguments (set operations belong in SQL); capped at 10,000 titles. The sidecar records `titles_count` + `titles_sha256` (order-independent) for reproducibility, enumerating the full list when ≤100 titles. Also available via `start_extract_job`
+- **`query_sql` `output_path:` argument**: Write ALL rows of a large result to a JSONL file (with a `.meta.json` sidecar recording the SQL, dump version, attach configuration, row count, and tool version) and return only a summary + 3-row sample — the extract_corpus D4 pattern generalized to SQL. Writes happen in the forked child (so the 30s SIGKILL deadline covers them) to a `.partial` file that the parent atomically renames on success and removes on every failure path (child crash, timeout kill, query error). Hard cap `SQL_FILE_ROW_LIMIT` = 5M rows (`truncated` flag), cells clipped at 64KB (`cells_clipped` count), duplicate column names are made unique (`_2` suffix), `limit` is ignored in this mode, and existing files require `overwrite: true`. The MCP layer's output-dir confinement is now a shared helper (`Wp2txt::OutputPath.confine`) used by extract_corpus, start_extract_job, and query_sql alike
+- **Interlanguage links (`--import-langlinks`)**: Import the official `{lang}wiki-{date}-langlinks.sql.gz` dump into the Tier 1 metadata index as a `langlinks` table (`ll_from` = source page_id, `ll_lang` = target language, `ll_title` = normalized target title). Version pinning is enforced: the langlinks file's dump name must equal the built index's dump version — a mismatch is rejected with no override. Streams the MySQL dump without loading it whole (escape-safe tuple parser, 10k-row transaction batches, indexes created after the load). `--langlinks-langs` restricts imported target languages (e.g. `en,de,fr,zh,ko`); re-import requires `-U` (otherwise a no-op reporting the previous import time). Provenance (source file, size, import time, wp2txt version, language filter, row count) is stamped into the index and reported by `dump_info`; a post-import sanity check samples titles per language and warns when the join rate against a locally installed target edition falls below 90%
+- **Multi-dump ATTACH in `query_sql`**: The MCP `query_sql` tool gains an optional `attach` argument (language codes only, never paths): `attach: ["en"]` read-only ATTACHes that language's locally installed metadata DB as `en_meta` and its FTS DB (when built) as `en_fts`, both sharing the main DB's schema. Codes are validated, only installed indexes are accepted, path resolution is server-side (same-date dump preferred; otherwise the latest build with a `dump_mismatch` note in the response's `attached` metadata). User SQL still cannot contain ATTACH/DETACH — attachments are issued by server code only, via `mode=ro` URIs on a read-only connection. Combined with the langlinks table, this enables single-query cross-language comparisons (e.g. section-structure diffs of article pairs)
+
+## [2.2.0] - 2026-07-22
+
+- **Index hardening (design review follow-up)**: `ord` now has identical semantics in `page_sections` and `fts_map` (lead = 0, first heading = 1; schema v2 — rebuild indexes with `--build-index -U`); section headings are normalized identically in both indexes (decorated headings like `== '''X''' ==` now match section filters); indexes record the wp2txt version (and the FTS index a rendering-config digest) so `dump_info` can flag indexes built by code whose text cleaning differs; rebuilds are atomic (built alongside, renamed on completion — a failed rebuild no longer destroys the working index)
+- **MCP safety hardening**: `query_sql` runs in a killable subprocess with a 30s wall-clock cap and returns a query-plan diagnosis on timeout; the keyword screen no longer rejects legitimate values inside string literals; `extract_corpus` output paths are confined to the server output directory and refuse to overwrite existing files without `overwrite: true`; background jobs are serialized (one at a time) and all SQLite connections are closed before forking extraction workers
+- **Cold-start parity for LLM clients**: closes the gaps that previously required internal knowledge to work around. `get_categories` tool; `find_articles`/`extract_corpus` accept `categories` (exact multi-category AND) and `category_match` (substring on category names); `extract_corpus` gains `content: "wikitext"` for structure mining (infoboxes/templates); `query_sql` read-only SQL escape hatch (SELECT/WITH only, keyword-screened, read-only connection, row/cell caps) with `describe_schema`; article titles are normalized like MediaWiki (underscores, capitalization); `get_article` truncates at `max_chars` (default 40k) with an explicit flag; `dump_info` reports `fulltext_current`
+- **Deferred FTS optimize**: `--skip-fts-optimize` skips the single-threaded segment-merge step of the full-text build (the dominant cost on many-core machines: 40-60+ min on a full dump), leaving a fully searchable index; `--fts-optimize` runs the merge later, standalone and idempotent. Optimize state is recorded in index metadata and reported by `dump_info`
+- **Full-text search (Tier 2)**: `--build-index --fulltext` builds a contentless SQLite FTS5 index over cleaned section text (tokenizer auto-selected by language: character-trigram for CJK, unicode61 for space-delimited; `--fts-tokenizer` to override, porter stemming opt-in). Search via `--search` (CLI) or the `search_text` MCP tool: literal phrase or raw FTS5 query modes, composable with category recursion and section filters, `count: "capped"|"exact"` (exact 0 = a verified absence claim for the dump version). Snippets are re-rendered from the dump on demand — the index stores no text, keeping disk cost to the inverted index only
+- **Background extraction jobs**: `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` MCP tools for extractions beyond the 5000-article synchronous cap. Each job runs in its own thread with isolated resources; extraction now streams to disk in batches (memory-safe at any scale)
+- **RAG chunking**: `extract_corpus` accepts `chunk_size` / `chunk_overlap`, emitting one record per chunk with `section_path`, `chunk_index`, and `chunk_count`; chunk boundaries prefer sentence/paragraph breaks
+- **Alias guardrail**: `save_alias_set` re-checks co-occurrence server-side and refuses groups containing frequently-coexisting heading pairs (likely distinct roles, not synonyms) unless `force` is passed — protocol compliance no longer depends on the calling model's discipline
+- **`discover_aliases` MCP prompt**: bundled recipe walking any agent through the alias verification protocol (discover via `section_stats` → verify via `section_cooccurrence` → sample borderline cases → save)
+- **MCP server (`wp2txt-mcp`)**: New binary exposing a local dump to LLM agents via the Model Context Protocol (stdio). Tools: `dump_info`, `get_article`, `get_sections`, `list_headings`, `find_articles`, `category_tree`, `section_stats`, `section_cooccurrence`, `save_alias_set`/`get_alias_set`/`list_alias_sets`, and `extract_corpus` (writes JSONL + reproducibility `.meta.json` sidecar; returns summary and sample only). Requires the `mcp` gem (`gem install mcp`); wp2txt itself does not depend on it
+- **LLM-generated section aliases**: Instead of shipping per-language alias dictionaries, agents discover real heading usage with `section_stats`, verify synonym hypotheses with `section_cooccurrence` (synonymous headings almost never co-occur in one article), and persist named alias sets per dump with `save_alias_set`. Queries reference them via `alias_set`; extraction metadata records the exact set contents used
+- **`Wp2txt::Corpus` facade**: Shared query/extraction layer used by the MCP server; lazy SQLite-backed title lookup avoids loading multi-million-entry indexes into memory
+- **Local metadata index (`--build-index`)**: New offline index built by scanning a multistream dump in parallel. Stores per-article categories, section headings, redirects, and the category hierarchy in SQLite (`~/.wp2txt/cache/*_meta.sqlite3`), keyed to the dump version. No API access required
+- **Offline exhaustive queries (`--find-articles`)**: List articles matching `--in-category` (recursive via `--depth`, powered by dump-derived category hierarchy), `--has-section` (alias-aware), and `--title-match` filters. Supports `--limit` and JSON output (`-j json`); redirects are excluded automatically. Enables queries like "all film articles that have a Plot section" against a version-pinned local dump
+
 ## [2.1.2] - 2026-07-19
 
 - **Fixed gem file permissions**: The published gem contained files with owner-only (0600) permissions inherited from the build machine, making them unreadable after `sudo gem install`. A `normalize_permissions` task now runs before `rake build`, ensuring all packaged files are world-readable (0644, or 0755 for executables)

data/Dockerfile

@@ -1,5 +1,8 @@
 FROM ruby:4.0.1-alpine3.23
 
+# Link the GHCR package to the source repository
+LABEL org.opencontainers.image.source="https://github.com/yohasebe/wp2txt"
+
 WORKDIR /wp2txt
 COPY . ./
 RUN rm -f Gemfile.lock

data/Gemfile

@@ -3,3 +3,5 @@
 source "https://rubygems.org"
 
 gemspec
+
+gem "mcp", "~> 0.25.0"

data/README.md

@@ -34,15 +34,19 @@ WP2TXT extracts plain text and category information from Wikipedia dump files. I
 - **Multilingual support** - Category and redirect detection for 350+ Wikipedia languages
 - **Streaming processing** - Process large dumps without intermediate files
 - **JSON output** - Machine-readable JSONL format for data pipelines
+- **Offline research indexes** - Exhaustive, version-pinned queries over categories, section headings, and full text; interlanguage links for cross-edition SQL
+- **MCP server** - Expose a local dump to LLM agents (Claude, ChatGPT, Gemini, local models) for grounded, reproducible corpus work
 
 ## Use Cases
 
 wp2txt is particularly suited for:
 
-- Building domain-specific corpora using category information
+- Building domain-specific corpora using category information (e.g., "the plot sections of all film articles")
 - Comparative linguistic research across topic areas
 - Extracting Wikipedia text with metadata for NLP tasks
 - Cross-linguistic studies using parallel category structures
+- Version-pinned RAG knowledge bases and LLM evaluation datasets
+- Exhaustive claims about a Wikipedia edition ("no article in this category mentions X") that web search cannot make
 
 ## Data Access
 
@@ -70,12 +74,30 @@ On Windows: Install [Bzip2 for Windows](http://gnuwin32.sourceforge.net/packages
 
 ### Docker (Alternative)
 
+The Docker image bundles everything — Ruby, decompression tools, and the MCP server — so no local Ruby setup is needed.
+
+**Interactive / batch CLI:**
+
 ```shell
-docker run -it -v /path/to/localdata:/data yohasebe/wp2txt
+docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 ```
 
 The `wp2txt` command is available inside the container. Use `/data` for input/output files.
 
+Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`); Docker Hub (`yohasebe/wp2txt`) is maintained as a mirror.
+
+**MCP server (no Ruby required on the host):**
+
+```shell
+# Build the indexes once (cached in a named volume so they persist)
+docker run -it -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt --build-index --fulltext -L ja
+
+# Register with an MCP client, e.g. Claude Code
+claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt-mcp -L ja
+```
+
+Note: use `-i` (not `-it`) when running the MCP server — a TTY would corrupt the JSON-RPC stream. The named volume (`wp2txt:`) holds downloaded dumps and indexes; without it they are lost when the container exits.
+
 ## Basic Usage
 
 ### Auto-download and process (Recommended)
@@ -233,6 +255,29 @@ By default, citation templates are removed. Use `--extract-citations` to extract
 
 Supported: `{{cite book}}`, `{{cite web}}`, `{{cite news}}`, `{{cite journal}}`, `{{Citation}}`, etc.
 
+## Research Infrastructure (Indexes, Exhaustive Queries, MCP)
+
+Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned research
+database**: SQLite indexes over categories, section headings, redirects, and (optionally)
+the full article text, plus interlanguage links for cross-edition comparison — all
+queryable offline, exhaustively, and exposed to LLM agents via an MCP server.
+
+```console
+$ wp2txt --build-index --fulltext --lang=ja      # build the indexes
+$ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
+$ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
+$ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
+$ wp2txt-mcp --lang=ja                           # stdio MCP server for LLM agents
+```
+
+Unlike web/API access, these queries scan every article (a `0 matches` result is a
+verifiable absence claim for that dump version) and are reproducible: extractions record
+the dump version and query in a `.meta.json` sidecar.
+
+**→ See the [Research Infrastructure Guide](docs/RESEARCH.md)** for index building,
+exhaustive queries, full-text search, interlanguage links, cross-language SQL, the full
+MCP tool list, and design principles.
+
 ## Command Line Options
 
     Usage: wp2txt [options]
@@ -306,6 +351,15 @@ Supported: `{{cite book}}`, `{{cite web}}`, `{{cite news}}`, `{{cite journal}}`,
       -v, --version                    Print version
       -h, --help                       Show help
 
+### Research infrastructure options
+
+    --build-index                    Build the metadata index (add --fulltext for FTS)
+    --find-articles / --search       Exhaustive offline queries (see docs/RESEARCH.md)
+    --import-langlinks               Import interlanguage links (version-matched)
+    --fts-optimize                   Optimize an existing full-text index
+
+See the [Research Infrastructure Guide](docs/RESEARCH.md) for details.
+
 ## Configuration File
 
 Create persistent settings with:
@@ -347,6 +401,10 @@ Turbo mode (default) splits bz2 into XML chunks first, then processes in paralle
 
 See [CHANGELOG.md](CHANGELOG.md) for detailed release notes.
 
+**v2.3.0 (July 2026)**: Interlanguage links import, cross-language SQL (multi-dump ATTACH), explicit-title extraction, SQL results to file with reproducibility sidecars, GHCR image publishing.
+
+**v2.2.0 (July 2026)**: Offline metadata index, FTS5 full-text search, MCP server, query_sql escape hatch, extraction jobs.
+
 **v2.1.0 (February 2026)**: SQLite caching, Ractor parallelism (Ruby 4.0+), template expansion, content markers, Docker image update.
 
 **v2.0.0 (January 2026)**: Auto-download mode, category-based extraction, article extraction by title, JSON output, streaming processing, Ruby 4.0 support.

data/Rakefile

@@ -39,7 +39,10 @@ task :push do
       # docker buildx create --name mybuilder
       # docker buildx use mybuilder
       # docker buildx inspect --bootstrap
-      docker buildx build --platform linux/amd64,linux/arm64 -t yohasebe/wp2txt:#{Wp2txt::VERSION} -t yohasebe/wp2txt:latest . --push
+      docker buildx build --platform linux/amd64,linux/arm64 \
+        -t ghcr.io/yohasebe/wp2txt:#{Wp2txt::VERSION} -t ghcr.io/yohasebe/wp2txt:latest \
+        -t yohasebe/wp2txt:#{Wp2txt::VERSION} -t yohasebe/wp2txt:latest \
+        . --push
     BASH
   SCRIPT
 end

data/bin/wp2txt

@@ -14,6 +14,7 @@ require_relative "../lib/wp2txt/cli_ui"
 require_relative "../lib/wp2txt/formatter"
 require_relative "../lib/wp2txt/extractor"
 require_relative "../lib/wp2txt/ractor_worker"
+require_relative "../lib/wp2txt/index_commands"
 
 require "etc"
 require "json"
@@ -28,6 +29,7 @@ class WpApp
   include Wp2txt::CliUI
   include Wp2txt::Formatter
   include Wp2txt::Extractor
+  include Wp2txt::IndexCommands
 
   # Debug mode flag
   DEBUG_MODE = false
@@ -704,6 +706,13 @@ class WpApp
       return EXIT_SUCCESS
     end
 
+    # Metadata index operations (offline exhaustive queries)
+    return run_build_index(opts) if opts[:build_index]
+    return run_find_articles(opts) if opts[:find_articles]
+    return run_search(opts) if opts[:search]
+    return run_fts_optimize(opts) if opts[:fts_optimize]
+    return run_import_langlinks(opts) if opts[:import_langlinks]
+
     # Determine input source
     if opts[:from_category] && opts[:lang]
       # Category extraction mode

data/bin/wp2txt-mcp

@@ -0,0 +1,395 @@
+#!/usr/bin/env ruby
+
+# frozen_string_literal: true
+
+# MCP (Model Context Protocol) server exposing a local Wikipedia dump for
+# exhaustive, version-pinned queries and corpus extraction.
+#
+# Usage:
+#   wp2txt-mcp --lang ja                 # use the cached dump for a language
+#   wp2txt-mcp --input dump.xml.bz2      # use an explicit multistream file
+#
+# Prerequisite: wp2txt --build-index -L ja  (downloads dump + builds metadata index)
+#
+# All logging goes to stderr; stdout carries JSON-RPC only.
+
+begin
+  require "mcp"
+rescue LoadError
+  abort "wp2txt-mcp requires the 'mcp' gem (not installed by default).\n" \
+        "Install it with: gem install mcp"
+end
+
+require "optimist"
+require "json"
+require_relative "../lib/wp2txt/corpus"
+require_relative "../lib/wp2txt/corpus_jobs"
+require_relative "../lib/wp2txt/output_path"
+require_relative "../lib/wp2txt/version"
+
+opts = Optimist.options do
+  version Wp2txt::VERSION
+  banner "wp2txt-mcp: MCP server for offline Wikipedia dump queries\n\nOptions:"
+  opt :lang, "Wikipedia language code of a cached dump (e.g., ja, en)", type: String, short: "-L"
+  opt :input, "Path to a multistream .xml.bz2 file", type: String, short: "-i"
+  opt :cache_dir, "Cache directory", type: String, default: File.expand_path("~/.wp2txt/cache")
+  opt :output_dir, "Default directory for extract_corpus output files", type: String, default: Dir.pwd
+end
+
+Optimist.die "Either --lang or --input is required" if opts[:lang].nil? && opts[:input].nil?
+Optimist.die "Cannot specify both --lang and --input" if opts[:lang] && opts[:input]
+
+corpus_factory = lambda do
+  if opts[:lang]
+    Wp2txt::Corpus.for_lang(opts[:lang], cache_dir: opts[:cache_dir])
+  else
+    Wp2txt::Corpus.for_input(File.expand_path(opts[:input]), cache_dir: opts[:cache_dir])
+  end
+end
+
+begin
+  corpus = corpus_factory.call
+rescue ArgumentError => e
+  abort e.message
+end
+
+job_manager = Wp2txt::CorpusJobManager.new(corpus_factory)
+
+unless corpus.metadata_built?
+  abort "Metadata index not built for this dump.\n" \
+        "Build it first with: wp2txt --build-index #{opts[:lang] ? "-L #{opts[:lang]}" : "-i #{opts[:input]}"}"
+end
+
+info = corpus.dump_info
+warn "wp2txt-mcp #{Wp2txt::VERSION} | dump: #{info[:dump]} | articles: #{info[:stats]&.dig(:article_count)}"
+warn "NOTE: metadata index is stale (dump file changed); consider re-running --build-index" unless info[:metadata_current]
+
+OUTPUT_DIR = opts[:output_dir]
+
+server = MCP::Server.new(name: "wp2txt", version: Wp2txt::VERSION)
+
+# Wrap a tool body: render result as JSON text, convert errors to tool errors
+def respond(&block)
+  result = block.call
+  MCP::Tool::Response.new([{ type: "text", text: JSON.generate(result) }])
+rescue ArgumentError => e
+  MCP::Tool::Response.new([{ type: "text", text: JSON.generate({ error: e.message }) }], error: true)
+rescue StandardError => e
+  MCP::Tool::Response.new([{ type: "text", text: JSON.generate({ error: "#{e.class}: #{e.message}" }) }], error: true)
+end
+
+server.define_tool(
+  name: "dump_info",
+  description: "Dump identity (language/date), available index tiers, and article/category/section counts. Call first to learn what this corpus contains.",
+  input_schema: { properties: {}, required: [] }
+) do |server_context:|
+  respond { corpus.dump_info }
+end
+
+server.define_tool(
+  name: "get_article",
+  description: "Fetch one article by title from the local dump (titles are normalized: underscores, capitalization; one redirect hop is followed). format: 'text' (cleaned plain text, default) or 'wikitext' (raw markup, includes infoboxes/templates). Long articles are truncated at max_chars (default 40000) with truncated:true and total_chars — raise max_chars only when you need more. If not found, retry via find_articles with title_match to locate the exact title.",
+  input_schema: {
+    properties: {
+      title: { type: "string" },
+      format: { type: "string", enum: ["text", "wikitext"] },
+      max_chars: { type: "integer", minimum: 1000 }
+    },
+    required: ["title"]
+  }
+) do |title:, format: "text", max_chars: 40_000, server_context:|
+  respond { corpus.get_article(title, format: format, max_chars: max_chars) || { error: "article not found: #{title}" } }
+end
+
+server.define_tool(
+  name: "get_categories",
+  description: "List the categories of one article (from the local index). Useful for building precise filters: inspect a few known-good articles' categories, then query find_articles with categories/category_match.",
+  input_schema: { properties: { title: { type: "string" } }, required: ["title"] }
+) do |title:, server_context:|
+  respond { corpus.get_categories(title) || { error: "article not found: #{title}" } }
+end
+
+server.define_tool(
+  name: "get_sections",
+  description: "Extract specific sections from one article by heading names. Use 'summary' for the lead text before the first heading. Optionally expand names through a saved alias_set.",
+  input_schema: {
+    properties: {
+      title: { type: "string" },
+      sections: { type: "array", items: { type: "string" } },
+      alias_set: { type: "string" }
+    },
+    required: ["title", "sections"]
+  }
+) do |title:, sections:, alias_set: nil, server_context:|
+  respond { corpus.get_sections(title, sections, alias_set: alias_set) || { error: "article not found: #{title}" } }
+end
+
+server.define_tool(
+  name: "list_headings",
+  description: "List the section heading tree (name + level) of one article. Useful before get_sections to see what headings exist.",
+  input_schema: { properties: { title: { type: "string" } }, required: ["title"] }
+) do |title:, server_context:|
+  respond { corpus.list_headings(title) || { error: "article not found: #{title}" } }
+end
+
+server.define_tool(
+  name: "find_articles",
+  description: "Exhaustively list articles matching filters, all AND-combined: category (single, with subcategory recursion via depth; NOTE category trees over-include, so dry-run with a small limit first), categories (exact multi-category intersection, e.g. year AND country), category_match (substring on category names, e.g. 'アメリカ合衆国の' matched against every category of the article), sections (article contains at least one of these headings; expanded via alias_set if given), title_match (title substring). Returns total count plus up to `limit` titles. Redirects are excluded.",
+  input_schema: {
+    properties: {
+      category: { type: "string" },
+      depth: { type: "integer", minimum: 0, maximum: 5 },
+      categories: { type: "array", items: { type: "string" } },
+      category_match: { type: "string" },
+      sections: { type: "array", items: { type: "string" } },
+      alias_set: { type: "string" },
+      title_match: { type: "string" },
+      limit: { type: "integer", minimum: 0 },
+      offset: { type: "integer", minimum: 0 }
+    },
+    required: []
+  }
+) do |category: nil, depth: 0, categories: nil, category_match: nil, sections: nil, alias_set: nil, title_match: nil, limit: 100, offset: 0, server_context:|
+  respond do
+    corpus.find_articles(category: category, depth: depth, categories: categories,
+                         category_match: category_match, sections: sections,
+                         alias_set: alias_set, title_match: title_match,
+                         limit: limit, offset: offset)
+  end
+end
+
+server.define_tool(
+  name: "query_sql",
+  description: "Read-only SQL (SELECT/WITH only) against the index databases — the escape hatch for questions the fixed tools cannot express (multi-way intersections, aggregations, joins). Call describe_schema first to see the tables. Main DB: pages(page_id, title, namespace, redirect_to, text_length), page_categories(page_id, category), page_sections(page_id, heading, level, ord), category_hierarchy(child, parent), alias_sets; a langlinks table (ll_from = source page_id, ll_lang, ll_title) is also present when interlanguage links were imported via wp2txt --import-langlinks. When the fulltext tier exists, the FTS DB is attached as `fts` (fts.fts_sections MATCH ..., fts.fts_map). Cross-language: pass attach: [\"en\",\"de\"] to read-only ATTACH other locally installed languages' indexes, referenced as {lang}_meta.pages / {lang}_meta.page_sections / ... and {lang}_fts when that language has a fulltext tier — attached DBs share the main DB's schema. Hyphenated language codes map to underscored aliases (e.g. zh-yue → zh_yue_meta). Only installed languages are accepted (the main language itself cannot be attached); the response's `attached` field lists what was attached (dump_name, fts, dump_mismatch when the dates differ). Join langlinks (ll_from = page_id) to map articles across languages. Filter articles with namespace = 0 AND redirect_to IS NULL. Rows are capped by `limit` (default 200, max 1000) with truncated flag; long cells are clipped. output_path: write ALL rows to a JSONL file (with a .meta.json recording the SQL, dump versions, and attach configuration) and return only a summary + 3-row sample — use this whenever the result is large; `limit` is ignored in this mode. Queries are killed after 30 seconds (the error explains the likely cause); the tables are large (millions of rows), so constrain joins with indexed keys (page_id) and add WHERE filters.",
+  input_schema: {
+    properties: {
+      sql: { type: "string" },
+      limit: { type: "integer", minimum: 1, maximum: 1000 },
+      attach: { type: "array", items: { type: "string" },
+                description: "Language codes of other locally installed dumps to ATTACH read-only as {lang}_meta / {lang}_fts (default: none)" },
+      output_path: { type: "string", description: "JSONL output path for ALL rows; relative paths resolve under the server's output dir" },
+      overwrite: { type: "boolean", description: "Replace an existing output file (default: refuse)" }
+    },
+    required: ["sql"]
+  }
+) do |sql:, limit: 200, attach: [], output_path: nil, overwrite: false, server_context:|
+  respond do
+    path = output_path && Wp2txt::OutputPath.confine(output_path, OUTPUT_DIR, overwrite: overwrite)
+    corpus.query_sql(sql, limit: limit, attach: attach, output_path: path, overwrite: overwrite)
+  end
+end
+
+server.define_tool(
+  name: "describe_schema",
+  description: "CREATE statements of all tables available to query_sql (metadata DB, plus the attached fts DB when built).",
+  input_schema: { properties: {}, required: [] }
+) do |server_context:|
+  respond { corpus.describe_schema }
+end
+
+server.define_tool(
+  name: "category_tree",
+  description: "Subcategory tree under a category (names + depth). Use to understand scope before recursive find_articles.",
+  input_schema: {
+    properties: { category: { type: "string" }, depth: { type: "integer", minimum: 1, maximum: 5 } },
+    required: ["category"]
+  }
+) do |category:, depth: 2, server_context:|
+  respond { corpus.category_tree(category, depth: depth) }
+end
+
+server.define_tool(
+  name: "section_stats",
+  description: "Section heading frequencies across articles, optionally scoped to a category (+depth). This is the primary tool for discovering how sections are actually named in this language edition — use it before assuming heading names, and as the raw material for building alias sets.",
+  input_schema: {
+    properties: {
+      category: { type: "string" },
+      depth: { type: "integer", minimum: 0, maximum: 5 },
+      top_n: { type: "integer", minimum: 1, maximum: 500 }
+    },
+    required: []
+  }
+) do |category: nil, depth: 0, top_n: 50, server_context:|
+  respond { corpus.section_stats(category: category, depth: depth, top_n: top_n) }
+end
+
+server.define_tool(
+  name: "section_cooccurrence",
+  description: "Test whether headings are plausible aliases: for each heading, article count and average position; for each pair, how many articles contain BOTH. Synonymous headings almost never co-occur in one article, so a high cooccurrence_ratio is evidence AGAINST grouping them as aliases. Use to verify a hypothesis before save_alias_set.",
+  input_schema: {
+    properties: {
+      headings: { type: "array", items: { type: "string" }, minItems: 2 },
+      category: { type: "string" },
+      depth: { type: "integer", minimum: 0, maximum: 5 }
+    },
+    required: ["headings"]
+  }
+) do |headings:, category: nil, depth: 0, server_context:|
+  respond { corpus.section_cooccurrence(headings, category: category, depth: depth) }
+end
+
+server.define_tool(
+  name: "search_text",
+  description: "Exhaustive full-text search over cleaned article text (requires the fulltext index tier — check dump_info). mode: 'phrase' (literal string, default; 3+ chars for CJK dumps) or 'query' (raw FTS5 syntax: AND/OR/NEAR/quotes). Composable with category (recursive via depth) and sections filters (alias_set-aware). count: 'capped' (fast, default; total_is_capped signals more matches exist) or 'exact' (may take seconds for very common terms, but gives the true exhaustive count — required for absence/quantification claims). Snippets are re-rendered from the dump. Total of 0 with count 'exact' is a verified absence claim for this dump version.",
+  input_schema: {
+    properties: {
+      query: { type: "string" },
+      mode: { type: "string", enum: ["phrase", "query"] },
+      sections: { type: "array", items: { type: "string" } },
+      alias_set: { type: "string" },
+      category: { type: "string" },
+      depth: { type: "integer", minimum: 0, maximum: 5 },
+      limit: { type: "integer", minimum: 1, maximum: 100 },
+      offset: { type: "integer", minimum: 0 },
+      count: { type: "string", enum: ["capped", "exact"] }
+    },
+    required: ["query"]
+  }
+) do |query:, mode: "phrase", sections: nil, alias_set: nil, category: nil, depth: 0, limit: 20, offset: 0, count: "capped", server_context:|
+  respond do
+    corpus.search_text(query, mode: mode, sections: sections, alias_set: alias_set,
+                       category: category, depth: depth, limit: limit, offset: offset, count: count)
+  end
+end
+
+server.define_tool(
+  name: "save_alias_set",
+  description: "Persist a named set of section-heading alias groups for this dump (e.g. [[\"あらすじ\",\"ストーリー\",\"物語\"]]). A server-side guardrail re-checks co-occurrence: a pair that appears in the same article more than 20% of the time (when both headings occur in at least 100 articles) is treated as distinct sections, not synonyms, and blocks the save with saved:false and the offending pairs listed — fix the groups, or pass force:true only if you verified them another way (e.g., by reading section contents with get_sections). The response echoes the saved groups and the exact criteria used. Saved sets are referenced via alias_set in find_articles / get_sections / extract_corpus and recorded in extraction metadata for reproducibility.",
+  input_schema: {
+    properties: {
+      name: { type: "string" },
+      groups: { type: "array", items: { type: "array", items: { type: "string" } } },
+      force: { type: "boolean" }
+    },
+    required: ["name", "groups"]
+  }
+) do |name:, groups:, force: false, server_context:|
+  respond { corpus.save_alias_set(name, groups, force: force) }
+end
+
+server.define_tool(
+  name: "get_alias_set",
+  description: "Retrieve a saved alias set by name (groups + creation time).",
+  input_schema: { properties: { name: { type: "string" } }, required: ["name"] }
+) do |name:, server_context:|
+  respond { corpus.get_alias_set(name) || { error: "alias set not found: #{name}" } }
+end
+
+server.define_tool(
+  name: "list_alias_sets",
+  description: "List all alias sets saved for this dump.",
+  input_schema: { properties: {}, required: [] }
+) do |server_context:|
+  respond { { alias_sets: corpus.list_alias_sets } }
+end
+
+EXTRACT_SCHEMA = {
+  properties: {
+    output_path: { type: "string", description: "JSONL output path; relative paths resolve under the server's output dir" },
+    content: { type: "string", enum: ["sections", "summary", "full", "wikitext"] },
+    titles: { type: "array", items: { type: "string" }, maxItems: 10_000,
+              description: "Explicit article titles to extract (mutually exclusive with the filter arguments)" },
+    sections: { type: "array", items: { type: "string" } },
+    alias_set: { type: "string" },
+    category: { type: "string" },
+    depth: { type: "integer", minimum: 0, maximum: 5 },
+    categories: { type: "array", items: { type: "string" } },
+    category_match: { type: "string" },
+    title_match: { type: "string" },
+    limit: { type: "integer", minimum: 0 },
+    chunk_size: { type: "integer", minimum: 100, description: "Split text into ~N-char chunks (one RAG-ready record per chunk)" },
+    chunk_overlap: { type: "integer", minimum: 0 },
+    overwrite: { type: "boolean", description: "Replace an existing output file (default: refuse)" }
+  },
+  required: ["output_path", "content"]
+}.freeze
+
+def extract_params(output_path:, content:, titles: nil, sections: nil, alias_set: nil, category: nil,
+                   depth: 0, categories: nil, category_match: nil,
+                   title_match: nil, limit: 0, chunk_size: nil, chunk_overlap: 0,
+                   overwrite: false)
+  { output_path: Wp2txt::OutputPath.confine(output_path, OUTPUT_DIR, overwrite: overwrite),
+    content: content, titles: titles,
+    sections: sections, alias_set: alias_set, category: category, depth: depth,
+    categories: categories, category_match: category_match,
+    title_match: title_match, limit: limit, chunk_size: chunk_size, chunk_overlap: chunk_overlap }
+end
+
+server.define_tool(
+  name: "extract_corpus",
+  description: "Extract matching articles to a JSONL file on disk (with a .meta.json sidecar recording dump version and query for reproducibility). Returns a summary and the first 3 records only — the full corpus stays on disk; process it with your own tools. content: 'sections' (named sections only), 'summary' (lead text), 'full' (whole cleaned article), or 'wikitext' (raw markup, for mining infoboxes/templates that cleaning replaces with markers). Filters are the same as find_articles (category/depth, categories, category_match, title_match). titles: extract an explicit list of articles (e.g. a set determined via query_sql); mutually exclusive with the filter arguments; redirects are resolved; missing titles are reported in not_found. Optional chunk_size/chunk_overlap produce one RAG-ready record per chunk with section_path and chunk_index (not with wikitext). Synchronous, capped at 5000 articles (result flags truncation) — use start_extract_job for larger extractions. Dry-run the filters with find_articles first.",
+  input_schema: EXTRACT_SCHEMA
+) do |server_context:, **args|
+  respond { corpus.extract_corpus(**extract_params(**args)) }
+end
+
+server.define_tool(
+  name: "start_extract_job",
+  description: "Start extract_corpus as a background job (same parameters, no article cap). Returns job_id immediately; poll with job_status. Use for extractions larger than the 5000-article synchronous cap. Jobs do not survive server restart.",
+  input_schema: EXTRACT_SCHEMA
+) do |server_context:, **args|
+  respond { job_manager.start_extract(extract_params(**args)) }
+end
+
+server.define_tool(
+  name: "job_status",
+  description: "Status of a background extraction job: running/completed/cancelled/error, progress (0-1), and on completion the same summary extract_corpus returns (output path, counts, sample).",
+  input_schema: { properties: { job_id: { type: "string" } }, required: ["job_id"] }
+) do |job_id:, server_context:|
+  respond { job_manager.status(job_id) || { error: "unknown job: #{job_id}" } }
+end
+
+server.define_tool(
+  name: "cancel_job",
+  description: "Request cancellation of a running job (takes effect at the next batch boundary; the partial output file remains on disk).",
+  input_schema: { properties: { job_id: { type: "string" } }, required: ["job_id"] }
+) do |job_id:, server_context:|
+  respond { job_manager.cancel(job_id) || { error: "unknown job: #{job_id}" } }
+end
+
+server.define_tool(
+  name: "list_jobs",
+  description: "List all extraction jobs started in this server session.",
+  input_schema: { properties: {}, required: [] }
+) do |server_context:|
+  respond { { jobs: job_manager.list } }
+end
+
+server.define_prompt(
+  name: "discover_aliases",
+  description: "Recipe for building a verified section-alias set for a concept (e.g. plot sections of film articles) in this dump's language, using section_stats, section_cooccurrence, and content sampling.",
+  arguments: [
+    MCP::Prompt::Argument.new(name: "concept", description: "The section concept to cover (e.g. 'plot/story summary of a film')", required: true),
+    MCP::Prompt::Argument.new(name: "category", description: "Category scope (e.g. '映画作品')", required: false),
+    MCP::Prompt::Argument.new(name: "depth", description: "Category recursion depth (default 2)", required: false)
+  ]
+) do |args, server_context:|
+  concept = args[:concept] || args["concept"]
+  category = args[:category] || args["category"]
+  depth = args[:depth] || args["depth"] || 2
+  scope = category ? "category \"#{category}\" (depth #{depth})" : "the whole dump"
+  MCP::Prompt::Result.new(
+    description: "Alias discovery protocol for: #{concept}",
+    messages: [
+      MCP::Prompt::Message.new(
+        role: "user",
+        content: MCP::Content::Text.new(<<~TEXT)
+          Build a verified section-heading alias set for the concept "#{concept}" in #{scope}, using this protocol:
+
+          1. DISCOVER: Call section_stats#{category ? " with category=\"#{category}\", depth=#{depth}" : ""} (top_n 50-100). From the real heading frequencies, shortlist headings that could denote "#{concept}" based on their meaning. Note each candidate's article count — it tells you how much coverage each adds.
+          2. VERIFY: Call section_cooccurrence with the shortlisted headings (same scope). Interpret:
+             - ratio below ~0.05 between two FREQUENT headings: strong evidence they fill the same slot (valid aliases) — frequent headings would co-occur by chance if they were independent.
+             - ratio above ~0.15: they coexist in the same articles, so they are different section roles — do NOT group them.
+             - avg_position should also be similar for true aliases.
+          3. SAMPLE (only for borderline ratios of 0.05-0.2 or rare headings where statistics are weak): pick 2-3 articles per heading via find_articles(sections: [heading], limit: 3) and read the content with get_sections. Judge whether the text under each heading is the same KIND of content.
+          4. DECIDE: Groups must contain only verified same-role headings. For middling evidence, present the numbers to the user and ask.
+          5. SAVE: Call save_alias_set with a descriptive versioned name (e.g. "#{category ? category.downcase.gsub(/\s+/, '-') : 'dump'}-#{concept.split.first(2).join('-').downcase}-v1"). The server re-checks co-occurrence and will refuse groups with high-ratio pairs unless you pass force:true with justification.
+          6. REPORT: Tell the user which headings you included and excluded, with the co-occurrence numbers that justified each decision.
+        TEXT
+      )
+    ]
+  )
+end
+
+transport = MCP::Server::Transports::StdioTransport.new(server)
+transport.open