wp2txt 2.1.2 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 03b8bbbab05c9ab7f56ed3f89c00ae14c3259fbb7b00d54010b04a0decb79f16
-  data.tar.gz: dd92bacf8e6f17e9d6d5ad33bb3e9c5777cb59a53f4f150136a2143dd84f5b1d
+  metadata.gz: 5e54ea4f824be0f060036923759b1c6755b20ee1e721245db58a23665b9efaab
+  data.tar.gz: ca0a5b1ae0efae6eb4c485a6319569ea9995c662054e7e697a70429632613a25
 SHA512:
-  metadata.gz: b4d95651f10fd30d3c0d23d2846c60b891716b2638a62151712a28bf6ea1f2a6e0da18abaa5b48553f6d736ba86bfeb8fd978ee1e07facfc45c56256734b239d
-  data.tar.gz: e10dcbcba41561a26df4f5740b6517bc2253fe6226eb68d588c17c9bf00f2ec6b9e41ba6eceeff63be0fc742856eb20c385c290a32f5ccc8d364b110fff7fcdb
+  metadata.gz: 34b61e5ee404431e521e77bc1223b9b7e1e0c9f2082ae4898e6e4b9fdee9de593c9ca45d661723f8d14f5ebffd62360dfb27d3a7b0d87d2a37691bb127c9ab6f
+  data.tar.gz: ca002ba2fc0c5a9bc8b2ae55dbc736f154e54452af681aad6f1b9ed30ca616535db94ee8cccb069defa56aacc4feb96b306d6fdb1a0102ee20611c98a213d5d0

data/CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.0] - 2026-07-22
+
+- **Index hardening (design review follow-up)**: `ord` now has identical semantics in `page_sections` and `fts_map` (lead = 0, first heading = 1; schema v2 — rebuild indexes with `--build-index -U`); section headings are normalized identically in both indexes (decorated headings like `== '''X''' ==` now match section filters); indexes record the wp2txt version (and the FTS index a rendering-config digest) so `dump_info` can flag indexes built by code whose text cleaning differs; rebuilds are atomic (built alongside, renamed on completion — a failed rebuild no longer destroys the working index)
+- **MCP safety hardening**: `query_sql` runs in a killable subprocess with a 30s wall-clock cap and returns a query-plan diagnosis on timeout; the keyword screen no longer rejects legitimate values inside string literals; `extract_corpus` output paths are confined to the server output directory and refuse to overwrite existing files without `overwrite: true`; background jobs are serialized (one at a time) and all SQLite connections are closed before forking extraction workers
+- **Cold-start parity for LLM clients**: closes the gaps that previously required internal knowledge to work around. `get_categories` tool; `find_articles`/`extract_corpus` accept `categories` (exact multi-category AND) and `category_match` (substring on category names); `extract_corpus` gains `content: "wikitext"` for structure mining (infoboxes/templates); `query_sql` read-only SQL escape hatch (SELECT/WITH only, keyword-screened, read-only connection, row/cell caps) with `describe_schema`; article titles are normalized like MediaWiki (underscores, capitalization); `get_article` truncates at `max_chars` (default 40k) with an explicit flag; `dump_info` reports `fulltext_current`
+- **Deferred FTS optimize**: `--skip-fts-optimize` skips the single-threaded segment-merge step of the full-text build (the dominant cost on many-core machines: 40-60+ min on a full dump), leaving a fully searchable index; `--fts-optimize` runs the merge later, standalone and idempotent. Optimize state is recorded in index metadata and reported by `dump_info`
+- **Full-text search (Tier 2)**: `--build-index --fulltext` builds a contentless SQLite FTS5 index over cleaned section text (tokenizer auto-selected by language: character-trigram for CJK, unicode61 for space-delimited; `--fts-tokenizer` to override, porter stemming opt-in). Search via `--search` (CLI) or the `search_text` MCP tool: literal phrase or raw FTS5 query modes, composable with category recursion and section filters, `count: "capped"|"exact"` (exact 0 = a verified absence claim for the dump version). Snippets are re-rendered from the dump on demand — the index stores no text, keeping disk cost to the inverted index only
+- **Background extraction jobs**: `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` MCP tools for extractions beyond the 5000-article synchronous cap. Each job runs in its own thread with isolated resources; extraction now streams to disk in batches (memory-safe at any scale)
+- **RAG chunking**: `extract_corpus` accepts `chunk_size` / `chunk_overlap`, emitting one record per chunk with `section_path`, `chunk_index`, and `chunk_count`; chunk boundaries prefer sentence/paragraph breaks
+- **Alias guardrail**: `save_alias_set` re-checks co-occurrence server-side and refuses groups containing frequently-coexisting heading pairs (likely distinct roles, not synonyms) unless `force` is passed — protocol compliance no longer depends on the calling model's discipline
+- **`discover_aliases` MCP prompt**: bundled recipe walking any agent through the alias verification protocol (discover via `section_stats` → verify via `section_cooccurrence` → sample borderline cases → save)
+- **MCP server (`wp2txt-mcp`)**: New binary exposing a local dump to LLM agents via the Model Context Protocol (stdio). Tools: `dump_info`, `get_article`, `get_sections`, `list_headings`, `find_articles`, `category_tree`, `section_stats`, `section_cooccurrence`, `save_alias_set`/`get_alias_set`/`list_alias_sets`, and `extract_corpus` (writes JSONL + reproducibility `.meta.json` sidecar; returns summary and sample only). Requires the `mcp` gem (`gem install mcp`); wp2txt itself does not depend on it
+- **LLM-generated section aliases**: Instead of shipping per-language alias dictionaries, agents discover real heading usage with `section_stats`, verify synonym hypotheses with `section_cooccurrence` (synonymous headings almost never co-occur in one article), and persist named alias sets per dump with `save_alias_set`. Queries reference them via `alias_set`; extraction metadata records the exact set contents used
+- **`Wp2txt::Corpus` facade**: Shared query/extraction layer used by the MCP server; lazy SQLite-backed title lookup avoids loading multi-million-entry indexes into memory
+- **Local metadata index (`--build-index`)**: New offline index built by scanning a multistream dump in parallel. Stores per-article categories, section headings, redirects, and the category hierarchy in SQLite (`~/.wp2txt/cache/*_meta.sqlite3`), keyed to the dump version. No API access required
+- **Offline exhaustive queries (`--find-articles`)**: List articles matching `--in-category` (recursive via `--depth`, powered by dump-derived category hierarchy), `--has-section` (alias-aware), and `--title-match` filters. Supports `--limit` and JSON output (`-j json`); redirects are excluded automatically. Enables queries like "all film articles that have a Plot section" against a version-pinned local dump
+
 ## [2.1.2] - 2026-07-19
 
 - **Fixed gem file permissions**: The published gem contained files with owner-only (0600) permissions inherited from the build machine, making them unreadable after `sudo gem install`. A `normalize_permissions` task now runs before `rake build`, ensuring all packaged files are world-readable (0644, or 0755 for executables)

data/Gemfile

@@ -3,3 +3,5 @@
 source "https://rubygems.org"
 
 gemspec
+
+gem "mcp", "~> 0.25.0"

data/README.md

@@ -34,15 +34,19 @@ WP2TXT extracts plain text and category information from Wikipedia dump files. I
 - **Multilingual support** - Category and redirect detection for 350+ Wikipedia languages
 - **Streaming processing** - Process large dumps without intermediate files
 - **JSON output** - Machine-readable JSONL format for data pipelines
+- **Offline metadata index** - Exhaustive, version-pinned queries over categories and section headings, no API required
+- **MCP server** - Expose a local dump to LLM agents (Claude, ChatGPT, Gemini, local models) for grounded, reproducible corpus work
 
 ## Use Cases
 
 wp2txt is particularly suited for:
 
-- Building domain-specific corpora using category information
+- Building domain-specific corpora using category information (e.g., "the plot sections of all film articles")
 - Comparative linguistic research across topic areas
 - Extracting Wikipedia text with metadata for NLP tasks
 - Cross-linguistic studies using parallel category structures
+- Version-pinned RAG knowledge bases and LLM evaluation datasets
+- Exhaustive claims about a Wikipedia edition ("no article in this category mentions X") that web search cannot make
 
 ## Data Access
 
@@ -70,12 +74,28 @@ On Windows: Install [Bzip2 for Windows](http://gnuwin32.sourceforge.net/packages
 
 ### Docker (Alternative)
 
+The Docker image bundles everything — Ruby, decompression tools, and the MCP server — so no local Ruby setup is needed.
+
+**Interactive / batch CLI:**
+
 ```shell
 docker run -it -v /path/to/localdata:/data yohasebe/wp2txt
 ```
 
 The `wp2txt` command is available inside the container. Use `/data` for input/output files.
 
+**MCP server (no Ruby required on the host):**
+
+```shell
+# Build the indexes once (cached in a named volume so they persist)
+docker run -it -v wp2txt:/root/.wp2txt yohasebe/wp2txt wp2txt --build-index --fulltext -L ja
+
+# Register with an MCP client, e.g. Claude Code
+claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt yohasebe/wp2txt wp2txt-mcp -L ja
+```
+
+Note: use `-i` (not `-it`) when running the MCP server — a TTY would corrupt the JSON-RPC stream. The named volume (`wp2txt:`) holds downloaded dumps and indexes; without it they are lost when the container exits.
+
 ## Basic Usage
 
 ### Auto-download and process (Recommended)
@@ -233,6 +253,64 @@ By default, citation templates are removed. Use `--extract-citations` to extract
 
 Supported: `{{cite book}}`, `{{cite web}}`, `{{cite news}}`, `{{cite journal}}`, `{{Citation}}`, etc.
 
+## Offline Metadata Index
+
+Build a local SQLite index of a dump — per-article categories, section headings, redirects, and the category hierarchy — extracted from the dump itself (no API access, no rate limits, version-pinned for reproducibility):
+
+    $ wp2txt --build-index --lang=ja     # downloads dump if needed; ja: ~15 min, ~1.7 GB index
+
+Then run exhaustive queries entirely offline:
+
+    # All film articles (recursing 3 levels of subcategories) that have a plot section
+    $ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
+
+    # Machine-readable output with total count
+    $ wp2txt --find-articles --in-category "Films" -D 2 -j json --limit 100 --lang=en
+
+Unlike web/API access, these queries are *exhaustive* (they scan every article, not search-ranked results) and *reproducible* (pinned to a specific dump version). Answering "which of the 1.5M articles have X" takes milliseconds once the index is built.
+
+### Full-Text Search
+
+Add `--fulltext` to also build an FTS5 index over the article text itself (tokenizer auto-selected: character-trigram for Japanese/Chinese/Korean, word-based for space-delimited languages):
+
+    $ wp2txt --build-index --fulltext --lang=ja
+    $ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
+
+Search hits report the article, section, and a snippet; totals are exhaustive counts, so `0 matches` is a verifiable absence claim for that dump version. The index is contentless (stores only the inverted index; snippets are re-rendered from the dump), keeping disk overhead moderate.
+
+## MCP Server (LLM Integration)
+
+`wp2txt-mcp` exposes a local dump to LLM agents (Claude, ChatGPT, Gemini, local models — any MCP-capable client) via the Model Context Protocol:
+
+    $ gem install mcp                    # optional dependency, needed only for the server
+    $ wp2txt --build-index --lang=ja     # prerequisite
+    $ wp2txt-mcp --lang=ja               # stdio MCP server
+
+Example client configuration (e.g., Claude Desktop / Claude Code):
+
+```json
+{
+  "mcpServers": {
+    "wp2txt": { "command": "wp2txt-mcp", "args": ["--lang", "ja"] }
+  }
+}
+```
+
+Tools provided:
+
+| Tool | Purpose |
+|------|---------|
+| `dump_info` | Dump version, index tiers, corpus statistics |
+| `get_article` / `get_sections` / `list_headings` | Single-article access (redirect-aware) |
+| `find_articles` | Exhaustive filtered listing (category recursion, section headings, title match) |
+| `category_tree` / `section_stats` | Scope exploration and heading frequency discovery |
+| `section_cooccurrence` | Verify section-alias hypotheses (synonyms rarely co-occur in one article) |
+| `save_alias_set` etc. | Persist verified per-dump alias groups (server-side co-occurrence guardrail) |
+| `extract_corpus` | Filtered extraction to JSONL + reproducibility sidecar; optional RAG chunking |
+| `start_extract_job` / `job_status` / `cancel_job` | Background jobs for large extractions |
+
+Design principles: large results go to disk (the model receives a summary plus a 3-record sample, never the full corpus); every output records the dump version and query in a `.meta.json` sidecar; section aliases are discovered from real dump statistics by the LLM and mechanically verified — no hand-maintained per-language dictionaries. The bundled `discover_aliases` prompt walks any agent through the verification protocol.
+
 ## Command Line Options
 
     Usage: wp2txt [options]

data/bin/wp2txt

@@ -14,6 +14,7 @@ require_relative "../lib/wp2txt/cli_ui"
 require_relative "../lib/wp2txt/formatter"
 require_relative "../lib/wp2txt/extractor"
 require_relative "../lib/wp2txt/ractor_worker"
+require_relative "../lib/wp2txt/index_commands"
 
 require "etc"
 require "json"
@@ -28,6 +29,7 @@ class WpApp
   include Wp2txt::CliUI
   include Wp2txt::Formatter
   include Wp2txt::Extractor
+  include Wp2txt::IndexCommands
 
   # Debug mode flag
   DEBUG_MODE = false
@@ -704,6 +706,12 @@ class WpApp
       return EXIT_SUCCESS
     end
 
+    # Metadata index operations (offline exhaustive queries)
+    return run_build_index(opts) if opts[:build_index]
+    return run_find_articles(opts) if opts[:find_articles]
+    return run_search(opts) if opts[:search]
+    return run_fts_optimize(opts) if opts[:fts_optimize]
+
     # Determine input source
     if opts[:from_category] && opts[:lang]
       # Category extraction mode

data/bin/wp2txt-mcp

@@ -0,0 +1,396 @@
+#!/usr/bin/env ruby
+
+# frozen_string_literal: true
+
+# MCP (Model Context Protocol) server exposing a local Wikipedia dump for
+# exhaustive, version-pinned queries and corpus extraction.
+#
+# Usage:
+#   wp2txt-mcp --lang ja                 # use the cached dump for a language
+#   wp2txt-mcp --input dump.xml.bz2      # use an explicit multistream file
+#
+# Prerequisite: wp2txt --build-index -L ja  (downloads dump + builds metadata index)
+#
+# All logging goes to stderr; stdout carries JSON-RPC only.
+
+begin
+  require "mcp"
+rescue LoadError
+  abort "wp2txt-mcp requires the 'mcp' gem (not installed by default).\n" \
+        "Install it with: gem install mcp"
+end
+
+require "optimist"
+require "json"
+require_relative "../lib/wp2txt/corpus"
+require_relative "../lib/wp2txt/corpus_jobs"
+require_relative "../lib/wp2txt/version"
+
+opts = Optimist.options do
+  version Wp2txt::VERSION
+  banner "wp2txt-mcp: MCP server for offline Wikipedia dump queries\n\nOptions:"
+  opt :lang, "Wikipedia language code of a cached dump (e.g., ja, en)", type: String, short: "-L"
+  opt :input, "Path to a multistream .xml.bz2 file", type: String, short: "-i"
+  opt :cache_dir, "Cache directory", type: String, default: File.expand_path("~/.wp2txt/cache")
+  opt :output_dir, "Default directory for extract_corpus output files", type: String, default: Dir.pwd
+end
+
+Optimist.die "Either --lang or --input is required" if opts[:lang].nil? && opts[:input].nil?
+Optimist.die "Cannot specify both --lang and --input" if opts[:lang] && opts[:input]
+
+corpus_factory = lambda do
+  if opts[:lang]
+    Wp2txt::Corpus.for_lang(opts[:lang], cache_dir: opts[:cache_dir])
+  else
+    Wp2txt::Corpus.for_input(File.expand_path(opts[:input]), cache_dir: opts[:cache_dir])
+  end
+end
+
+begin
+  corpus = corpus_factory.call
+rescue ArgumentError => e
+  abort e.message
+end
+
+job_manager = Wp2txt::CorpusJobManager.new(corpus_factory)
+
+unless corpus.metadata_built?
+  abort "Metadata index not built for this dump.\n" \
+        "Build it first with: wp2txt --build-index #{opts[:lang] ? "-L #{opts[:lang]}" : "-i #{opts[:input]}"}"
+end
+
+info = corpus.dump_info
+warn "wp2txt-mcp #{Wp2txt::VERSION} | dump: #{info[:dump]} | articles: #{info[:stats]&.dig(:article_count)}"
+warn "NOTE: metadata index is stale (dump file changed); consider re-running --build-index" unless info[:metadata_current]
+
+OUTPUT_DIR = opts[:output_dir]
+
+server = MCP::Server.new(name: "wp2txt", version: Wp2txt::VERSION)
+
+# Wrap a tool body: render result as JSON text, convert errors to tool errors
+def respond(&block)
+  result = block.call
+  MCP::Tool::Response.new([{ type: "text", text: JSON.generate(result) }])
+rescue ArgumentError => e
+  MCP::Tool::Response.new([{ type: "text", text: JSON.generate({ error: e.message }) }], error: true)
+rescue StandardError => e
+  MCP::Tool::Response.new([{ type: "text", text: JSON.generate({ error: "#{e.class}: #{e.message}" }) }], error: true)
+end
+
+server.define_tool(
+  name: "dump_info",
+  description: "Dump identity (language/date), available index tiers, and article/category/section counts. Call first to learn what this corpus contains.",
+  input_schema: { properties: {}, required: [] }
+) do |server_context:|
+  respond { corpus.dump_info }
+end
+
+server.define_tool(
+  name: "get_article",
+  description: "Fetch one article by title from the local dump (titles are normalized: underscores, capitalization; one redirect hop is followed). format: 'text' (cleaned plain text, default) or 'wikitext' (raw markup, includes infoboxes/templates). Long articles are truncated at max_chars (default 40000) with truncated:true and total_chars — raise max_chars only when you need more. If not found, retry via find_articles with title_match to locate the exact title.",
+  input_schema: {
+    properties: {
+      title: { type: "string" },
+      format: { type: "string", enum: ["text", "wikitext"] },
+      max_chars: { type: "integer", minimum: 1000 }
+    },
+    required: ["title"]
+  }
+) do |title:, format: "text", max_chars: 40_000, server_context:|
+  respond { corpus.get_article(title, format: format, max_chars: max_chars) || { error: "article not found: #{title}" } }
+end
+
+server.define_tool(
+  name: "get_categories",
+  description: "List the categories of one article (from the local index). Useful for building precise filters: inspect a few known-good articles' categories, then query find_articles with categories/category_match.",
+  input_schema: { properties: { title: { type: "string" } }, required: ["title"] }
+) do |title:, server_context:|
+  respond { corpus.get_categories(title) || { error: "article not found: #{title}" } }
+end
+
+server.define_tool(
+  name: "get_sections",
+  description: "Extract specific sections from one article by heading names. Use 'summary' for the lead text before the first heading. Optionally expand names through a saved alias_set.",
+  input_schema: {
+    properties: {
+      title: { type: "string" },
+      sections: { type: "array", items: { type: "string" } },
+      alias_set: { type: "string" }
+    },
+    required: ["title", "sections"]
+  }
+) do |title:, sections:, alias_set: nil, server_context:|
+  respond { corpus.get_sections(title, sections, alias_set: alias_set) || { error: "article not found: #{title}" } }
+end
+
+server.define_tool(
+  name: "list_headings",
+  description: "List the section heading tree (name + level) of one article. Useful before get_sections to see what headings exist.",
+  input_schema: { properties: { title: { type: "string" } }, required: ["title"] }
+) do |title:, server_context:|
+  respond { corpus.list_headings(title) || { error: "article not found: #{title}" } }
+end
+
+server.define_tool(
+  name: "find_articles",
+  description: "Exhaustively list articles matching filters, all AND-combined: category (single, with subcategory recursion via depth; NOTE category trees over-include, so dry-run with a small limit first), categories (exact multi-category intersection, e.g. year AND country), category_match (substring on category names, e.g. 'アメリカ合衆国の' matched against every category of the article), sections (article contains at least one of these headings; expanded via alias_set if given), title_match (title substring). Returns total count plus up to `limit` titles. Redirects are excluded.",
+  input_schema: {
+    properties: {
+      category: { type: "string" },
+      depth: { type: "integer", minimum: 0, maximum: 5 },
+      categories: { type: "array", items: { type: "string" } },
+      category_match: { type: "string" },
+      sections: { type: "array", items: { type: "string" } },
+      alias_set: { type: "string" },
+      title_match: { type: "string" },
+      limit: { type: "integer", minimum: 0 },
+      offset: { type: "integer", minimum: 0 }
+    },
+    required: []
+  }
+) do |category: nil, depth: 0, categories: nil, category_match: nil, sections: nil, alias_set: nil, title_match: nil, limit: 100, offset: 0, server_context:|
+  respond do
+    corpus.find_articles(category: category, depth: depth, categories: categories,
+                         category_match: category_match, sections: sections,
+                         alias_set: alias_set, title_match: title_match,
+                         limit: limit, offset: offset)
+  end
+end
+
+server.define_tool(
+  name: "query_sql",
+  description: "Read-only SQL (SELECT/WITH only) against the index databases — the escape hatch for questions the fixed tools cannot express (multi-way intersections, aggregations, joins). Call describe_schema first to see the tables. Main DB: pages(page_id, title, namespace, redirect_to, text_length), page_categories(page_id, category), page_sections(page_id, heading, level, ord), category_hierarchy(child, parent), alias_sets. When the fulltext tier exists, the FTS DB is attached as `fts` (fts.fts_sections MATCH ..., fts.fts_map). Filter articles with namespace = 0 AND redirect_to IS NULL. Rows are capped by `limit` (default 200, max 1000) with truncated flag; long cells are clipped. Queries are killed after 30 seconds (the error explains the likely cause); the tables are large (millions of rows), so constrain joins with indexed keys (page_id) and add WHERE filters.",
+  input_schema: {
+    properties: {
+      sql: { type: "string" },
+      limit: { type: "integer", minimum: 1, maximum: 1000 }
+    },
+    required: ["sql"]
+  }
+) do |sql:, limit: 200, server_context:|
+  respond { corpus.query_sql(sql, limit: limit) }
+end
+
+server.define_tool(
+  name: "describe_schema",
+  description: "CREATE statements of all tables available to query_sql (metadata DB, plus the attached fts DB when built).",
+  input_schema: { properties: {}, required: [] }
+) do |server_context:|
+  respond { corpus.describe_schema }
+end
+
+server.define_tool(
+  name: "category_tree",
+  description: "Subcategory tree under a category (names + depth). Use to understand scope before recursive find_articles.",
+  input_schema: {
+    properties: { category: { type: "string" }, depth: { type: "integer", minimum: 1, maximum: 5 } },
+    required: ["category"]
+  }
+) do |category:, depth: 2, server_context:|
+  respond { corpus.category_tree(category, depth: depth) }
+end
+
+server.define_tool(
+  name: "section_stats",
+  description: "Section heading frequencies across articles, optionally scoped to a category (+depth). This is the primary tool for discovering how sections are actually named in this language edition — use it before assuming heading names, and as the raw material for building alias sets.",
+  input_schema: {
+    properties: {
+      category: { type: "string" },
+      depth: { type: "integer", minimum: 0, maximum: 5 },
+      top_n: { type: "integer", minimum: 1, maximum: 500 }
+    },
+    required: []
+  }
+) do |category: nil, depth: 0, top_n: 50, server_context:|
+  respond { corpus.section_stats(category: category, depth: depth, top_n: top_n) }
+end
+
+server.define_tool(
+  name: "section_cooccurrence",
+  description: "Test whether headings are plausible aliases: for each heading, article count and average position; for each pair, how many articles contain BOTH. Synonymous headings almost never co-occur in one article, so a high cooccurrence_ratio is evidence AGAINST grouping them as aliases. Use to verify a hypothesis before save_alias_set.",
+  input_schema: {
+    properties: {
+      headings: { type: "array", items: { type: "string" }, minItems: 2 },
+      category: { type: "string" },
+      depth: { type: "integer", minimum: 0, maximum: 5 }
+    },
+    required: ["headings"]
+  }
+) do |headings:, category: nil, depth: 0, server_context:|
+  respond { corpus.section_cooccurrence(headings, category: category, depth: depth) }
+end
+
+server.define_tool(
+  name: "search_text",
+  description: "Exhaustive full-text search over cleaned article text (requires the fulltext index tier — check dump_info). mode: 'phrase' (literal string, default; 3+ chars for CJK dumps) or 'query' (raw FTS5 syntax: AND/OR/NEAR/quotes). Composable with category (recursive via depth) and sections filters (alias_set-aware). count: 'capped' (fast, default; total_is_capped signals more matches exist) or 'exact' (may take seconds for very common terms, but gives the true exhaustive count — required for absence/quantification claims). Snippets are re-rendered from the dump. Total of 0 with count 'exact' is a verified absence claim for this dump version.",
+  input_schema: {
+    properties: {
+      query: { type: "string" },
+      mode: { type: "string", enum: ["phrase", "query"] },
+      sections: { type: "array", items: { type: "string" } },
+      alias_set: { type: "string" },
+      category: { type: "string" },
+      depth: { type: "integer", minimum: 0, maximum: 5 },
+      limit: { type: "integer", minimum: 1, maximum: 100 },
+      offset: { type: "integer", minimum: 0 },
+      count: { type: "string", enum: ["capped", "exact"] }
+    },
+    required: ["query"]
+  }
+) do |query:, mode: "phrase", sections: nil, alias_set: nil, category: nil, depth: 0, limit: 20, offset: 0, count: "capped", server_context:|
+  respond do
+    corpus.search_text(query, mode: mode, sections: sections, alias_set: alias_set,
+                       category: category, depth: depth, limit: limit, offset: offset, count: count)
+  end
+end
+
+server.define_tool(
+  name: "save_alias_set",
+  description: "Persist a named set of section-heading alias groups for this dump (e.g. [[\"あらすじ\",\"ストーリー\",\"物語\"]]). A server-side guardrail re-checks co-occurrence: a pair that appears in the same article more than 20% of the time (when both headings occur in at least 100 articles) is treated as distinct sections, not synonyms, and blocks the save with saved:false and the offending pairs listed — fix the groups, or pass force:true only if you verified them another way (e.g., by reading section contents with get_sections). The response echoes the saved groups and the exact criteria used. Saved sets are referenced via alias_set in find_articles / get_sections / extract_corpus and recorded in extraction metadata for reproducibility.",
+  input_schema: {
+    properties: {
+      name: { type: "string" },
+      groups: { type: "array", items: { type: "array", items: { type: "string" } } },
+      force: { type: "boolean" }
+    },
+    required: ["name", "groups"]
+  }
+) do |name:, groups:, force: false, server_context:|
+  respond { corpus.save_alias_set(name, groups, force: force) }
+end
+
+server.define_tool(
+  name: "get_alias_set",
+  description: "Retrieve a saved alias set by name (groups + creation time).",
+  input_schema: { properties: { name: { type: "string" } }, required: ["name"] }
+) do |name:, server_context:|
+  respond { corpus.get_alias_set(name) || { error: "alias set not found: #{name}" } }
+end
+
+server.define_tool(
+  name: "list_alias_sets",
+  description: "List all alias sets saved for this dump.",
+  input_schema: { properties: {}, required: [] }
+) do |server_context:|
+  respond { { alias_sets: corpus.list_alias_sets } }
+end
+
+EXTRACT_SCHEMA = {
+  properties: {
+    output_path: { type: "string", description: "JSONL output path; relative paths resolve under the server's output dir" },
+    content: { type: "string", enum: ["sections", "summary", "full", "wikitext"] },
+    sections: { type: "array", items: { type: "string" } },
+    alias_set: { type: "string" },
+    category: { type: "string" },
+    depth: { type: "integer", minimum: 0, maximum: 5 },
+    categories: { type: "array", items: { type: "string" } },
+    category_match: { type: "string" },
+    title_match: { type: "string" },
+    limit: { type: "integer", minimum: 0 },
+    chunk_size: { type: "integer", minimum: 100, description: "Split text into ~N-char chunks (one RAG-ready record per chunk)" },
+    chunk_overlap: { type: "integer", minimum: 0 },
+    overwrite: { type: "boolean", description: "Replace an existing output file (default: refuse)" }
+  },
+  required: ["output_path", "content"]
+}.freeze
+
+def extract_params(output_path:, content:, sections: nil, alias_set: nil, category: nil,
+                   depth: 0, categories: nil, category_match: nil,
+                   title_match: nil, limit: 0, chunk_size: nil, chunk_overlap: 0,
+                   overwrite: false)
+  # Confine writes to the server's output directory (an agent mixing up paths
+  # must not be able to clobber arbitrary user files), and refuse to silently
+  # replace existing outputs
+  path = File.expand_path(output_path, OUTPUT_DIR)
+  base = File.expand_path(OUTPUT_DIR)
+  unless path == base || path.start_with?(base + File::SEPARATOR)
+    raise ArgumentError, "output_path must stay within the server output directory (#{base})"
+  end
+  if File.exist?(path) && !overwrite
+    raise ArgumentError, "output file already exists: #{path} (pass overwrite: true to replace it)"
+  end
+
+  { output_path: path, content: content,
+    sections: sections, alias_set: alias_set, category: category, depth: depth,
+    categories: categories, category_match: category_match,
+    title_match: title_match, limit: limit, chunk_size: chunk_size, chunk_overlap: chunk_overlap }
+end
+
+server.define_tool(
+  name: "extract_corpus",
+  description: "Extract matching articles to a JSONL file on disk (with a .meta.json sidecar recording dump version and query for reproducibility). Returns a summary and the first 3 records only — the full corpus stays on disk; process it with your own tools. content: 'sections' (named sections only), 'summary' (lead text), 'full' (whole cleaned article), or 'wikitext' (raw markup, for mining infoboxes/templates that cleaning replaces with markers). Filters are the same as find_articles (category/depth, categories, category_match, title_match). Optional chunk_size/chunk_overlap produce one RAG-ready record per chunk with section_path and chunk_index (not with wikitext). Synchronous, capped at 5000 articles (result flags truncation) — use start_extract_job for larger extractions. Dry-run the filters with find_articles first.",
+  input_schema: EXTRACT_SCHEMA
+) do |server_context:, **args|
+  respond { corpus.extract_corpus(**extract_params(**args)) }
+end
+
+server.define_tool(
+  name: "start_extract_job",
+  description: "Start extract_corpus as a background job (same parameters, no article cap). Returns job_id immediately; poll with job_status. Use for extractions larger than the 5000-article synchronous cap. Jobs do not survive server restart.",
+  input_schema: EXTRACT_SCHEMA
+) do |server_context:, **args|
+  respond { job_manager.start_extract(extract_params(**args)) }
+end
+
+server.define_tool(
+  name: "job_status",
+  description: "Status of a background extraction job: running/completed/cancelled/error, progress (0-1), and on completion the same summary extract_corpus returns (output path, counts, sample).",
+  input_schema: { properties: { job_id: { type: "string" } }, required: ["job_id"] }
+) do |job_id:, server_context:|
+  respond { job_manager.status(job_id) || { error: "unknown job: #{job_id}" } }
+end
+
+server.define_tool(
+  name: "cancel_job",
+  description: "Request cancellation of a running job (takes effect at the next batch boundary; the partial output file remains on disk).",
+  input_schema: { properties: { job_id: { type: "string" } }, required: ["job_id"] }
+) do |job_id:, server_context:|
+  respond { job_manager.cancel(job_id) || { error: "unknown job: #{job_id}" } }
+end
+
+server.define_tool(
+  name: "list_jobs",
+  description: "List all extraction jobs started in this server session.",
+  input_schema: { properties: {}, required: [] }
+) do |server_context:|
+  respond { { jobs: job_manager.list } }
+end
+
+server.define_prompt(
+  name: "discover_aliases",
+  description: "Recipe for building a verified section-alias set for a concept (e.g. plot sections of film articles) in this dump's language, using section_stats, section_cooccurrence, and content sampling.",
+  arguments: [
+    MCP::Prompt::Argument.new(name: "concept", description: "The section concept to cover (e.g. 'plot/story summary of a film')", required: true),
+    MCP::Prompt::Argument.new(name: "category", description: "Category scope (e.g. '映画作品')", required: false),
+    MCP::Prompt::Argument.new(name: "depth", description: "Category recursion depth (default 2)", required: false)
+  ]
+) do |args, server_context:|
+  concept = args[:concept] || args["concept"]
+  category = args[:category] || args["category"]
+  depth = args[:depth] || args["depth"] || 2
+  scope = category ? "category \"#{category}\" (depth #{depth})" : "the whole dump"
+  MCP::Prompt::Result.new(
+    description: "Alias discovery protocol for: #{concept}",
+    messages: [
+      MCP::Prompt::Message.new(
+        role: "user",
+        content: MCP::Content::Text.new(<<~TEXT)
+          Build a verified section-heading alias set for the concept "#{concept}" in #{scope}, using this protocol:
+
+          1. DISCOVER: Call section_stats#{category ? " with category=\"#{category}\", depth=#{depth}" : ""} (top_n 50-100). From the real heading frequencies, shortlist headings that could denote "#{concept}" based on their meaning. Note each candidate's article count — it tells you how much coverage each adds.
+          2. VERIFY: Call section_cooccurrence with the shortlisted headings (same scope). Interpret:
+             - ratio below ~0.05 between two FREQUENT headings: strong evidence they fill the same slot (valid aliases) — frequent headings would co-occur by chance if they were independent.
+             - ratio above ~0.15: they coexist in the same articles, so they are different section roles — do NOT group them.
+             - avg_position should also be similar for true aliases.
+          3. SAMPLE (only for borderline ratios of 0.05-0.2 or rare headings where statistics are weak): pick 2-3 articles per heading via find_articles(sections: [heading], limit: 3) and read the content with get_sections. Judge whether the text under each heading is the same KIND of content.
+          4. DECIDE: Groups must contain only verified same-role headings. For middling evidence, present the numbers to the user and ask.
+          5. SAVE: Call save_alias_set with a descriptive versioned name (e.g. "#{category ? category.downcase.gsub(/\s+/, '-') : 'dump'}-#{concept.split.first(2).join('-').downcase}-v1"). The server re-checks co-occurrence and will refuse groups with high-ratio pairs unless you pass force:true with justification.
+          6. REPORT: Tell the user which headings you included and excluded, with the co-occurrence numbers that justified each decision.
+        TEXT
+      )
+    ]
+  )
+end
+
+transport = MCP::Server::Transports::StdioTransport.new(server)
+transport.open