wp2txt 2.2.0 → 2.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5e54ea4f824be0f060036923759b1c6755b20ee1e721245db58a23665b9efaab
-  data.tar.gz: ca0a5b1ae0efae6eb4c485a6319569ea9995c662054e7e697a70429632613a25
+  metadata.gz: 076aa3a7681c0e0e6e13948e5c66f6f9e15a0307507a944a78f29c0e7856eea0
+  data.tar.gz: f8f2d3a4efc82d1922f9ca2764b23bb348aa924e752439805ab61683ff66a737
 SHA512:
-  metadata.gz: 34b61e5ee404431e521e77bc1223b9b7e1e0c9f2082ae4898e6e4b9fdee9de593c9ca45d661723f8d14f5ebffd62360dfb27d3a7b0d87d2a37691bb127c9ab6f
-  data.tar.gz: ca002ba2fc0c5a9bc8b2ae55dbc736f154e54452af681aad6f1b9ed30ca616535db94ee8cccb069defa56aacc4feb96b306d6fdb1a0102ee20611c98a213d5d0
+  metadata.gz: ae377878c20dde5a376d96efec19f84a74a20e66dd8f45ff5d41c438f519ced7cf5d4e6a6102ceee5c4b59333f0b1ca42167cf5e178abf803de868855b4efa6c
+  data.tar.gz: c23cb4598a7afeda092d75688be12e9e777874a4b9da289aeb1d97123b5cc64b3b56337722290fcc4352fde9031a856d4ba81393055ef8596fe8818d51e258ac

data/.gitignore

@@ -23,6 +23,9 @@ tags
 # Claude Code development files
 CLAUDE.md
 
+# Private design/research notes (not for publication)
+research-notes/
+
 # Error logs
 error_log.txt
 spec/examples.txt

data/CHANGELOG.md

@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.3.0] - 2026-07-24
+
+- **Documentation split**: README now focuses on text extraction; the research layer (indexes, exhaustive queries, full-text search, langlinks, cross-language SQL, MCP server) is documented in the new [Research Infrastructure Guide](docs/RESEARCH.md), including the complete MCP tool table
+- **Container images on GHCR**: images are now published to `ghcr.io/yohasebe/wp2txt` (Docker Hub `yohasebe/wp2txt` is maintained as a mirror)
+
+- **`extract_corpus` `titles:` argument**: Extract an explicit set of article titles (e.g. a set determined via `query_sql`) — titles are normalized (MediaWiki rules), deduplicated preserving input order, and one redirect hop is resolved; missing titles (including redirects to nowhere) are skipped and reported as `not_found` (count + 20-title sample, also in the `.meta.json` sidecar). Mutually exclusive with the filter arguments (set operations belong in SQL); capped at 10,000 titles. The sidecar records `titles_count` + `titles_sha256` (order-independent) for reproducibility, enumerating the full list when ≤100 titles. Also available via `start_extract_job`
+- **`query_sql` `output_path:` argument**: Write ALL rows of a large result to a JSONL file (with a `.meta.json` sidecar recording the SQL, dump version, attach configuration, row count, and tool version) and return only a summary + 3-row sample — the extract_corpus D4 pattern generalized to SQL. Writes happen in the forked child (so the 30s SIGKILL deadline covers them) to a `.partial` file that the parent atomically renames on success and removes on every failure path (child crash, timeout kill, query error). Hard cap `SQL_FILE_ROW_LIMIT` = 5M rows (`truncated` flag), cells clipped at 64KB (`cells_clipped` count), duplicate column names are made unique (`_2` suffix), `limit` is ignored in this mode, and existing files require `overwrite: true`. The MCP layer's output-dir confinement is now a shared helper (`Wp2txt::OutputPath.confine`) used by extract_corpus, start_extract_job, and query_sql alike
+- **Interlanguage links (`--import-langlinks`)**: Import the official `{lang}wiki-{date}-langlinks.sql.gz` dump into the Tier 1 metadata index as a `langlinks` table (`ll_from` = source page_id, `ll_lang` = target language, `ll_title` = normalized target title). Version pinning is enforced: the langlinks file's dump name must equal the built index's dump version — a mismatch is rejected with no override. Streams the MySQL dump without loading it whole (escape-safe tuple parser, 10k-row transaction batches, indexes created after the load). `--langlinks-langs` restricts imported target languages (e.g. `en,de,fr,zh,ko`); re-import requires `-U` (otherwise a no-op reporting the previous import time). Provenance (source file, size, import time, wp2txt version, language filter, row count) is stamped into the index and reported by `dump_info`; a post-import sanity check samples titles per language and warns when the join rate against a locally installed target edition falls below 90%
+- **Multi-dump ATTACH in `query_sql`**: The MCP `query_sql` tool gains an optional `attach` argument (language codes only, never paths): `attach: ["en"]` read-only ATTACHes that language's locally installed metadata DB as `en_meta` and its FTS DB (when built) as `en_fts`, both sharing the main DB's schema. Codes are validated, only installed indexes are accepted, path resolution is server-side (same-date dump preferred; otherwise the latest build with a `dump_mismatch` note in the response's `attached` metadata). User SQL still cannot contain ATTACH/DETACH — attachments are issued by server code only, via `mode=ro` URIs on a read-only connection. Combined with the langlinks table, this enables single-query cross-language comparisons (e.g. section-structure diffs of article pairs)
+
 ## [2.2.0] - 2026-07-22
 
 - **Index hardening (design review follow-up)**: `ord` now has identical semantics in `page_sections` and `fts_map` (lead = 0, first heading = 1; schema v2 — rebuild indexes with `--build-index -U`); section headings are normalized identically in both indexes (decorated headings like `== '''X''' ==` now match section filters); indexes record the wp2txt version (and the FTS index a rendering-config digest) so `dump_info` can flag indexes built by code whose text cleaning differs; rebuilds are atomic (built alongside, renamed on completion — a failed rebuild no longer destroys the working index)

data/Dockerfile

@@ -1,5 +1,8 @@
 FROM ruby:4.0.1-alpine3.23
 
+# Link the GHCR package to the source repository
+LABEL org.opencontainers.image.source="https://github.com/yohasebe/wp2txt"
+
 WORKDIR /wp2txt
 COPY . ./
 RUN rm -f Gemfile.lock

data/README.md

@@ -34,7 +34,7 @@ WP2TXT extracts plain text and category information from Wikipedia dump files. I
 - **Multilingual support** - Category and redirect detection for 350+ Wikipedia languages
 - **Streaming processing** - Process large dumps without intermediate files
 - **JSON output** - Machine-readable JSONL format for data pipelines
-- **Offline metadata index** - Exhaustive, version-pinned queries over categories and section headings, no API required
+- **Offline research indexes** - Exhaustive, version-pinned queries over categories, section headings, and full text; interlanguage links for cross-edition SQL
 - **MCP server** - Expose a local dump to LLM agents (Claude, ChatGPT, Gemini, local models) for grounded, reproducible corpus work
 
 ## Use Cases
@@ -79,19 +79,21 @@ The Docker image bundles everything — Ruby, decompression tools, and the MCP s
 **Interactive / batch CLI:**
 
 ```shell
-docker run -it -v /path/to/localdata:/data yohasebe/wp2txt
+docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 ```
 
 The `wp2txt` command is available inside the container. Use `/data` for input/output files.
 
+Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`); Docker Hub (`yohasebe/wp2txt`) is maintained as a mirror.
+
 **MCP server (no Ruby required on the host):**
 
 ```shell
 # Build the indexes once (cached in a named volume so they persist)
-docker run -it -v wp2txt:/root/.wp2txt yohasebe/wp2txt wp2txt --build-index --fulltext -L ja
+docker run -it -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt --build-index --fulltext -L ja
 
 # Register with an MCP client, e.g. Claude Code
-claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt yohasebe/wp2txt wp2txt-mcp -L ja
+claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt-mcp -L ja
 ```
 
 Note: use `-i` (not `-it`) when running the MCP server — a TTY would corrupt the JSON-RPC stream. The named volume (`wp2txt:`) holds downloaded dumps and indexes; without it they are lost when the container exits.
@@ -253,63 +255,28 @@ By default, citation templates are removed. Use `--extract-citations` to extract
 
 Supported: `{{cite book}}`, `{{cite web}}`, `{{cite news}}`, `{{cite journal}}`, `{{Citation}}`, etc.
 
-## Offline Metadata Index
-
-Build a local SQLite index of a dump — per-article categories, section headings, redirects, and the category hierarchy — extracted from the dump itself (no API access, no rate limits, version-pinned for reproducibility):
-
-    $ wp2txt --build-index --lang=ja     # downloads dump if needed; ja: ~15 min, ~1.7 GB index
-
-Then run exhaustive queries entirely offline:
-
-    # All film articles (recursing 3 levels of subcategories) that have a plot section
-    $ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
-
-    # Machine-readable output with total count
-    $ wp2txt --find-articles --in-category "Films" -D 2 -j json --limit 100 --lang=en
-
-Unlike web/API access, these queries are *exhaustive* (they scan every article, not search-ranked results) and *reproducible* (pinned to a specific dump version). Answering "which of the 1.5M articles have X" takes milliseconds once the index is built.
-
-### Full-Text Search
-
-Add `--fulltext` to also build an FTS5 index over the article text itself (tokenizer auto-selected: character-trigram for Japanese/Chinese/Korean, word-based for space-delimited languages):
-
-    $ wp2txt --build-index --fulltext --lang=ja
-    $ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
-
-Search hits report the article, section, and a snippet; totals are exhaustive counts, so `0 matches` is a verifiable absence claim for that dump version. The index is contentless (stores only the inverted index; snippets are re-rendered from the dump), keeping disk overhead moderate.
+## Research Infrastructure (Indexes, Exhaustive Queries, MCP)
 
-## MCP Server (LLM Integration)
+Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned research
+database**: SQLite indexes over categories, section headings, redirects, and (optionally)
+the full article text, plus interlanguage links for cross-edition comparison — all
+queryable offline, exhaustively, and exposed to LLM agents via an MCP server.
 
-`wp2txt-mcp` exposes a local dump to LLM agents (Claude, ChatGPT, Gemini, local models — any MCP-capable client) via the Model Context Protocol:
-
-    $ gem install mcp                    # optional dependency, needed only for the server
-    $ wp2txt --build-index --lang=ja     # prerequisite
-    $ wp2txt-mcp --lang=ja               # stdio MCP server
-
-Example client configuration (e.g., Claude Desktop / Claude Code):
-
-```json
-{
-  "mcpServers": {
-    "wp2txt": { "command": "wp2txt-mcp", "args": ["--lang", "ja"] }
-  }
-}
+```console
+$ wp2txt --build-index --fulltext --lang=ja      # build the indexes
+$ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
+$ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
+$ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
+$ wp2txt-mcp --lang=ja                           # stdio MCP server for LLM agents
 ```
 
-Tools provided:
-
-| Tool | Purpose |
-|------|---------|
-| `dump_info` | Dump version, index tiers, corpus statistics |
-| `get_article` / `get_sections` / `list_headings` | Single-article access (redirect-aware) |
-| `find_articles` | Exhaustive filtered listing (category recursion, section headings, title match) |
-| `category_tree` / `section_stats` | Scope exploration and heading frequency discovery |
-| `section_cooccurrence` | Verify section-alias hypotheses (synonyms rarely co-occur in one article) |
-| `save_alias_set` etc. | Persist verified per-dump alias groups (server-side co-occurrence guardrail) |
-| `extract_corpus` | Filtered extraction to JSONL + reproducibility sidecar; optional RAG chunking |
-| `start_extract_job` / `job_status` / `cancel_job` | Background jobs for large extractions |
+Unlike web/API access, these queries scan every article (a `0 matches` result is a
+verifiable absence claim for that dump version) and are reproducible: extractions record
+the dump version and query in a `.meta.json` sidecar.
 
-Design principles: large results go to disk (the model receives a summary plus a 3-record sample, never the full corpus); every output records the dump version and query in a `.meta.json` sidecar; section aliases are discovered from real dump statistics by the LLM and mechanically verified — no hand-maintained per-language dictionaries. The bundled `discover_aliases` prompt walks any agent through the verification protocol.
+**→ See the [Research Infrastructure Guide](docs/RESEARCH.md)** for index building,
+exhaustive queries, full-text search, interlanguage links, cross-language SQL, the full
+MCP tool list, and design principles.
 
 ## Command Line Options
 
@@ -384,6 +351,15 @@ Design principles: large results go to disk (the model receives a summary plus a
       -v, --version                    Print version
       -h, --help                       Show help
 
+### Research infrastructure options
+
+    --build-index                    Build the metadata index (add --fulltext for FTS)
+    --find-articles / --search       Exhaustive offline queries (see docs/RESEARCH.md)
+    --import-langlinks               Import interlanguage links (version-matched)
+    --fts-optimize                   Optimize an existing full-text index
+
+See the [Research Infrastructure Guide](docs/RESEARCH.md) for details.
+
 ## Configuration File
 
 Create persistent settings with:
@@ -425,6 +401,10 @@ Turbo mode (default) splits bz2 into XML chunks first, then processes in paralle
 
 See [CHANGELOG.md](CHANGELOG.md) for detailed release notes.
 
+**v2.3.0 (July 2026)**: Interlanguage links import, cross-language SQL (multi-dump ATTACH), explicit-title extraction, SQL results to file with reproducibility sidecars, GHCR image publishing.
+
+**v2.2.0 (July 2026)**: Offline metadata index, FTS5 full-text search, MCP server, query_sql escape hatch, extraction jobs.
+
 **v2.1.0 (February 2026)**: SQLite caching, Ractor parallelism (Ruby 4.0+), template expansion, content markers, Docker image update.
 
 **v2.0.0 (January 2026)**: Auto-download mode, category-based extraction, article extraction by title, JSON output, streaming processing, Ruby 4.0 support.

data/Rakefile

@@ -39,7 +39,10 @@ task :push do
       # docker buildx create --name mybuilder
       # docker buildx use mybuilder
       # docker buildx inspect --bootstrap
-      docker buildx build --platform linux/amd64,linux/arm64 -t yohasebe/wp2txt:#{Wp2txt::VERSION} -t yohasebe/wp2txt:latest . --push
+      docker buildx build --platform linux/amd64,linux/arm64 \
+        -t ghcr.io/yohasebe/wp2txt:#{Wp2txt::VERSION} -t ghcr.io/yohasebe/wp2txt:latest \
+        -t yohasebe/wp2txt:#{Wp2txt::VERSION} -t yohasebe/wp2txt:latest \
+        . --push
     BASH
   SCRIPT
 end

data/bin/wp2txt

@@ -711,6 +711,7 @@ class WpApp
     return run_find_articles(opts) if opts[:find_articles]
     return run_search(opts) if opts[:search]
     return run_fts_optimize(opts) if opts[:fts_optimize]
+    return run_import_langlinks(opts) if opts[:import_langlinks]
 
     # Determine input source
     if opts[:from_category] && opts[:lang]

data/bin/wp2txt-mcp

@@ -24,6 +24,7 @@ require "optimist"
 require "json"
 require_relative "../lib/wp2txt/corpus"
 require_relative "../lib/wp2txt/corpus_jobs"
+require_relative "../lib/wp2txt/output_path"
 require_relative "../lib/wp2txt/version"
 
 opts = Optimist.options do
@@ -159,16 +160,23 @@ end
 
 server.define_tool(
   name: "query_sql",
-  description: "Read-only SQL (SELECT/WITH only) against the index databases — the escape hatch for questions the fixed tools cannot express (multi-way intersections, aggregations, joins). Call describe_schema first to see the tables. Main DB: pages(page_id, title, namespace, redirect_to, text_length), page_categories(page_id, category), page_sections(page_id, heading, level, ord), category_hierarchy(child, parent), alias_sets. When the fulltext tier exists, the FTS DB is attached as `fts` (fts.fts_sections MATCH ..., fts.fts_map). Filter articles with namespace = 0 AND redirect_to IS NULL. Rows are capped by `limit` (default 200, max 1000) with truncated flag; long cells are clipped. Queries are killed after 30 seconds (the error explains the likely cause); the tables are large (millions of rows), so constrain joins with indexed keys (page_id) and add WHERE filters.",
+  description: "Read-only SQL (SELECT/WITH only) against the index databases — the escape hatch for questions the fixed tools cannot express (multi-way intersections, aggregations, joins). Call describe_schema first to see the tables. Main DB: pages(page_id, title, namespace, redirect_to, text_length), page_categories(page_id, category), page_sections(page_id, heading, level, ord), category_hierarchy(child, parent), alias_sets; a langlinks table (ll_from = source page_id, ll_lang, ll_title) is also present when interlanguage links were imported via wp2txt --import-langlinks. When the fulltext tier exists, the FTS DB is attached as `fts` (fts.fts_sections MATCH ..., fts.fts_map). Cross-language: pass attach: [\"en\",\"de\"] to read-only ATTACH other locally installed languages' indexes, referenced as {lang}_meta.pages / {lang}_meta.page_sections / ... and {lang}_fts when that language has a fulltext tier — attached DBs share the main DB's schema. Hyphenated language codes map to underscored aliases (e.g. zh-yue → zh_yue_meta). Only installed languages are accepted (the main language itself cannot be attached); the response's `attached` field lists what was attached (dump_name, fts, dump_mismatch when the dates differ). Join langlinks (ll_from = page_id) to map articles across languages. Filter articles with namespace = 0 AND redirect_to IS NULL. Rows are capped by `limit` (default 200, max 1000) with truncated flag; long cells are clipped. output_path: write ALL rows to a JSONL file (with a .meta.json recording the SQL, dump versions, and attach configuration) and return only a summary + 3-row sample — use this whenever the result is large; `limit` is ignored in this mode. Queries are killed after 30 seconds (the error explains the likely cause); the tables are large (millions of rows), so constrain joins with indexed keys (page_id) and add WHERE filters.",
   input_schema: {
     properties: {
       sql: { type: "string" },
-      limit: { type: "integer", minimum: 1, maximum: 1000 }
+      limit: { type: "integer", minimum: 1, maximum: 1000 },
+      attach: { type: "array", items: { type: "string" },
+                description: "Language codes of other locally installed dumps to ATTACH read-only as {lang}_meta / {lang}_fts (default: none)" },
+      output_path: { type: "string", description: "JSONL output path for ALL rows; relative paths resolve under the server's output dir" },
+      overwrite: { type: "boolean", description: "Replace an existing output file (default: refuse)" }
     },
     required: ["sql"]
   }
-) do |sql:, limit: 200, server_context:|
-  respond { corpus.query_sql(sql, limit: limit) }
+) do |sql:, limit: 200, attach: [], output_path: nil, overwrite: false, server_context:|
+  respond do
+    path = output_path && Wp2txt::OutputPath.confine(output_path, OUTPUT_DIR, overwrite: overwrite)
+    corpus.query_sql(sql, limit: limit, attach: attach, output_path: path, overwrite: overwrite)
+  end
 end
 
 server.define_tool(
@@ -279,6 +287,8 @@ EXTRACT_SCHEMA = {
   properties: {
     output_path: { type: "string", description: "JSONL output path; relative paths resolve under the server's output dir" },
     content: { type: "string", enum: ["sections", "summary", "full", "wikitext"] },
+    titles: { type: "array", items: { type: "string" }, maxItems: 10_000,
+              description: "Explicit article titles to extract (mutually exclusive with the filter arguments)" },
     sections: { type: "array", items: { type: "string" } },
     alias_set: { type: "string" },
     category: { type: "string" },
@@ -294,23 +304,12 @@ EXTRACT_SCHEMA = {
   required: ["output_path", "content"]
 }.freeze
 
-def extract_params(output_path:, content:, sections: nil, alias_set: nil, category: nil,
+def extract_params(output_path:, content:, titles: nil, sections: nil, alias_set: nil, category: nil,
                    depth: 0, categories: nil, category_match: nil,
                    title_match: nil, limit: 0, chunk_size: nil, chunk_overlap: 0,
                    overwrite: false)
-  # Confine writes to the server's output directory (an agent mixing up paths
-  # must not be able to clobber arbitrary user files), and refuse to silently
-  # replace existing outputs
-  path = File.expand_path(output_path, OUTPUT_DIR)
-  base = File.expand_path(OUTPUT_DIR)
-  unless path == base || path.start_with?(base + File::SEPARATOR)
-    raise ArgumentError, "output_path must stay within the server output directory (#{base})"
-  end
-  if File.exist?(path) && !overwrite
-    raise ArgumentError, "output file already exists: #{path} (pass overwrite: true to replace it)"
-  end
-
-  { output_path: path, content: content,
+  { output_path: Wp2txt::OutputPath.confine(output_path, OUTPUT_DIR, overwrite: overwrite),
+    content: content, titles: titles,
     sections: sections, alias_set: alias_set, category: category, depth: depth,
     categories: categories, category_match: category_match,
     title_match: title_match, limit: limit, chunk_size: chunk_size, chunk_overlap: chunk_overlap }
@@ -318,7 +317,7 @@ end
 
 server.define_tool(
   name: "extract_corpus",
-  description: "Extract matching articles to a JSONL file on disk (with a .meta.json sidecar recording dump version and query for reproducibility). Returns a summary and the first 3 records only — the full corpus stays on disk; process it with your own tools. content: 'sections' (named sections only), 'summary' (lead text), 'full' (whole cleaned article), or 'wikitext' (raw markup, for mining infoboxes/templates that cleaning replaces with markers). Filters are the same as find_articles (category/depth, categories, category_match, title_match). Optional chunk_size/chunk_overlap produce one RAG-ready record per chunk with section_path and chunk_index (not with wikitext). Synchronous, capped at 5000 articles (result flags truncation) — use start_extract_job for larger extractions. Dry-run the filters with find_articles first.",
+  description: "Extract matching articles to a JSONL file on disk (with a .meta.json sidecar recording dump version and query for reproducibility). Returns a summary and the first 3 records only — the full corpus stays on disk; process it with your own tools. content: 'sections' (named sections only), 'summary' (lead text), 'full' (whole cleaned article), or 'wikitext' (raw markup, for mining infoboxes/templates that cleaning replaces with markers). Filters are the same as find_articles (category/depth, categories, category_match, title_match). titles: extract an explicit list of articles (e.g. a set determined via query_sql); mutually exclusive with the filter arguments; redirects are resolved; missing titles are reported in not_found. Optional chunk_size/chunk_overlap produce one RAG-ready record per chunk with section_path and chunk_index (not with wikitext). Synchronous, capped at 5000 articles (result flags truncation) — use start_extract_job for larger extractions. Dry-run the filters with find_articles first.",
   input_schema: EXTRACT_SCHEMA
 ) do |server_context:, **args|
   respond { corpus.extract_corpus(**extract_params(**args)) }

data/docs/RESEARCH.md

@@ -0,0 +1,207 @@
+# wp2txt Research Infrastructure Guide
+
+This guide covers the research-oriented layer of wp2txt: local indexes over Wikipedia
+dumps, exhaustive offline queries, full-text search, cross-language SQL, and the MCP
+server that exposes all of this to LLM agents.
+
+For plain-text extraction (the classic wp2txt), see the [README](../README.md).
+
+## Concept
+
+Web search and the Wikipedia API operate on ranked, paginated, ever-changing data. They
+can show that something *exists*, but they cannot make **exhaustive** claims ("342 of the
+11,486 film articles with a plot section mention X — and none of the others do"), and
+their answers change from day to day.
+
+wp2txt takes the opposite approach: build local indexes over an official dump file, so that
+every query is
+
+- **exhaustive** — it scans every article, not search-ranked results;
+- **version-pinned** — results are tied to one dump (e.g. `jawiki-20260701`) and
+  reproducible later;
+- **agent-operable** — the MCP server exposes the whole layer to LLM agents, with
+  guardrails and provenance records designed for autonomous use.
+
+## 1. Building the indexes
+
+```console
+# Tier 1: metadata index (categories, section headings, redirects, category hierarchy)
+$ wp2txt --build-index --lang=ja
+
+# Tier 1 + Tier 2: add an FTS5 full-text index over the cleaned article text
+$ wp2txt --build-index --fulltext --lang=ja
+```
+
+The dump is downloaded automatically if needed and everything is cached under
+`~/.wp2txt/cache/`. Ballpark figures (Apple Silicon laptop): Japanese Wikipedia ~15 min /
+~1.7 GB for the metadata index, ~1.5 h / ~12 GB with full text; English roughly 4× the
+time, ~10 GB / ~12 GB respectively.
+
+The full-text tokenizer is selected per language: character trigrams for Japanese,
+Chinese, and Korean; word-based (`unicode61`, no stemming) for space-delimited languages.
+The choice is recorded in the index metadata.
+
+Indexes are rebuilt atomically (a failed rebuild never destroys the working index), and
+each index records the wp2txt version that built it — `dump_info` can flag indexes whose
+text-cleaning code differs from the running version.
+
+## 2. Exhaustive queries from the CLI
+
+```console
+# All film articles (recursing 3 subcategory levels) that have a plot section
+$ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
+
+# Machine-readable output with total count
+$ wp2txt --find-articles --in-category "Films" -D 2 -j json --limit 100 --lang=en
+
+# Full-text search composed with metadata filters
+$ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
+```
+
+Search totals are exhaustive counts, so `0 matches` is a verifiable **absence claim** for
+that dump version — something ranked web search cannot provide.
+
+## 3. Interlanguage links (langlinks)
+
+Import the official `langlinks` dump into the metadata index to map articles across
+language editions:
+
+```console
+$ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
+```
+
+- **Version pinning is enforced**: the langlinks file must carry the same dump date as
+  the built index; a mismatch is rejected with no override.
+- Import provenance (source file, time, tool version, row count, rows skipped for
+  invalid encoding) is stamped into the index and reported by `dump_info`.
+- A post-import sanity check joins a per-language sample against locally installed
+  editions and reports the match rate.
+
+This adds a `langlinks` table (`ll_from` = source page_id, `ll_lang`, `ll_title`) that can
+be joined in SQL. Tip: filter `ll_title != ''` — real dumps contain a few empty-title rows.
+
+## 4. The MCP server
+
+`wp2txt-mcp` exposes a local dump to any MCP-capable LLM client (Claude, ChatGPT, Gemini,
+local models):
+
+```console
+$ gem install mcp                    # optional dependency, needed only for the server
+$ wp2txt --build-index --lang=ja     # prerequisite
+$ wp2txt-mcp --lang=ja               # stdio MCP server
+```
+
+Example client configuration (e.g., Claude Desktop / Claude Code):
+
+```json
+{
+  "mcpServers": {
+    "wp2txt": { "command": "wp2txt-mcp", "args": ["--lang", "ja"] }
+  }
+}
+```
+
+Or run everything from the container image (no Ruby required on the host):
+
+```console
+$ docker run -it -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt --build-index --fulltext -L ja
+$ claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt-mcp -L ja
+```
+
+(Use `-i`, not `-it`, for the MCP server — a TTY would corrupt the JSON-RPC stream.)
+
+### Tools
+
+| Tool | Purpose |
+|------|---------|
+| `dump_info` | Dump identity, index tiers, corpus statistics, langlinks provenance |
+| `get_article` / `get_sections` / `list_headings` / `get_categories` | Single-article access (redirect-aware) |
+| `find_articles` | Exhaustive filtered listing (category recursion, category AND / pattern match, section headings, title match) |
+| `category_tree` / `section_stats` | Scope exploration and heading-frequency discovery |
+| `search_text` | Full-text search composed with metadata filters; exact or capped exhaustive counts |
+| `query_sql` | Read-only SQL (SELECT/WITH) over the index databases — the escape hatch for queries the fixed tools cannot express; supports cross-language `attach` and file output |
+| `describe_schema` | Table/column introspection for query_sql |
+| `section_cooccurrence` | Verify section-alias hypotheses (synonymous headings rarely co-occur in one article) |
+| `save_alias_set` / `get_alias_set` / `list_alias_sets` | Persist verified per-dump alias groups (server-side co-occurrence guardrail) |
+| `extract_corpus` | Filtered or explicit-title extraction to JSONL + reproducibility sidecar; optional RAG chunking |
+| `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` | Background jobs for large extractions |
+
+### Design principles
+
+- **Division of labor**: the tool does mechanical, exhaustive narrowing and counting;
+  semantic judgment is left to the LLM. The LLM never has to count.
+- **Context economy**: large results go to disk; the model receives a summary plus a
+  3-record sample, never the full corpus.
+- **Reproducibility**: every extraction and file-writing query records the dump version,
+  the query, and any alias sets in a `.meta.json` sidecar.
+- **Guardrails**: SQL is screened and executed read-only in a killable subprocess;
+  alias sets are re-verified server-side before saving; output paths are confined to the
+  server's output directory.
+
+## 5. Cross-language SQL
+
+With more than one language installed, `query_sql` can ATTACH other editions read-only:
+
+```
+query_sql(
+  attach: ["en"],
+  sql: "SELECT p.title AS ja_title, ll.ll_title AS en_title,
+               (SELECT COUNT(*) FROM page_sections s WHERE s.page_id = p.page_id) AS ja_secs,
+               (SELECT COUNT(*) FROM en_meta.page_sections s2
+                JOIN en_meta.pages p2 ON p2.page_id = s2.page_id
+                WHERE p2.title = ll.ll_title) AS en_secs
+        FROM pages p
+        JOIN langlinks ll ON ll.ll_from = p.page_id AND ll.ll_lang = 'en'
+        WHERE p.namespace = 0 AND p.redirect_to IS NULL AND ll.ll_title != ''"
+)
+```
+
+Attached databases appear as `{lang}_meta` (and `{lang}_fts` when that language has a
+full-text tier) and share the main database's schema. Language codes are validated and
+resolved server-side; user SQL can never contain ATTACH itself. The response records what
+was attached (dump names included), and flags date mismatches between editions.
+
+To the best of our knowledge no other system offers version-pinned, cross-edition SQL over
+both metadata **and** article text, fully offline.
+
+## 6. Large results, explicit sets, and reproducibility
+
+```
+# Write ALL rows of a query to disk; receive a summary + 3-row sample
+query_sql(sql: "...", attach: ["en"], output_path: "pairs_ja_en.jsonl")
+
+# Extract an explicit article set (e.g., determined via query_sql)
+extract_corpus(titles: ["東京物語", "羅生門", ...], content: "summary",
+               output_path: "screen.jsonl")
+```
+
+- `query_sql` + `output_path` streams every row to JSONL (atomically — a partial file is
+  never left behind) and writes a `.meta.json` sidecar recording the SQL, the dump
+  versions of every attached edition, and row counts.
+- `extract_corpus` + `titles:` accepts up to 10,000 explicit titles, resolves one
+  redirect hop, and reports unmatched titles in `not_found` — closing the loop
+  *SQL decides the set → the tool materializes it → the LLM reads it*.
+
+## 7. The alias discovery loop
+
+Section headings vary ("Plot" vs "Synopsis"; 「あらすじ」 vs 「ストーリー」). wp2txt ships
+no per-language dictionaries. Instead, agents discover aliases from the dump itself:
+
+1. `section_stats` — find the actual headings used in a scope
+2. LLM proposes synonym groups
+3. `section_cooccurrence` — verify mechanically (true synonyms almost never co-occur in
+   the same article; a high co-occurrence ratio is evidence *against* the hypothesis)
+4. `save_alias_set` — persist the verified groups, re-checked server-side, and recorded
+   in every extraction that uses them
+
+The bundled `discover_aliases` MCP prompt walks any agent through this protocol.
+
+## 8. Honest limitations
+
+- Queries operate on the **cleaned-text space**: content replaced by markers
+  (`[MATH]`, `[CODE]`, `[TABLE]`, …) is not searchable.
+- Trigram languages (ja/zh/ko) cannot match queries shorter than 3 characters;
+  word-based languages have no stemming (`run` ≠ `running`). Both are deliberate:
+  exact counts and absence claims require predictable matching.
+- Categories and links come from the dump itself, as written by editors — they inherit
+  Wikipedia's own inconsistencies, which is precisely what makes them worth studying.

data/lib/wp2txt/cli.rb

@@ -142,6 +142,14 @@ module Wp2txt
           opt :fts_optimize, "Optimize an existing full-text index (merge segments; idempotent)",
               default: false, short: :none
 
+          # Langlinks import (interlanguage links into the metadata index)
+          opt :import_langlinks, "Import interlanguage links (langlinks dump) into the metadata index (requires --lang)",
+              default: false, short: :none
+          opt :langlinks_file, "Use a local langlinks .sql(.gz) file instead of downloading (with --import-langlinks)",
+              type: String, short: :none
+          opt :langlinks_langs, "Comma-separated target languages to import with --import-langlinks (default: all)",
+              type: String, short: :none
+
           opt :file_size, "Approximate size (in MB) of each output file (0 for single file)",
               default: 10, short: "-f"
           opt :num_procs, "Number of parallel processes (auto-detected based on CPU/memory)",
@@ -352,6 +360,41 @@ module Wp2txt
           Optimist.die "--search cannot be combined with --build-index/--find-articles"
         end
 
+        # Langlinks import is a standalone mode requiring --lang
+        if opts[:import_langlinks]
+          Optimist.die "--import-langlinks requires --lang" if opts[:lang].nil?
+
+          conflicts = []
+          conflicts << "--build-index" if opts[:build_index]
+          conflicts << "--find-articles" if opts[:find_articles]
+          conflicts << "--search" if opts[:search]
+          conflicts << "--fts-optimize" if opts[:fts_optimize]
+          conflicts << "--articles" if opts[:articles]
+          conflicts << "--from-category" if opts[:from_category]
+          conflicts << "--section-stats" if opts[:section_stats]
+          unless conflicts.empty?
+            Optimist.die "--import-langlinks cannot be combined with #{conflicts.join(', ')}"
+          end
+        end
+
+        if opts[:langlinks_file] && !opts[:import_langlinks]
+          Optimist.die "--langlinks-file requires --import-langlinks"
+        end
+
+        if opts[:langlinks_file] && !File.exist?(opts[:langlinks_file])
+          Optimist.die :langlinks_file, "file does not exist"
+        end
+
+        if opts[:langlinks_langs] && !opts[:import_langlinks]
+          Optimist.die "--langlinks-langs requires --import-langlinks"
+        end
+
+        if opts[:langlinks_langs]
+          invalid = opts[:langlinks_langs].split(",").map(&:strip).reject(&:empty?) -
+                    opts[:langlinks_langs].split(",").map(&:strip).grep(/\A[a-z][a-z0-9-]{1,11}\z/)
+          Optimist.die :langlinks_langs, "invalid language code(s): #{invalid.join(', ')}" unless invalid.empty?
+        end
+
         if opts[:build_index] || opts[:find_articles] || opts[:search] || opts[:fts_optimize]
           conflicts = []
           conflicts << "--articles" if opts[:articles]