wp2txt 2.3.1 → 2.3.3

data/README.md

@@ -84,7 +84,7 @@ docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 
 The `wp2txt` command is available inside the container. Use `/data` for input/output files.
 
-Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`). The former Docker Hub repository (`yohasebe/wp2txt`) is no longer updated as of 2.3.1.
+Images are published to GitHub Container Registry (`ghcr.io/yohasebe/wp2txt`). The former Docker Hub repository has been removed — pull from GHCR.
 
 **MCP server (no Ruby required on the host):**
 
@@ -255,12 +255,12 @@ By default, citation templates are removed. Use `--extract-citations` to extract
 
 Supported: `{{cite book}}`, `{{cite web}}`, `{{cite news}}`, `{{cite journal}}`, `{{Citation}}`, etc.
 
-## Research Infrastructure (Indexes, Exhaustive Queries, MCP)
+## Offline Indexes, Queries, and the MCP Server
 
-Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned research
-database**: SQLite indexes over categories, section headings, redirects, and (optionally)
-the full article text, plus interlanguage links for cross-edition comparison — all
-queryable offline, exhaustively, and exposed to LLM agents via an MCP server.
+Beyond text extraction, wp2txt can turn a dump into a **local, version-pinned database**:
+SQLite indexes over categories, section headings, redirects, and (optionally) the full
+article text, plus interlanguage links for cross-edition comparison — all queryable
+offline, and usable from an LLM client via an MCP server.
 
 ```console
 $ wp2txt --build-index --fulltext --lang=ja      # build the indexes
@@ -270,13 +270,12 @@ $ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
 $ wp2txt-mcp --lang=ja                           # stdio MCP server for LLM agents
 ```
 
-Unlike web/API access, these queries scan every article (a `0 matches` result is a
-verifiable absence claim for that dump version) and are reproducible: extractions record
-the dump version and query in a `.meta.json` sidecar.
+These queries scan every article — `0 matches` means the term is absent from that dump
+version — and extractions record the dump version and query in a `.meta.json` sidecar,
+so results can be reproduced later.
 
-**→ See the [Research Infrastructure Guide](docs/RESEARCH.md)** for index building,
-exhaustive queries, full-text search, interlanguage links, cross-language SQL, the full
-MCP tool list, and design principles.
+**→ See [docs/INDEXES.md](docs/INDEXES.md)** for index building, offline queries,
+full-text search, interlanguage links, cross-language SQL, and the full MCP tool list.
 
 ## Command Line Options
 
@@ -354,11 +353,11 @@ MCP tool list, and design principles.
 ### Research infrastructure options
 
     --build-index                    Build the metadata index (add --fulltext for FTS)
-    --find-articles / --search       Exhaustive offline queries (see docs/RESEARCH.md)
+    --find-articles / --search       Offline queries over a whole edition (see docs/INDEXES.md)
     --import-langlinks               Import interlanguage links (version-matched)
     --fts-optimize                   Optimize an existing full-text index
 
-See the [Research Infrastructure Guide](docs/RESEARCH.md) for details.
+See [docs/INDEXES.md](docs/INDEXES.md) for details.
 
 ## Configuration File
 

data/README_ja.md

@@ -80,7 +80,7 @@ docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txt
 
 `wp2txt`コマンドはコンテナ内で使用可能です。入出力には`/data`ディレクトリを使用してください。
 
-イメージはGitHub Container Registry（`ghcr.io/yohasebe/wp2txt`）で公開されています。旧 Docker Hub リポジトリ（`yohasebe/wp2txt`）は 2.3.1 以降更新されません。
+イメージはGitHub Container Registry（`ghcr.io/yohasebe/wp2txt`）で公開されています。旧 Docker Hub リポジトリは削除済みです — GHCR から pull してください。
 
 ## 基本的な使い方
 
@@ -333,9 +333,9 @@ defaults:
 
 コマンドラインオプションは設定ファイルの設定を上書きします。
 
-## 研究基盤（索引・悉皆クエリ・MCP）
+## オフライン索引・クエリ・MCPサーバー
 
-テキスト抽出に加えて、wp2txtはダンプを**ローカルな版固定研究データベース**に変換できます。
+テキスト抽出に加えて、wp2txtはダンプを**ローカルな版固定データベース**に変換できます。
 カテゴリ・節見出し・リダイレクト・（オプションで）記事全文のSQLite索引、言語版横断比較のための
 言語間リンク（langlinks）— すべてオフラインで悉皆的にクエリでき、MCPサーバー経由で
 LLMエージェントにも公開できます。
@@ -348,12 +348,12 @@ $ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
 $ wp2txt-mcp --lang=ja                           # LLMエージェント向けMCPサーバー
 ```
 
-web/API アクセスと異なり、これらのクエリは全記事を走査します（`0件` は当該ダンプ版に対する
-検証可能な不在の言明になります）。抽出結果には dump 版とクエリを記録した `.meta.json`
-サイドカーが付き、再現可能です。
+これらのクエリは全記事を走査します（`0件` は当該ダンプ版にその語が存在しないことを
+意味します）。抽出結果には dump 版とクエリを記録した `.meta.json` サイドカーが付き、
+あとから再現できます。
 
-**→ 詳細は [Research Infrastructure Guide](docs/RESEARCH.md)（英語）を参照**：
-索引構築、悉皆クエリ、全文検索、言語間リンク、言語版横断SQL、MCPツール一覧、設計原則。
+**→ 詳細は [docs/INDEXES.md](docs/INDEXES.md)（英語）を参照**：
+索引構築、オフラインクエリ、全文検索、言語間リンク、言語版横断SQL、MCPツール一覧。
 
 ## パフォーマンス
 

data/Rakefile

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "bundler/gem_tasks"
+require "open3"
 require "rspec/core"
 require "rspec/core/rake_task"
 require_relative "./lib/wp2txt/version"
@@ -28,20 +29,25 @@ end
 
 Rake::Task["build"].enhance([:normalize_permissions])
 
+# Pre-release gate: verify the built gem's payload against spec.files and scan
+# it for names, content, and modes that must never ship (code-security protocol).
+Rake::Task["build"].enhance { sh "ruby", "scripts/verify_gem.rb" }
+
 # =============================================================================
 # Docker
 # =============================================================================
 
 # Paths that must never reach a published image. The image is built from the
 # working tree, so anything ignored locally (private notes, scratch files)
-# would otherwise ride along; 2.3.0's images shipped research-notes/ this way.
-IMAGE_FORBIDDEN_PATHS = %w[/wp2txt/research-notes /wp2txt/tmp /wp2txt/.git /wp2txt/CLAUDE.md /wp2txt/.claude].freeze
+# would otherwise ride along.
+IMAGE_FORBIDDEN_PATHS = %w[/wp2txt/research-notes /wp2txt/tmp /wp2txt/.git /wp2txt/CLAUDE.md /wp2txt/.claude /wp2txt/.private-doc-tokens].freeze
 
 desc "Verify a built image contains no private material (run before pushing)"
 task :verify_image, [:tag] do |_t, args|
   tag = args[:tag] || "wp2txt-verify:local"
   checks = IMAGE_FORBIDDEN_PATHS.map { |p| "test -e #{p} && echo LEAK:#{p}" }.join("; ")
-  out = `docker run --rm #{tag} sh -c '#{checks}; true' 2>&1`
+  out, status = Open3.capture2e("docker", "run", "--rm", tag, "sh", "-c", "#{checks}; true")
+  abort "Image verification failed for #{tag}: #{out}" unless status.success?
   leaks = out.lines.grep(/^LEAK:/).map(&:strip)
   abort "Image #{tag} contains private paths:\n  #{leaks.join("\n  ")}" unless leaks.empty?
 
@@ -54,18 +60,18 @@ task :check_image do
   Rake::Task[:verify_image].invoke
 end
 
-desc "Build and push Docker images to GHCR (verifies a local build first)"
-task push: :check_image do
-  # Docker Hub was retired after 2.3.0; GHCR (ghcr.io/yohasebe/wp2txt) is the
-  # only registry. Requires `docker buildx use multiarch` and a ghcr.io login.
-  sh <<-SCRIPT.strip_heredoc, { verbose: false }
-    /bin/bash -xeu <<'BASH'
-      # docker buildx create --name multiarch
-      # docker buildx use multiarch
-      # docker buildx inspect --bootstrap
-      docker buildx build --platform linux/amd64,linux/arm64 \
-        -t ghcr.io/yohasebe/wp2txt:#{Wp2txt::VERSION} -t ghcr.io/yohasebe/wp2txt:latest \
-        . --push
-    BASH
-  SCRIPT
+desc "Explain how images are published (they are built and pushed by CI)"
+task :push do
+  abort <<~MESSAGE
+    Images are published by GitHub Actions, not from here.
+
+    A local build sends this working tree as the build context, so untracked
+    files ride along; a runner starts from a clean checkout, where they do not
+    exist. Push a v* tag and .github/workflows/publish-image.yml takes over:
+
+        rake release            # tags and pushes (also publishes the gem)
+
+    To rehearse without publishing, run the workflow from the Actions tab with
+    "push" left off. To check a local build, run `rake check_image`.
+  MESSAGE
 end

data/bin/wp2txt

@@ -217,7 +217,8 @@ class WpApp
           next unless title_match
 
           title = title_match[1]
-          next if title.nil? || title.empty? || title.include?(":")
+          next if title.nil? || title.empty?
+          next unless Wp2txt.namespace_id(page_xml[%r{<ns>(-?\d+)</ns>}, 1]).zero?
 
           # Extract text content
           text_match = TEXT_REGEX.match(page_xml)
@@ -609,7 +610,8 @@ class WpApp
         next unless title_match
 
         title = title_match[1]
-        next if title.nil? || title.empty? || title.include?(":")
+        next if title.nil? || title.empty?
+        next unless Wp2txt.namespace_id(page_xml[%r{<ns>(-?\d+)</ns>}, 1]).zero?
 
         text_match = TEXT_REGEX.match(page_xml)
         next unless text_match

data/bin/wp2txt-mcp

@@ -73,7 +73,7 @@ def respond(&block)
   result = block.call
   MCP::Tool::Response.new([{ type: "text", text: JSON.generate(result) }])
 rescue ArgumentError => e
-  MCP::Tool::Response.new([{ type: "text", text: JSON.generate({ error: e.message }) }], error: true)
+  MCP::Tool::Response.new([{ type: "text", text: JSON.generate({ error: e.message, code: (e.code if e.respond_to?(:code)) }.compact) }], error: true)
 rescue StandardError => e
   MCP::Tool::Response.new([{ type: "text", text: JSON.generate({ error: "#{e.class}: #{e.message}" }) }], error: true)
 end
@@ -312,7 +312,8 @@ def extract_params(output_path:, content:, titles: nil, sections: nil, alias_set
     content: content, titles: titles,
     sections: sections, alias_set: alias_set, category: category, depth: depth,
     categories: categories, category_match: category_match,
-    title_match: title_match, limit: limit, chunk_size: chunk_size, chunk_overlap: chunk_overlap }
+    title_match: title_match, limit: limit, chunk_size: chunk_size, chunk_overlap: chunk_overlap,
+    overwrite: overwrite }
 end
 
 server.define_tool(
@@ -341,7 +342,7 @@ end
 
 server.define_tool(
   name: "cancel_job",
-  description: "Request cancellation of a running job (takes effect at the next batch boundary; the partial output file remains on disk).",
+  description: "Request cancellation of a running job (takes effect at the next batch boundary; staged output is removed).",
   input_schema: { properties: { job_id: { type: "string" } }, required: ["job_id"] }
 ) do |job_id:, server_context:|
   respond { job_manager.cancel(job_id) || { error: "unknown job: #{job_id}" } }

data/docs/{RESEARCH.md → INDEXES.md}

@@ -1,40 +1,31 @@
-# wp2txt Research Infrastructure Guide
+# Offline Indexes, Queries, and the MCP Server
 
-This guide covers the research-oriented layer of wp2txt: local indexes over Wikipedia
-dumps, exhaustive offline queries, full-text search, cross-language SQL, and the MCP
-server that exposes all of this to LLM agents.
+This guide covers wp2txt's index-based features: local indexes over Wikipedia dumps,
+offline queries across a whole edition, full-text search, interlanguage links,
+cross-language SQL, and the MCP server for connecting an LLM client.
 
 For plain-text extraction (the classic wp2txt), see the [README](../README.md).
 
-## Concept
-
-Web search and the Wikipedia API operate on ranked, paginated, ever-changing data. They
-can show that something *exists*, but they cannot make **exhaustive** claims ("342 of the
-11,486 film articles with a plot section mention X — and none of the others do"), and
-their answers change from day to day.
-
-wp2txt takes the opposite approach: build local indexes over an official dump file, so that
-every query is
-
-- **exhaustive** — it scans every article, not search-ranked results;
-- **version-pinned** — results are tied to one dump (e.g. `jawiki-20260701`) and
-  reproducible later;
-- **agent-operable** — the MCP server exposes the whole layer to LLM agents, with
-  guardrails and provenance records designed for autonomous use.
+Everything below runs against a downloaded dump file: queries cover every article of the
+edition rather than a page of search results, results are tied to one dump (e.g.
+`jawiki-20260701`) and can be reproduced later, and nothing goes over the network once
+the dump is downloaded. Typical uses: "which of the 1.5M articles have a plot section",
+"how many film articles mention X, and which ones don't", "how do the section structures
+of the same article differ between the English and Japanese editions".
 
 ## 1. Building the indexes
 
 ```console
-# Tier 1: metadata index (categories, section headings, redirects, category hierarchy)
+# Metadata index (categories, section headings, redirects, category hierarchy)
 $ wp2txt --build-index --lang=ja
 
-# Tier 1 + Tier 2: add an FTS5 full-text index over the cleaned article text
+# Metadata index + FTS5 full-text index over the cleaned article text
 $ wp2txt --build-index --fulltext --lang=ja
 ```
 
 The dump is downloaded automatically if needed and everything is cached under
 `~/.wp2txt/cache/`. Ballpark figures (Apple Silicon laptop): Japanese Wikipedia ~15 min /
-~1.7 GB for the metadata index, ~1.5 h / ~12 GB with full text; English roughly 4× the
+~1.7 GB for the metadata index, ~1.5 h / ~12 GB with full text; English roughly 2.5–3× the
 time, ~10 GB / ~12 GB respectively.
 
 The full-text tokenizer is selected per language: character trigrams for Japanese,
@@ -58,8 +49,8 @@ $ wp2txt --find-articles --in-category "Films" -D 2 -j json --limit 100 --lang=e
 $ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
 ```
 
-Search totals are exhaustive counts, so `0 matches` is a verifiable **absence claim** for
-that dump version — something ranked web search cannot provide.
+Search totals count every match in the edition, so `0 matches` means the term is absent
+from that dump version — a result you can state and re-verify later.
 
 ## 3. Interlanguage links (langlinks)
 
@@ -114,7 +105,7 @@ $ claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yo
 
 | Tool | Purpose |
 |------|---------|
-| `dump_info` | Dump identity, index tiers, corpus statistics, langlinks provenance |
+| `dump_info` | Dump identity, installed indexes, corpus statistics, langlinks provenance |
 | `get_article` / `get_sections` / `list_headings` / `get_categories` | Single-article access (redirect-aware) |
 | `find_articles` | Exhaustive filtered listing (category recursion, category AND / pattern match, section headings, title match) |
 | `category_tree` / `section_stats` | Scope exploration and heading-frequency discovery |
@@ -126,17 +117,23 @@ $ claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yo
 | `extract_corpus` | Filtered or explicit-title extraction to JSONL + reproducibility sidecar; optional RAG chunking |
 | `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` | Background jobs for large extractions |
 
-### Design principles
-
-- **Division of labor**: the tool does mechanical, exhaustive narrowing and counting;
-  semantic judgment is left to the LLM. The LLM never has to count.
-- **Context economy**: large results go to disk; the model receives a summary plus a
-  3-record sample, never the full corpus.
-- **Reproducibility**: every extraction and file-writing query records the dump version,
-  the query, and any alias sets in a `.meta.json` sidecar.
-- **Guardrails**: SQL is screened and executed read-only in a killable subprocess;
-  alias sets are re-verified server-side before saving; output paths are confined to the
-  server's output directory.
+### What happens when your assistant uses these tools
+
+- **Filtering and counting run over the whole dump**, and your assistant reads the result
+  rather than tallying articles itself.
+- **Large results are written to a file**; the reply carries a summary and a short sample.
+  Your corpus lands on disk intact instead of being paraphrased through the chat.
+- **Extractions are traceable.** Extractions and file-writing queries leave a `.meta.json`
+  next to the output recording the dump version, the query, and any alias sets used, so
+  you can reproduce or cite the result later.
+- **The tools cannot change your data.** Queries run read-only and are stopped after 30
+  seconds, saved alias sets are re-checked before being stored, and files can only be
+  written under the server's output directory — worth knowing if you plan to let an
+  assistant work unattended.
+  Output confinement assumes a dedicated, trusted output directory: symlink paths are
+  rejected, both output and sidecar are exclusively reserved unless overwrite is requested,
+  and unique temporary files are renamed on success; concurrent replacement of parent
+  directories is outside this guarantee.
 
 ## 5. Cross-language SQL
 
@@ -157,12 +154,12 @@ query_sql(
 ```
 
 Attached databases appear as `{lang}_meta` (and `{lang}_fts` when that language has a
-full-text tier) and share the main database's schema. Language codes are validated and
-resolved server-side; user SQL can never contain ATTACH itself. The response records what
-was attached (dump names included), and flags date mismatches between editions.
+full-text index) and share the main database's schema. You pass language codes, not paths,
+and the SQL you write cannot attach anything itself. The response lists what was attached,
+including each edition's dump name, and flags it when the dates differ.
 
-To the best of our knowledge no other system offers version-pinned, cross-edition SQL over
-both metadata **and** article text, fully offline.
+This runs entirely offline against pinned dump versions, so a cross-edition comparison can
+be re-run later and produce the same numbers.
 
 ## 6. Large results, explicit sets, and reproducibility
 
@@ -182,26 +179,34 @@ extract_corpus(titles: ["東京物語", "羅生門", ...], content: "summary",
   redirect hop, and reports unmatched titles in `not_found` — closing the loop
   *SQL decides the set → the tool materializes it → the LLM reads it*.
 
-## 7. The alias discovery loop
-
-Section headings vary ("Plot" vs "Synopsis"; 「あらすじ」 vs 「ストーリー」). wp2txt ships
-no per-language dictionaries. Instead, agents discover aliases from the dump itself:
-
-1. `section_stats` — find the actual headings used in a scope
-2. LLM proposes synonym groups
-3. `section_cooccurrence` — verify mechanically (true synonyms almost never co-occur in
-   the same article; a high co-occurrence ratio is evidence *against* the hypothesis)
-4. `save_alias_set` — persist the verified groups, re-checked server-side, and recorded
-   in every extraction that uses them
-
-The bundled `discover_aliases` MCP prompt walks any agent through this protocol.
-
-## 8. Honest limitations
-
-- Queries operate on the **cleaned-text space**: content replaced by markers
-  (`[MATH]`, `[CODE]`, `[TABLE]`, …) is not searchable.
-- Trigram languages (ja/zh/ko) cannot match queries shorter than 3 characters;
-  word-based languages have no stemming (`run` ≠ `running`). Both are deliberate:
-  exact counts and absence claims require predictable matching.
-- Categories and links come from the dump itself, as written by editors — they inherit
-  Wikipedia's own inconsistencies, which is precisely what makes them worth studying.
+## 7. Section alias sets
+
+Section headings vary by article and by language ("Plot" vs "Synopsis"; 「あらすじ」 vs
+「ストーリー」), and wp2txt ships no per-language dictionaries. Three tools manage named
+groups of equivalent headings instead:
+
+- `section_stats` lists the headings actually used in a scope, with counts.
+- `section_cooccurrence` reports how often two headings appear in the same article.
+  Headings that mean the same thing rarely co-occur, so a high ratio is evidence that
+  they are *different* sections (「概要」 and 「あらすじ」 co-occur often — not synonyms).
+- `save_alias_set` stores a named group. The group is re-checked against co-occurrence
+  before saving; a failing group is not stored and the call returns `saved: false`
+  rather than an error. Pass `force` to override, and use `list_alias_sets` to see
+  what is stored.
+
+Queries then accept `alias_set: "name"` in place of a heading list, and extractions
+record the group's exact contents in their `.meta.json`. The bundled `discover_aliases`
+prompt walks an LLM client through building and saving a set.
+
+## 8. Limitations to keep in mind
+
+- Searches run over the **cleaned text**: content that extraction replaces with a marker
+  (`[MATH]`, `[CODE]`, `[TABLE]`, …) cannot be matched. A count is a count of the cleaned
+  text, not of the raw wikitext.
+- Japanese, Chinese, and Korean indexes cannot match queries shorter than 3 characters.
+  Word-based languages match exact forms only — `run` does not find `running`. Plan your
+  search terms accordingly, especially when you intend to report a zero result.
+- Categories and interlanguage links are read from the dump as editors wrote them.
+  Categories added by a template rather than written in the article text are not visible
+  to wp2txt, which can make a category look much smaller than it is on the website —
+  check against the article text if a count looks wrong.

data/lib/wp2txt/constants.rb

@@ -1,6 +1,11 @@
 # frozen_string_literal: true
 
 module Wp2txt
+  # Missing namespace elements use the metadata scanner's historical ns=0 default.
+  def self.namespace_id(value)
+    (value || "0").to_i
+  end
+
   # =========================================================================
   # Custom Exception Classes
   # =========================================================================