wp2txt 2.3.4 → 2.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3972afe52897df60befbb96bed211e4fd05e33cce380ed46ced223637aed6954
-  data.tar.gz: 6674325567d9b21cfb8819110827a16327aaee2e7e0e509115953e3b97229b74
+  metadata.gz: 54e6c29a38bad95092494bf88db73029769d942940953480e1db4e6433982cad
+  data.tar.gz: 811faae9a5b59bd3e58ca4210314d52daadfe356de0da4629732e9fe3af3fe60
 SHA512:
-  metadata.gz: b7e36ac5cc6ab08699f3765768d9629b8cf193d1c134fcaeb9925228739b960bbc71c9ac5056c0bda5add9d1110cf993220b0a5a508e29caa9529eac84830278
-  data.tar.gz: abba849021598f7003a6fce9798517b2f497486853ce65522f687ea767eea1a1b91b87046c715f90d7908bff228cc441beec6cf4907c4885adeddd9967335165
+  metadata.gz: 387f9374165f327a5f398306232ed32e5fa096e01b84519f8e1ab41f0e8b773d9f2cfccbc2e9504080550b9dcdb31ac269102845cae4a7a0e547ad6db864b5f1
+  data.tar.gz: 1a10d09d4e12bd94147fd3fa09126bb6450d141041c07013c71c4d51abc8537f80e95e822186f86591da94b4f7a6a0066b469e9379880f315938a82c7e03cbfb

data/.dockerignore

@@ -1,13 +1,11 @@
 .git
 .github
-.claude
 image
 pkg
 spec
 coverage
 tmp
 .bundle
-research-notes
 benchmark_results
 data/output_samples
 scripts
@@ -15,10 +13,8 @@ scripts
 .solargraph.yml
 .rubocop.yml
 Gemfile.lock
-.private-doc-tokens
 **/.DS_Store
 .ruby-version
-CLAUDE.md
 DEVELOPMENT.md
 DEVELOPMENT_ja.md
 *.gem

data/.gitignore

@@ -20,11 +20,7 @@ tmp
 *.~
 tags
 
-# Claude Code development files
-CLAUDE.md
 
-# Private design/research notes (not for publication)
-research-notes/
 
 # Error logs
 error_log.txt
@@ -36,4 +32,6 @@ benchmark_results/
 
 # Developer-specific Ruby version
 .ruby-version
-.private-doc-tokens
+
+# Machine-local files (never committed)
+.local/

data/CHANGELOG.md

@@ -8,6 +8,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 Entries were rewritten in August 2026 to describe what changed for people using
 wp2txt, rather than how it was implemented. The changes themselves are unaltered.
 
+## [2.4.0] - 2026-10-02
+
+- **Readings and link text are no longer dropped from Japanese articles**: template expansion, on by default, removed `{{読み仮名}}`, ruby templates, and `{{仮リンク}}` before they could be rendered, so a lead like "'''{{読み仮名|言語|げんご}}'''は…" came out as "は…" and names linked through 仮リンク vanished from running text (one in six Japanese articles uses 仮リンク). Re-extract if you rely on these articles
+- **Interlanguage links import from current dumps again**: dumps now write one row per line, which the importer read as nothing while still reporting success. If you imported langlinks from a dump dated September 2026 or later, re-import with `-U`. An import that reads no rows now fails instead of succeeding quietly
+- **`--lead-terms` (JSON output)**: lists the terms an article introduces in its lead — the bold terms of the first lead paragraph that has one (up to five), the parenthesized text written right after each (split at top-level commas and semicolons, and unsplit), and reading templates as text/reading pairs. Each term carries character offsets into the article's wikitext as stored in the dump, so its source can be cut out and checked. Nothing is interpreted: whether a note is a reading, a native spelling, or a date is left to you. Not available with `--ractor`
+- **`--count-links`**: adds, for every article in an existing metadata index, how many articles link to it — each linking article counted once, through one redirect hop, using the wiki's own capitalization rule, and ignoring links that the wiki does not render (comments, `nowiki`, `pre`, `math`, and similar). About 12 minutes for Japanese Wikipedia
+- **`--import-page-props`**: adds each article's Wikidata ID, MediaWiki's disambiguation mark, and its sort key from the `page_props` dump of the same date as the index. Once imported, JSON output, `get_article`, and `extract_corpus` carry `qid`, `sort_key`, and `disambiguation` (null or false when a page has none). The disambiguation mark finds many more disambiguation pages than their titles do. On Japanese Wikipedia the sort key is a reading normalized for sorting (voicing marks dropped), useful for checking reading candidates
+- **`dump_info` reports where the new data came from**, including the `page_props` file's SHA-256 and the rule used to count links
+
 ## [2.3.4] - 2026-09-30
 
 - **Records are no longer duplicated in `--no-turbo` output or in large `extract_corpus` runs**: the last record written before each batch could appear once more for every worker process — up to eight identical lines in a row. The default mode for `.bz2` dumps was not affected. If you extracted more than 200 articles with `extract_corpus`, or used `--no-turbo`, check earlier output for repeated lines; the repeats are byte-for-byte identical, so removing exact duplicate lines restores it

data/DEVELOPMENT.md

@@ -406,7 +406,7 @@ local build sends the working tree as its context, so untracked files ride
 along, while a runner starts from a clean checkout.
 
 ```bash
-rake check_image   # Build locally and run the same gate the workflow runs
+rake check_image   # Build from a clean copy of the last commit and run the workflow's gate
 ```
 
 The gate (`scripts/verify_image.rb`) lists what the image holds under `/wp2txt`

data/DEVELOPMENT_ja.md

@@ -402,7 +402,7 @@ docs/INDEXES.md の MCP ツール表は spec/docs_sync_spec.rb が実際のサ
 Dockerイメージのビルドとプッシュ：
 
 ```bash
-rake check_image   # ローカルでビルドし、CI と同じゲートを走らせる
+rake check_image   # 最新のコミットのクリーンな複製からビルドし、CI と同じゲートを走らせる
 ```
 
 ## リリースプロセス

data/README.md

@@ -177,6 +177,45 @@ Each line contains one JSON object:
 
 `page_id` and `revision_id` are the dump's own `<id>` values for the page and its
 revision, so a record can be traced back to the exact version of the article it came from.
+If page properties have been imported into the dump's metadata index
+(`--import-page-props`, see [docs/INDEXES.md](docs/INDEXES.md)), three fields always
+follow the source IDs: `qid`, `sort_key`, and `disambiguation`. Missing QIDs and sort
+keys are `null`; an absent disambiguation flag is `false`. Before import, all three
+fields are omitted.
+
+`disambiguation` is MediaWiki's own flag, which identifies pages that a title-based
+check would miss. `sort_key` is the stored sorting key, whose meaning varies by wiki:
+Japanese Wikipedia commonly uses normalized readings without voicing marks and with
+small kana enlarged; English Wikipedia may use "surname, given name". It is not a
+reading itself, but can be used to compare reading candidates. WP2TXT returns the
+stored value without interpreting it.
+
+With `--lead-terms`, each record also lists the terms the article introduces in its lead:
+
+```json
+"lead_terms": [
+  {"index": 0, "text": "東京", "notes": ["とうきょう", "Tokyo"], "notes_text": "とうきょう、Tokyo",
+   "source": "bold", "span": {"bold": [0, 8], "paren": [8, 21]}}
+]
+```
+
+- `source: "bold"` — a bold term in the first lead paragraph that has one (up to five),
+  with the parenthesized text written right after it, split at top-level commas and
+  semicolons into `notes` (`notes_text` keeps it unsplit). Bold text inside templates,
+  image captions, references, literal regions such as `nowiki` and `pre`, and
+  `gallery`/`timeline` content is not counted. `code` is an ordinary formatting tag:
+  bold text inside it can be a lead term.
+- `source: <template name>` — a reading template such as `{{読み仮名}}`, reported as
+  `text` and `reading`.
+- `span` gives character offsets `[start, end)` into the article's wikitext as stored in
+  the dump (XML entities decoded, nothing else changed), so the source of each term can be
+  cut out and checked later.
+- Nothing is interpreted: whether a note is a reading, a native spelling, or a date is left
+  to you.
+
+`--lead-terms` cannot be combined with `--ractor`. The experimental Ractor JSON path
+also omits page IDs and revision IDs; the CLI warns when it is selected. Imported
+page properties are attached in the parent process on this path too.
 
 For redirect articles:
 

data/README_ja.md

@@ -161,6 +161,36 @@ CATEGORIES: カテゴリ1, カテゴリ2, カテゴリ3
 
 `page_id` と `revision_id` はダンプに記録されたページと版の `<id>` です。各レコードが
 どの記事のどの版から取り出されたかを後から辿れます。
+そのダンプのメタデータ索引にページ属性を取り込んである場合（`--import-page-props`、
+[docs/INDEXES.md](docs/INDEXES.md) 参照）、その後に `qid`・`sort_key`・`disambiguation` の3項目が常に続きます。
+QID・並べ替えの鍵がなければ `null`、曖昧さ回避の印がなければ `false`。未取り込みなら3項目とも出しません。
+
+`disambiguation` は MediaWiki 自身の印で、題名の形だけでは拾えない曖昧さ回避ページも識別できます。
+`sort_key` の意味は言語版ごとに異なります。日本語版では通常、濁点・半濁点を落とし、小書きの仮名を
+大書きにした並べ替え用の読み（例: 言語→「けんこ」）、英語版では「姓, 名」の形などです。
+読みそのものではありませんが、読みの候補の照合に使えます。WP2TXT は値を解釈せず、そのまま渡します。
+
+`--lead-terms` を付けると、記事の導入部で示される語の一覧も出力します：
+
+```json
+"lead_terms": [
+  {"index": 0, "text": "東京", "notes": ["とうきょう", "Tokyo"], "notes_text": "とうきょう、Tokyo",
+   "source": "bold", "span": {"bold": [0, 8], "paren": [8, 21]}}
+]
+```
+
+- `source: "bold"` — 導入部で太字を含む最初の段落の太字（最大5個）と、その直後の括弧の中身。
+  括弧の中身は最上位の読点・カンマ・セミコロンで区切って `notes` に入れる（`notes_text` は区切る前）。
+  テンプレート・画像の説明文・脚注、`nowiki`・`pre` などの字義どおりの領域、および
+  `gallery`・`timeline` の太字は対象外。`code` は通常の書式タグなので、中の太字も対象
+- `source: <テンプレート名>` — `{{読み仮名}}` などの読みのテンプレート。`text` と `reading` の組
+- `span` はダンプに記録された記事の wikitext（XML の実体参照を戻しただけのもの）での文字位置
+  `[開始, 終了)`。各語の根拠をあとから切り出して確かめられる
+- 解釈はしない。括弧の中のどれが読みで、どれが原語や日付かの判断は利用側に任せる
+
+`--lead-terms` と `--ractor` は併用できません。実験的な Ractor 経路の JSON には
+ページ ID・リビジョン ID が含まれないため、選択時に警告します。取り込み済みの属性3項目は、
+この経路でも親プロセス側で付与します。
 
 リダイレクト記事の場合：
 

data/Rakefile

@@ -30,34 +30,23 @@ end
 Rake::Task["build"].enhance([:normalize_permissions])
 
 # Pre-release gate: verify the built gem's payload against spec.files and scan
-# it for names, content, and modes that must never ship (code-security protocol).
+# it for names, content, and modes that must never ship.
 Rake::Task["build"].enhance { sh "ruby", "scripts/verify_gem.rb" }
 
 # =============================================================================
 # Docker
 # =============================================================================
 
-# Paths that must never reach a published image. The image is built from the
-# working tree, so anything ignored locally (private notes, scratch files)
-# would otherwise ride along.
-IMAGE_FORBIDDEN_PATHS = %w[/wp2txt/research-notes /wp2txt/tmp /wp2txt/.git /wp2txt/CLAUDE.md /wp2txt/.claude /wp2txt/.private-doc-tokens].freeze
-
-desc "Verify a built image contains no private material (run before pushing)"
-task :verify_image, [:tag] do |_t, args|
-  tag = args[:tag] || "wp2txt-verify:local"
-  checks = IMAGE_FORBIDDEN_PATHS.map { |p| "test -e #{p} && echo LEAK:#{p}" }.join("; ")
-  out, status = Open3.capture2e("docker", "run", "--rm", tag, "sh", "-c", "#{checks}; true")
-  abort "Image verification failed for #{tag}: #{out}" unless status.success?
-  leaks = out.lines.grep(/^LEAK:/).map(&:strip)
-  abort "Image #{tag} contains private paths:\n  #{leaks.join("\n  ")}" unless leaks.empty?
-
-  puts "OK: #{tag} contains none of #{IMAGE_FORBIDDEN_PATHS.join(', ')}"
-end
-
-desc "Build the image locally and verify it, without pushing"
+desc "Build the image from a clean copy of the last commit and run the gate CI runs"
 task :check_image do
-  sh "docker build -t wp2txt-verify:local ."
-  Rake::Task[:verify_image].invoke
+  # A clean clone is what CI builds from: files that exist only in this working
+  # tree never reach the context, and the gate compares the image against it.
+  require "tmpdir"
+  Dir.mktmpdir("wp2txt-image-") do |dir|
+    sh "git", "clone", "--quiet", "--no-local", ".", dir
+    sh "docker", "build", "-t", "wp2txt-verify:local", dir
+    sh "ruby", "scripts/verify_image.rb", "wp2txt-verify:local", "--context", dir
+  end
 end
 
 desc "Explain how images are published (they are built and pushed by CI)"

data/bin/wp2txt

@@ -15,6 +15,7 @@ require_relative "../lib/wp2txt/formatter"
 require_relative "../lib/wp2txt/extractor"
 require_relative "../lib/wp2txt/ractor_worker"
 require_relative "../lib/wp2txt/index_commands"
+require_relative "../lib/wp2txt/lead_terms"
 
 require "etc"
 require "json"
@@ -55,6 +56,40 @@ class WpApp
     used
   end
 
+  # Attach source identifiers and, when asked, lead terms to an article.
+  # raw_wikitext is the article as stored in the dump (XML-decoded, before
+  # comment removal): lead-term positions are offsets into exactly that text.
+  def set_source_fields(article, page_id, revision_id, raw_wikitext, config)
+    article.page_id = page_id
+    article.revision_id = revision_id
+    if config[:page_properties]
+      article.page_properties = Wp2txt::MetadataIndex.property_values(config[:page_properties][page_id])
+    end
+    return unless config[:lead_terms] && raw_wikitext
+
+    article.lead_terms = Wp2txt::LeadTerms.extract(raw_wikitext, render: ->(fragment) { format_wiki(fragment, config) })
+  end
+
+  # Page properties from the metadata index of this dump, when imported
+  # (wp2txt --import-page-props); nil otherwise
+  def load_page_properties(input_path, cache_dir)
+    db_path = Wp2txt::MetadataIndex.path_for(File.expand_path(input_path), cache_dir: cache_dir)
+    return nil unless File.exist?(db_path)
+
+    db = SQLite3::Database.new(db_path, readonly: true)
+    return nil unless Wp2txt::MetadataIndex.page_properties_imported?(db)
+
+    rows = {}
+    db.execute("SELECT page_id, qid, sort_key, disambiguation FROM page_properties") do |id, qid, key, flag|
+      rows[id] = [qid, key, flag]
+    end
+    rows
+  rescue SQLite3::Exception
+    nil
+  ensure
+    db&.close
+  end
+
   # Process articles using turbo mode (split-first architecture from v1.x)
   # This splits the bz2 file into XML chunks first, then processes in parallel
   # Much faster for large dumps due to parallel decompression benefit
@@ -232,6 +267,8 @@ class WpApp
           text = page_xml[text_start...text_end_match.begin(0)]
           next if text.nil? || text.empty?
 
+          raw_wikitext = Wp2txt.xml_decode(text) if config[:lead_terms]
+
           # Decode XML entities
           text = text.gsub("&lt;", "<").gsub("&gt;", ">").gsub("&amp;", "&").gsub("&quot;", '"')
 
@@ -245,8 +282,7 @@ class WpApp
 
           article = Article.new(text, title, strip_tmarker)
           ids = Wp2txt.page_ids(page_xml)
-          article.page_id = ids[:page_id]
-          article.revision_id = ids[:revision_id]
+          set_source_fields(article, ids[:page_id], ids[:revision_id], raw_wikitext, config)
           result = format_article(article, config)
           next unless result
 
@@ -290,7 +326,7 @@ class WpApp
     base_name = base_name.sub(/\.xml$/, "") # Handle .xml.bz2
 
     # Create stream processor
-    stream = StreamProcessor.new(input_path, bz2_gem: bz2_gem)
+    stream = StreamProcessor.new(input_path, bz2_gem: bz2_gem, keep_raw_text: config[:lead_terms])
 
     # Create output writer
     writer = OutputWriter.new(
@@ -444,11 +480,12 @@ class WpApp
   # Process a batch of pages in parallel
   # Uses Ractor for true parallelism when enabled, otherwise falls back to Parallel gem
   def process_batch(pages, writer, config, strip_tmarker, num_processes)
-    results = if config[:use_ractor] && Wp2txt::RactorWorker.available?
+    use_ractor = config[:use_ractor] && Wp2txt::RactorWorker.available?
+    results = if use_ractor
                 # Use Ractor-based parallel processing (true parallelism)
                 Wp2txt::RactorWorker.process_articles(
                   pages,
-                  config: config,
+                  config: config.reject { |key, _| key == :page_properties },
                   strip_tmarker: strip_tmarker,
                   num_workers: num_processes
                 )
@@ -459,13 +496,22 @@ class WpApp
                 writer.flush
                 Parallel.map(pages, in_processes: num_processes) do |title, text, ids|
                   article = Article.new(text, title, strip_tmarker)
-                  article.page_id = ids&.dig(:page_id)
-                  article.revision_id = ids&.dig(:revision_id)
+                  set_source_fields(article, ids&.dig(:page_id), ids&.dig(:revision_id), ids&.dig(:raw_text), config)
                   format_article(article, config)
                 end
               end
 
-    results.each do |result|
+    results.each_with_index do |result, index|
+      # Keep the bulk properties table in the parent, outside Ractor's
+      # shareable config. Results retain the order of the input pages.
+      if use_ractor && result.is_a?(Hash) && config[:page_properties]
+        id = pages[index][2]&.dig(:page_id)
+        properties = Wp2txt::MetadataIndex.property_values(config[:page_properties][id]).transform_keys(&:to_s)
+        result = result.each_with_object({}) do |(key, value), row|
+          row[key] = value
+          row.merge!(properties) if key == "title"
+        end
+      end
       writer.write(result) if result
     end
   end
@@ -724,6 +770,8 @@ class WpApp
     return run_search(opts) if opts[:search]
     return run_fts_optimize(opts) if opts[:fts_optimize]
     return run_import_langlinks(opts) if opts[:import_langlinks]
+    return run_import_page_props(opts) if opts[:import_page_props]
+    return run_count_links(opts) if opts[:count_links]
 
     # Determine input source
     if opts[:from_category] && opts[:lang]
@@ -763,7 +811,7 @@ class WpApp
       no_turbo: opts[:no_turbo]
     }
 
-    %i[title list heading table pre ref redirect multiline category category_only
+    %i[lead_terms title list heading table pre ref redirect multiline category category_only
        summary_only metadata_only marker extract_citations expand_templates
        section_output min_section_length skip_empty
        alias_file no_section_aliases section_stats show_matched_sections].each do |opt|
@@ -778,6 +826,10 @@ class WpApp
     # Parse markers option
     config[:markers] = parse_markers_option(opts[:markers])
 
+    # Imported page properties ride along in JSON output;
+    # loaded here, before any worker is forked, so workers share the table
+    config[:page_properties] = load_page_properties(input_path, opts[:cache_dir]) if format == :json
+
     # Handle section-stats mode (standalone, outputs to stdout)
     if opts[:section_stats]
       return process_section_stats(input_path, config)

data/bin/wp2txt-mcp

@@ -80,7 +80,7 @@ end
 
 server.define_tool(
   name: "dump_info",
-  description: "Dump identity (language/date), available index tiers, and article/category/section counts. Call first to learn what this corpus contains.",
+  description: "Dump identity (language/date), available index tiers, and article/category/section counts, plus provenance and counts for imported page properties (Wikidata IDs, disambiguation flags, sort keys), interlanguage links, and incoming-link counts. Call first to learn what this corpus contains.",
   input_schema: { properties: {}, required: [] }
 ) do |server_context:|
   respond { corpus.dump_info }

data/docs/INDEXES.md

@@ -71,6 +71,66 @@ $ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
 This adds a `langlinks` table (`ll_from` = source page_id, `ll_lang`, `ll_title`) that can
 be joined in SQL. Tip: filter `ll_title != ''` — real dumps contain a few empty-title rows.
 
+### Page properties and incoming links
+
+Two more signals can be added to an existing metadata index, each with one command:
+
+```console
+$ wp2txt --import-page-props -L ja   # each article's Wikidata item ID
+$ wp2txt --count-links -L ja         # how many articles link to each article
+```
+
+- `--import-page-props` reads the official `page_props` dump of the **same date** as the
+  index (a mismatch is refused) and creates `page_properties` with columns `page_id`
+  (integer primary key), `qid` (nullable text), `disambiguation` (integer, default 0,
+  not null), and `sort_key` (nullable text). A row exists only for a page with at least
+  one of these properties.
+  The source file's name, size, SHA-256, import time, page count, each property's count,
+  and the count of invalid UTF-8 sort keys skipped are reported under
+  `dump_info.page_properties`. Once imported, JSON records always carry `qid`,
+  `sort_key`, and `disambiguation`, including null/false values for absent properties.
+  Before import, all three are omitted. `get_article` and `extract_corpus` use the
+  same rule. The three imported properties are `wikibase_item`, `defaultsort`, and
+  `disambiguation`; unrelated properties are ignored.
+  MediaWiki's disambiguation flag is more reliable than matching the page title.
+  Sort keys are language-specific: Japanese Wikipedia commonly removes voicing marks
+  and enlarges small kana (言語 → けんこ), while English may use "surname, given name".
+  A sort key is not a reading itself, but can help compare reading candidates.
+- `--count-links` scans the dump and adds `page_inlinks` (`page_id`, `inlinks`,
+  `via_redirects`) for every article. Each linking article counts **once** per target,
+  however many times it links; a link to a redirect counts for the redirect's target
+  (`via_redirects` is how many articles reached it only that way). Only links written in
+  articles' own text count — links that navigation templates add when a page is rendered
+  are not in the dump, so they are not counted. Links inside comments or literal
+  regions (`nowiki`, `pre`, `math`, `chem`, `ce`, `score`, `syntaxhighlight`, `source`,
+  `graph`, `mapframe`, `templatedata`) and links from redirects or other namespaces
+  do not count. Links inside `gallery`, `timeline`, and ordinary formatting tags
+  such as `code` do count. Lead-term extraction separately excludes `gallery` and
+  `timeline` because their contents are not lead prose.
+  Japanese Wikipedia takes about 12 minutes on
+  an Apple Silicon laptop.
+  Counting rule version 2 decodes title entities and respects the dump's siteinfo case
+  rule, recorded as `case_rule` in metadata when the index is built. Existing indexes
+  without this value use `first-letter`; rebuild the metadata index before counting
+  links for a case-sensitive wiki. Re-run `--count-links` to replace counts from rule 1.
+
+Together with langlinks these let a query select articles by how referenced they are
+within an edition, how many editions cover them, and what Wikidata says they are:
+
+```sql
+SELECT p.title, i.inlinks, q.qid, q.sort_key, q.disambiguation,
+       (SELECT COUNT(DISTINCT ll_lang) FROM langlinks l WHERE l.ll_from = p.page_id) AS editions
+FROM pages p
+JOIN page_inlinks i USING (page_id)
+LEFT JOIN page_properties q USING (page_id)
+WHERE p.namespace = 0 AND p.redirect_to IS NULL AND i.inlinks >= 5
+ORDER BY i.inlinks DESC
+```
+
+Link counts measure how much an edition refers to a topic, not whether it is a proper
+noun: common nouns ("singer", "high school") rank high too. Use the Wikidata ID to tell
+kinds of entities apart.
+
 ## 4. The MCP server
 
 `wp2txt-mcp` exposes a local dump to any MCP-capable LLM client (Claude, ChatGPT, Gemini,
@@ -105,7 +165,7 @@ $ claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yo
 
 | Tool | Purpose |
 |------|---------|
-| `dump_info` | Dump identity, installed indexes, corpus statistics, langlinks provenance |
+| `dump_info` | Dump identity, installed indexes, corpus statistics, provenance of langlinks, Wikidata IDs, and link counts |
 | `get_article` / `get_sections` / `list_headings` / `get_categories` | Single-article access (redirect-aware) |
 | `find_articles` | Exhaustive filtered listing (category recursion, category AND / pattern match, section headings, title match) |
 | `category_tree` / `section_stats` | Scope exploration and heading-frequency discovery |

data/lib/wp2txt/article.rb

@@ -26,7 +26,7 @@ module Wp2txt
   # an article contains elements, each of which is [TYPE, string]
   class Article
     include Wp2txt
-    attr_accessor :elements, :title, :categories, :page_id, :revision_id
+    attr_accessor :elements, :title, :categories, :page_id, :revision_id, :page_properties, :lead_terms
 
     def initialize(text, title = "", strip_tmarker = false)
       @title = title.strip

data/lib/wp2txt/cli.rb

@@ -150,6 +150,16 @@ module Wp2txt
           opt :langlinks_langs, "Comma-separated target languages to import with --import-langlinks (default: all)",
               type: String, short: :none
 
+          # Wikidata item IDs and incoming-link counts, added to an existing metadata index
+          opt :import_page_props, "Import Wikidata IDs, disambiguation flags, and sort keys (page_props dump) into the metadata index (requires --lang)",
+              default: false, short: :none
+          opt :page_props_file, "Use a local page_props .sql(.gz) file instead of downloading (with --import-page-props)",
+              type: String, short: :none
+          opt :count_links, "Count incoming links to each article and store them in the metadata index (requires --lang)",
+              default: false, short: :none
+          opt :lead_terms, "Add the lead's bold terms, their parenthesized notes, and reading templates to JSON output",
+              default: false, short: :none
+
           opt :file_size, "Approximate size (in MB) of each output file (0 for single file)",
               default: 10, short: "-f"
           opt :num_procs, "Number of parallel processes (auto-detected based on CPU/memory)",
@@ -377,6 +387,38 @@ module Wp2txt
           end
         end
 
+        # Page-props import and link counting are standalone modes on an existing index
+        { import_page_props: "--import-page-props", count_links: "--count-links" }.each do |key, flag|
+          next unless opts[key]
+
+          Optimist.die "#{flag} requires --lang" if opts[:lang].nil?
+          others = { build_index: "--build-index", find_articles: "--find-articles", search: "--search",
+                     fts_optimize: "--fts-optimize", import_langlinks: "--import-langlinks",
+                     import_page_props: "--import-page-props", count_links: "--count-links",
+                     articles: "--articles", from_category: "--from-category", section_stats: "--section-stats" }
+          conflicts = others.reject { |k, _| k == key }.select { |k, _| opts[k] }.values
+          Optimist.die "#{flag} cannot be combined with #{conflicts.join(', ')}" unless conflicts.empty?
+        end
+
+        if opts[:page_props_file] && !opts[:import_page_props]
+          Optimist.die "--page-props-file requires --import-page-props"
+        end
+
+        if opts[:page_props_file] && !File.exist?(opts[:page_props_file])
+          Optimist.die :page_props_file, "file does not exist"
+        end
+
+        if opts[:lead_terms] && opts[:format].to_s != "json"
+          Optimist.die "--lead-terms requires --format json"
+        end
+
+        if opts[:lead_terms] && opts[:ractor]
+          Optimist.die "--lead-terms cannot be combined with --ractor"
+        end
+        if opts[:ractor] && opts[:format].to_s == "json"
+          warn "Warning: --ractor JSON output does not include page IDs or revision IDs"
+        end
+
         if opts[:langlinks_file] && !opts[:import_langlinks]
           Optimist.die "--langlinks-file requires --import-langlinks"
         end

data/lib/wp2txt/constants.rb

@@ -6,6 +6,19 @@ module Wp2txt
     (value || "0").to_i
   end
 
+  XML_ENTITIES = { "lt" => "<", "gt" => ">", "amp" => "&", "quot" => '"', "apos" => "'" }.freeze
+
+  # Decode XML character references in one pass, as an XML parser does
+  # (a single pass keeps "&amp;lt;" as the literal text "&lt;")
+  def self.xml_decode(str)
+    str.gsub(/&(?:(lt|gt|amp|quot|apos)|#(\d+)|#x(\h+));/) do
+      if Regexp.last_match(1) then XML_ENTITIES[Regexp.last_match(1)]
+      elsif Regexp.last_match(2) then Regexp.last_match(2).to_i.chr(Encoding::UTF_8)
+      else Regexp.last_match(3).to_i(16).chr(Encoding::UTF_8)
+      end
+    end
+  end
+
   # Page and revision IDs from one <page> element of a dump. The page's own
   # <id> precedes <revision>; the revision's <id> is its first child (the
   # contributor's <id> comes later). Only the header before <text> is read.

data/lib/wp2txt/corpus.rb

@@ -115,7 +115,9 @@ module Wp2txt
         fulltext_current: fts.built? && fts.valid_for?(@multistream_path),
         stats: stats,
         fulltext: fts.built? ? fts.stats : nil,
-        langlinks: @metadata.built? ? @metadata.langlinks_provenance : nil
+        langlinks: @metadata.built? ? @metadata.langlinks_provenance : nil,
+        page_properties: @metadata.built? ? @metadata.page_props_provenance : nil,
+        inlinks: @metadata.built? ? @metadata.links_provenance : nil
       }
     end
 
@@ -147,7 +149,10 @@ module Wp2txt
              else
                render_text(page)
              end
-      result = { id: page[:id], title: page[:title], format: format.to_s }
+      result = { id: page[:id], title: page[:title] }
+      properties = @metadata.built? ? @metadata.properties_for(page[:id]) : nil
+      result.merge!(properties) if properties
+      result[:format] = format.to_s
       if max_chars && body.length > max_chars
         result.merge(text: body[0, max_chars], truncated: true, total_chars: body.length)
       else
@@ -409,6 +414,8 @@ module Wp2txt
               next unless page
 
               records = build_records(page, content, resolved_sections, chunk_size, chunk_overlap)
+              properties = @metadata.built? ? @metadata.properties_for(page[:id]) : nil
+              records.each { |record| record.merge!(properties) } if properties
               next if records.empty?
 
               articles_extracted += 1

data/lib/wp2txt/data/template_aliases.json

@@ -82,7 +82,7 @@
     "Cite web", "Citeweb", "Cit"
   ],
   "ruby_text_templates": [
-    "読み仮名", "よみがな", "ふりがな",
+    "読み仮名", "読み仮名 ruby不使用", "よみがな", "ふりがな",
     "ruby", "Ruby", "ruby-ja", "Ruby-ja",
     "ruby text", "ruby annotation",
     "rubi", "rubytext",

data/lib/wp2txt/extractor.rb

@@ -111,8 +111,7 @@ module Wp2txt
 
         if page
           article = Article.new(page[:text], page[:title], !config[:marker])
-          article.page_id = page[:id]
-          article.revision_id = page[:revision_id]
+          set_source_fields(article, page[:id], page[:revision_id], page[:text], config)
           result = format_article(article, config)
           writer.write(result)
           extracted_count += 1
@@ -255,11 +254,17 @@ module Wp2txt
         bz2_gem: opts[:bz2_gem]
       }
 
-      %i[title list heading table pre ref redirect multiline category category_only
+      %i[lead_terms title list heading table pre ref redirect multiline category category_only
          summary_only metadata_only marker extract_citations expand_templates].each do |opt|
         config[opt] = opts[opt]
       end
 
+      # Page properties, when imported into the index of the full cached dump
+      if format == :json && opts[:lang]
+        full_dump = Wp2txt::DumpManager.new(opts[:lang], cache_dir: opts[:cache_dir]).cached_multistream_path
+        config[:page_properties] = load_page_properties(full_dump, opts[:cache_dir])
+      end
+
       # Section extraction options
       %i[sections section_output min_section_length skip_empty
          alias_file no_section_aliases show_matched_sections].each do |opt|
@@ -482,8 +487,7 @@ module Wp2txt
 
         pages.each do |page|
           article = Article.new(page[:text], page[:title], !config[:marker])
-          article.page_id = page[:id]
-          article.revision_id = page[:revision_id]
+          set_source_fields(article, page[:id], page[:revision_id], page[:text], config)
           result = format_article(article, config)
           writer.write(result)
           extracted_count += 1
@@ -497,8 +501,7 @@ module Wp2txt
 
           if page
             article = Article.new(page[:text], page[:title], !config[:marker])
-            article.page_id = page[:id]
-            article.revision_id = page[:revision_id]
+            set_source_fields(article, page[:id], page[:revision_id], page[:text], config)
             result = format_article(article, config)
             writer.write(result)
             extracted_count += 1