wp2txt 2.3.0 → 2.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.dockerignore +2 -1
- data/.gitignore +1 -0
- data/CHANGELOG.md +81 -275
- data/DEVELOPMENT.md +10 -1
- data/DEVELOPMENT_ja.md +9 -1
- data/Dockerfile +7 -1
- data/README.md +13 -14
- data/README_ja.md +33 -1
- data/Rakefile +28 -5
- data/bin/wp2txt-mcp +1 -1
- data/docs/{RESEARCH.md → INDEXES.md} +64 -63
- data/lib/wp2txt/corpus.rb +27 -0
- data/lib/wp2txt/regex.rb +7 -0
- data/lib/wp2txt/text_processing.rb +8 -0
- data/lib/wp2txt/utils.rb +17 -1
- data/lib/wp2txt/version.rb +1 -1
- data/spec/corpus_spec.rb +25 -0
- data/spec/docs_sync_spec.rb +61 -0
- data/spec/spec_helper.rb +22 -0
- data/spec/utils_spec.rb +129 -0
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 874d5f0c6463fc612080339cbbcbe560b51aba31a8a5e9478d95719c53744051
|
|
4
|
+
data.tar.gz: e8377c0df46cd7d35ca8a85cea4f52e6dc4ca57330004ce5fc6f11913aa1abcf
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ce2ca7660379c37a3f36d6d731aa3a9c1798998c10dfb5ff012503f571653b26aef6324e579c1a8af7f0d2ff52cf90fc8c6a50f44f21d40b73d132277cb50a26
|
|
7
|
+
data.tar.gz: d8197213c46510742d015dcea77c8ff5c40d48dac7beff8cc409ce1e4ce5f3e138e47d7df9ce675a0614785b21bb266664f8c082c94080c4f3a29c712ffeab07
|
data/.dockerignore
CHANGED
data/.gitignore
CHANGED
data/CHANGELOG.md
CHANGED
|
@@ -5,312 +5,118 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Entries were rewritten in August 2026 to describe what changed for people using
|
|
9
|
+
wp2txt, rather than how it was implemented. The changes themselves are unaltered.
|
|
10
|
+
|
|
11
|
+
## [2.3.2] - 2026-08-13
|
|
12
|
+
|
|
13
|
+
- **Documentation release — no code changes.** The changelog and guides bundled with the gem and the container image are rewritten to say what each change means for someone using wp2txt, rather than how it was implemented. The guide formerly at `docs/RESEARCH.md` is now [docs/INDEXES.md](docs/INDEXES.md) ("Offline Indexes, Queries, and the MCP Server")
|
|
14
|
+
|
|
15
|
+
## [2.3.1] - 2026-08-12
|
|
9
16
|
|
|
10
|
-
- **
|
|
11
|
-
- **
|
|
17
|
+
- **Fix: references leaked into extracted text**: reference contents — tag remnants like `ref…/ref` and the bibliographic text inside them — could end up in the extracted body text, breaking sentence splitting and mixing citation details into running prose. In a spot check of five English articles this affected about 70 places per article. References are now removed cleanly (or kept intact as `[ref]…[/ref]` with `--ref`), including references written across several lines and empty ones
|
|
18
|
+
- **Fix: `--extract-citations` ignored multi-line citation templates**: a `{{cite …}}` template split over several lines was left as raw markup instead of being formatted, so citations extracted from such references were silently missing. Single-line and multi-line templates now produce the same result
|
|
19
|
+
- **Long-running queries can no longer outlive their timeout**: an interrupted session could leave a `query_sql` process running at full CPU indefinitely. Queries now stop on their own even if the process that started them goes away. Queries that complete within the time limit are unaffected
|
|
20
|
+
- **Container images move to GitHub Container Registry**: images are published to `ghcr.io/yohasebe/wp2txt` only. The Docker Hub repository is no longer updated — please switch:
|
|
12
21
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
22
|
+
docker pull ghcr.io/yohasebe/wp2txt
|
|
23
|
+
|
|
24
|
+
## [2.3.0] - 2026-07-24
|
|
25
|
+
|
|
26
|
+
- **Extract an explicit list of articles**: `extract_corpus` accepts `titles:` — hand it the set you arrived at some other way (a `query_sql` result, your own list) instead of describing it with filters again. Titles are normalized the way MediaWiki treats them, duplicates are dropped, and one redirect hop is followed. Titles with no article are skipped and reported back as `not_found`, so a missing article is visible rather than silently absent. Up to 10,000 titles; cannot be combined with the filter arguments. Also available through `start_extract_job`
|
|
27
|
+
- **Write a large SQL result to a file**: `query_sql` accepts `output_path:` — all rows go to a JSONL file and the reply carries a summary plus three sample rows, instead of a row cap. The file is written completely or not at all: an interrupted or failing query leaves no partial output. Up to 5 million rows, cells longer than 64KB are clipped (both are reported back), and an existing file is only replaced with `overwrite: true`. A `.meta.json` beside the output records the SQL, the dump versions involved, and the row count
|
|
28
|
+
- **Interlanguage links (`--import-langlinks`)**: import the official `langlinks` dump into the metadata index to map articles across editions. The langlinks file must carry the same dump date as your index — a mismatch is refused, so links can never be mixed across versions. `--langlinks-langs en,de,fr,zh,ko` limits which target languages are imported; re-importing requires `-U`. After the import, a sample of titles per language is checked against any editions you have installed and the match rate is reported. `dump_info` shows what was imported and when
|
|
29
|
+
- **Query several editions in one statement**: `query_sql` accepts `attach: ["en"]`, which makes that language's index available as `en_meta` (and `en_fts` where a full-text index exists) alongside your own. You pass language codes, not paths, and only editions you have installed are accepted. The reply lists what was attached, including each edition's dump name, and says so when the dates differ. Together with the `langlinks` table this makes cross-edition comparisons a single query
|
|
30
|
+
- **Documentation split**: the README now covers text extraction; the index and MCP features (indexes, offline queries, full-text search, interlanguage links, cross-language SQL, MCP server) moved to a separate guide — now [docs/INDEXES.md](docs/INDEXES.md) — which lists every MCP tool
|
|
31
|
+
- **Container images on GitHub Container Registry**: images are now published to `ghcr.io/yohasebe/wp2txt`
|
|
17
32
|
|
|
18
33
|
## [2.2.0] - 2026-07-22
|
|
19
34
|
|
|
20
|
-
- **Index
|
|
21
|
-
- **MCP
|
|
22
|
-
- **
|
|
23
|
-
- **
|
|
24
|
-
- **Full-text search
|
|
25
|
-
- **Background extraction jobs**: `start_extract_job` / `job_status` / `cancel_job` / `list_jobs`
|
|
26
|
-
- **RAG
|
|
27
|
-
- **Alias
|
|
28
|
-
- **`discover_aliases`
|
|
29
|
-
- **MCP server (`wp2txt-mcp`)**:
|
|
30
|
-
- **
|
|
31
|
-
-
|
|
32
|
-
- **
|
|
33
|
-
- **Offline
|
|
35
|
+
- **Index format change — rebuild existing indexes with `--build-index -U`** (schema v2). Section headings are now recorded consistently, so headings written with decoration (`== '''X''' ==`) match section filters instead of being missed. Each index records the wp2txt version that built it, and `dump_info` warns when an index was built by a version whose text cleaning differs from the one you are running. A rebuild that fails partway no longer destroys the index you already had
|
|
36
|
+
- **Safer defaults for MCP use**: `query_sql` stops after 30 seconds and explains what made the query slow; queries containing words like `Update` inside a quoted value are no longer rejected by mistake. `extract_corpus` can only write under the server's output directory and will not replace an existing file unless you pass `overwrite: true`. Background jobs run one at a time
|
|
37
|
+
- **More ways to select articles from an LLM client**: a `get_categories` tool; `categories` (article must be in all of them) and `category_match` (substring on category names) on `find_articles` and `extract_corpus`; `content: "wikitext"` on `extract_corpus` for reading infoboxes and templates that cleaned text replaces with markers; `query_sql` for read-only SQL when the fixed tools cannot express a question, with `describe_schema` to see the tables. Article titles are matched the way MediaWiki treats them (underscores, capitalization). `get_article` stops at `max_chars` (default 40,000) and says when it truncated; `dump_info` reports whether the full-text index is up to date
|
|
38
|
+
- **Split the full-text build in two**: `--skip-fts-optimize` leaves out the final merge step — the slowest part on a full dump, 40–60 minutes and single-threaded — and still gives you a searchable index. Run `--fts-optimize` later when convenient; it is safe to repeat. `dump_info` shows whether the merge has been done
|
|
39
|
+
- **Full-text search**: `--build-index --fulltext` adds a searchable index over the cleaned article text. The tokenizer is chosen by language (character trigrams for Japanese, Chinese, Korean; word-based elsewhere, `--fts-tokenizer` to override). Search from the CLI with `--search` or from an LLM client with `search_text`, combined with category and section filters, as a literal phrase or a full-text query. Counts can be capped for speed or exact — an exact `0` means the term is absent from that dump. The index stores no article text (snippets are re-read from the dump when needed), so it costs far less disk than a copy of the corpus
|
|
40
|
+
- **Background extraction jobs**: `start_extract_job` / `job_status` / `cancel_job` / `list_jobs` for extractions beyond the 5,000-article limit of a direct call. Extraction writes to disk as it goes, so memory use no longer grows with the size of the job
|
|
41
|
+
- **Chunked output for RAG**: `extract_corpus` accepts `chunk_size` / `chunk_overlap` and emits one record per chunk with its `section_path`, `chunk_index`, and `chunk_count`. Chunks prefer to break at sentence and paragraph boundaries
|
|
42
|
+
- **Alias sets are checked before they are saved**: `save_alias_set` refuses a group whose headings frequently appear together in the same article — a sign they are different sections rather than synonyms — unless you pass `force`
|
|
43
|
+
- **`discover_aliases` prompt**: a bundled walkthrough of finding heading variants (`section_stats` → `section_cooccurrence` → save), so you do not have to remember the steps
|
|
44
|
+
- **MCP server (`wp2txt-mcp`)**: a new command that exposes a local dump to an LLM client over the Model Context Protocol. Tools cover dump identity, single articles and their sections, filtered listings, category trees, heading statistics, alias sets, and extraction to JSONL with a `.meta.json` recording the dump version and query. Requires the `mcp` gem (`gem install mcp`); wp2txt itself does not depend on it
|
|
45
|
+
- **Section aliases come from the dump, not from a bundled dictionary**: find the headings actually used with `section_stats`, check whether two of them mean the same thing with `section_cooccurrence`, and save the group with `save_alias_set`. Queries then refer to it by name via `alias_set`, and extractions record which group was used
|
|
46
|
+
- **Ruby API**: `Wp2txt::Corpus` provides the query and extraction layer used by the MCP server, for embedding wp2txt in your own code. Title lookup reads from SQLite on demand rather than loading a multi-million-entry index into memory
|
|
47
|
+
- **Offline metadata index (`--build-index`)**: builds a local index of a dump — each article's categories, section headings, redirects, and the category hierarchy — by scanning it in parallel. Stored under `~/.wp2txt/cache/` and tied to the dump version. No API access needed
|
|
48
|
+
- **Offline queries over a whole edition (`--find-articles`)**: list articles by `--in-category` (with `--depth` to include subcategories), `--has-section` (alias-aware), and `--title-match`. Supports `--limit` and JSON output (`-j json`); redirects are excluded. This is what makes questions like "all film articles that have a plot section" answerable against a pinned dump
|
|
34
49
|
|
|
35
50
|
## [2.1.2] - 2026-07-19
|
|
36
51
|
|
|
37
|
-
- **Fixed gem
|
|
52
|
+
- **Fixed unreadable files after `sudo gem install`**: files in the published gem carried owner-only permissions, so wp2txt failed to run for anyone but the installing user. Packaged files are now readable by everyone (0644, and 0755 for executables). If you hit this, reinstall the gem
|
|
38
53
|
|
|
39
54
|
## [2.1.1] - 2026-02-21
|
|
40
55
|
|
|
41
|
-
- **
|
|
42
|
-
- **
|
|
43
|
-
- **
|
|
56
|
+
- **Section aliases match in both directions**: asking for "Synopsis" now also finds sections headed "Plot", and the other way round, instead of only matching the name you happened to type
|
|
57
|
+
- **More section aliases out of the box**: 12 groups instead of 2, covering common English Wikipedia sections (Plot, Reception, References, Bibliography, Awards, Legacy, Early life, Career, and others)
|
|
58
|
+
- **Fixed options being ignored**: `--pre`, `--ref`, `--expand-templates`, and `--metadata-only` had no effect when used with `--articles` or `--from-category`. They now apply in those modes too
|
|
44
59
|
|
|
45
60
|
## [2.1.0] - 2026-02-19
|
|
46
61
|
|
|
47
|
-
- **
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
- **
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
- Performance: Parallel gem (~3x) remains faster, Ractor (~2x) uses less memory
|
|
68
|
-
|
|
69
|
-
- **Template expansion**: New `--expand-templates` (`-E`) option expands common templates to readable text:
|
|
70
|
-
- Date templates: `{{birth date|1990|5|15}}` → "May 15, 1990"
|
|
71
|
-
- Convert templates: `{{convert|100|km|mi}}` → "100 km (62 mi)"
|
|
72
|
-
- Coordinate templates: `{{coord|35|41|N|139|41|E}}` → "35°41′N 139°41′E"
|
|
73
|
-
- Language templates: `{{lang|ja|日本語}}` → "日本語"
|
|
74
|
-
- Quote templates: `{{blockquote|text}}` → "text"
|
|
75
|
-
- And 20+ more template types
|
|
76
|
-
- **Enabled by default** - use `--no-expand-templates` to disable
|
|
77
|
-
- Parser functions support: `{{#if:}}`, `{{#switch:}}`, `{{#ifeq:}}`, `{{#expr:}}`
|
|
78
|
-
- Magic words support: `{{PAGENAME}}`, `{{CURRENTYEAR}}`, `{{NAMESPACE}}`
|
|
79
|
-
|
|
80
|
-
- **Removed legacy test data**: Deleted obsolete static test files:
|
|
81
|
-
- `data/testdata_en.bz2` (2.8MB, from 2022)
|
|
82
|
-
- `data/testdata_ja.bz2` (2.6MB, from 2022)
|
|
83
|
-
- `data/output_samples/` directory (~20MB)
|
|
84
|
-
- Tests now use live Wikipedia data with caching
|
|
85
|
-
|
|
86
|
-
- **Incremental dump downloads**: Smart handling of partial dump files when downloading full dumps:
|
|
87
|
-
- Detects existing partial downloads and offers to resume (download only remaining data)
|
|
88
|
-
- Validates dump dates - if dates match, can resume; if outdated, offers choices
|
|
89
|
-
- User options: resume download, download fresh, keep old partial, or use old as-is
|
|
90
|
-
- Automatic bz2 validation before and after incremental download
|
|
91
|
-
- Falls back to full download if server doesn't support HTTP Range headers
|
|
92
|
-
|
|
93
|
-
- **bz2 file validation**: New `Bz2Validator` module detects corrupt or invalid bz2 files before processing:
|
|
94
|
-
- Validates magic bytes (`BZ`), version byte (`h`), and block size (`1`-`9`)
|
|
95
|
-
- Optional decompression test to verify file integrity
|
|
96
|
-
- `StreamProcessor` validates bz2 files by default (configurable via `validate_bz2: false`)
|
|
97
|
-
- Detailed error types: `not_found`, `too_small`, `invalid_magic`, `invalid_version`, `invalid_block_size`, `decompression_failed`
|
|
98
|
-
|
|
99
|
-
- **Memory monitoring**: New `MemoryMonitor` module for adaptive resource management:
|
|
100
|
-
- Cross-platform memory detection (Linux, macOS, Windows)
|
|
101
|
-
- Adaptive buffer sizing based on available memory
|
|
102
|
-
- Memory statistics: `current_memory_usage`, `available_memory`, `memory_usage_percent`
|
|
103
|
-
- Automatic garbage collection when memory is low
|
|
104
|
-
|
|
105
|
-
- **Parallel article extraction**: `MultistreamReader` now supports parallel processing:
|
|
106
|
-
- `extract_articles_parallel(titles, num_processes: 4)` - Extract multiple articles in parallel
|
|
107
|
-
- `each_article_parallel(entries, num_processes: 4)` - Iterate with parallel processing
|
|
108
|
-
- Automatically groups articles by stream offset to minimize bz2 decompression overhead
|
|
109
|
-
|
|
110
|
-
- **Performance optimizations**:
|
|
111
|
-
- Pre-compiled 14 additional regex patterns for text cleanup
|
|
112
|
-
- Consolidated gsub chains (3 fewer calls per cleanup operation)
|
|
113
|
-
- Adaptive buffer sizing in `StreamProcessor` based on system memory
|
|
114
|
-
|
|
115
|
-
- **Cache staleness warnings**: Cache status now shows age and staleness information:
|
|
116
|
-
- Displays cache date and age (e.g., "2025-01-05 - 4 days ago")
|
|
117
|
-
- Warns when cache exceeds configured `dump_expiry_days` (default: 30 days)
|
|
118
|
-
- New `--update-cache` (`-U`) option to force refresh of cached dump files
|
|
119
|
-
- Users can choose to use stale cache or force update
|
|
120
|
-
|
|
121
|
-
- **Category-based extraction**: New `--from-category` option extracts all articles from a Wikipedia category:
|
|
122
|
-
- `wp2txt --lang=ja --from-category="日本の都市" -o ./output` extracts all articles in the category
|
|
123
|
-
- `--depth` option for subcategory recursion (e.g., `--depth=2` includes 2 levels of subcategories)
|
|
124
|
-
- `--dry-run` for preview mode (shows article counts without downloading)
|
|
125
|
-
- `--yes` to skip confirmation prompt for automation
|
|
126
|
-
- Circular reference prevention for category hierarchies
|
|
127
|
-
- Rate limiting for Wikipedia API requests
|
|
128
|
-
|
|
129
|
-
- **Configuration file**: New `--config-init` option creates persistent configuration:
|
|
130
|
-
- Settings stored in `~/.wp2txt/config.yml`
|
|
131
|
-
- Configurable: `dump_expiry_days`, `category_expiry_days`, `cache.directory`
|
|
132
|
-
- Default output format and subcategory depth
|
|
133
|
-
- CLI options override config file settings
|
|
134
|
-
|
|
135
|
-
- **Deprecated `--markers=none`**: Complete removal of special content is now deprecated
|
|
136
|
-
- Removing inline content (e.g., math formulas) makes surrounding text nonsensical
|
|
137
|
-
- `--markers=none` now shows a warning and behaves like `--markers=all`
|
|
138
|
-
- Use `--markers=math,code` to show only specific marker types
|
|
139
|
-
|
|
140
|
-
- **CLI option validation**: Extraction modes are now mutually exclusive with clear error messages:
|
|
141
|
-
- `--category-only`, `--summary-only`, `--metadata-only` cannot be combined
|
|
142
|
-
- `--sections` cannot be used with extraction modes
|
|
143
|
-
- `--section-stats` cannot be combined with extraction modes or `--sections`
|
|
144
|
-
|
|
145
|
-
- **Network retry with exponential backoff**: HTTP requests now retry on transient errors:
|
|
146
|
-
- Retries up to 3 times with exponential backoff (2, 4, 8 seconds)
|
|
147
|
-
- Handles timeouts, connection resets, and DNS failures
|
|
148
|
-
- CategoryFetcher API requests now log failures instead of silently returning nil
|
|
149
|
-
|
|
150
|
-
- **Disk full error handling**: OutputWriter now handles `Errno::ENOSPC` gracefully:
|
|
151
|
-
- Raises `Wp2txt::FileIOError` with descriptive message on disk full or I/O errors
|
|
152
|
-
- Properly closes file handles before raising
|
|
153
|
-
|
|
154
|
-
- **File rotation at article boundaries**: OutputWriter `write_from_file` now rotates output files only at blank lines (article boundaries):
|
|
155
|
-
- Prevents articles from being split across output files
|
|
156
|
-
- Eliminates UTF-8 character corruption at file boundaries (e.g., 3-byte Japanese characters split mid-byte)
|
|
157
|
-
- Uses line-by-line reading (`each_line` with `"r:UTF-8"`) instead of fixed-size byte chunks
|
|
158
|
-
- Verified with full Japanese Wikipedia (1.49M articles) and English Wikipedia (24.2 GB) dumps
|
|
159
|
-
|
|
160
|
-
- **HTTP timeout consistency**: All HTTP methods in `DumpManager` now use `DEFAULT_HTTP_TIMEOUT`:
|
|
161
|
-
- Added `open_timeout`/`read_timeout` to `download_incremental`, `get_remote_file_size`, `download_file_with_progress`, `download_file_range`
|
|
162
|
-
- Previously these methods had no timeout, risking indefinite hangs on network issues
|
|
163
|
-
|
|
164
|
-
- **Security: Command injection prevention**: All `IO.popen` calls now use array form:
|
|
165
|
-
- Fixed unsafe string interpolation in `wp2txt.rb`, `stream_processor.rb`, `bz2_validator.rb`, `memory_monitor.rb`
|
|
166
|
-
- Prevents shell metacharacter interpretation in file paths
|
|
167
|
-
|
|
168
|
-
- **Security: SSL certificate verification**: Restored proper TLS certificate validation:
|
|
169
|
-
- Removed `verify_callback` that unconditionally returned `true` (7 locations in `multistream.rb`)
|
|
170
|
-
- `VERIFY_PEER` now performs actual certificate verification
|
|
171
|
-
|
|
172
|
-
- **Security: Temp file handling**: `file_mod` now uses `Tempfile` instead of hardcoded `"temp"` filename:
|
|
173
|
-
- Prevents predictable file names and potential race conditions
|
|
174
|
-
- Temp files created in same directory as target file
|
|
175
|
-
|
|
176
|
-
- **CLI option fixes**:
|
|
177
|
-
- Added missing `--table` option (keep wiki table content)
|
|
178
|
-
- Added missing `--multiline` option (keep multi-line templates)
|
|
179
|
-
- Added missing `--pre` option (keep preformatted text blocks)
|
|
180
|
-
- Fixed `--ref` option not being transferred to processing config
|
|
181
|
-
- Reference removal is now conditional (respects `--ref` flag)
|
|
182
|
-
|
|
183
|
-
- **Ractor turbo mode warning**: Shows explicit warning when `--ractor` is used with turbo mode (unsupported combination)
|
|
184
|
-
|
|
185
|
-
- **Constants extraction**: Replaced magic numbers with named constants:
|
|
186
|
-
- `DEFAULT_HTTP_TIMEOUT`, `DEFAULT_PROGRESS_INTERVAL`, `INDEX_PROGRESS_THRESHOLD`
|
|
187
|
-
- `DEFAULT_TOP_N_SECTIONS`, `RESUME_METADATA_MAX_AGE_DAYS`, `MAX_HTTP_RETRIES`
|
|
188
|
-
|
|
189
|
-
- **Marker classification**: Markers now categorized as inline or block
|
|
190
|
-
- **Inline markers** (`[MATH]`, `[CODE]`, `[CHEM]`, `[IPA]`): Content that appears mid-sentence; removal would break grammar
|
|
191
|
-
- **Block markers** (`[TABLE]`, `[CODEBLOCK]`, `[INFOBOX]`, etc.): Standalone content that can be safely removed
|
|
192
|
-
- New `[CODEBLOCK]` marker for `<syntaxhighlight>`, `<source>`, `<pre>` tags (block-level code)
|
|
193
|
-
- `[CODE]` marker now only applies to inline `<code>` tags
|
|
62
|
+
- **Faster startup and repeated runs**: parsed data files, the Wikipedia category hierarchy (per language, 7-day expiry), and the multistream dump index are now cached under `~/.wp2txt/cache/`. Re-reading the index of a full dump used to take ~10 minutes; later runs now start in seconds
|
|
63
|
+
- **`--ractor` (Ruby 4.0+)**: thread-based parallel processing — about 2× faster than sequential, with a smaller memory footprint than the default process-based mode (which remains the fastest at ~3×). On Ruby 3.x the option falls back to sequential processing
|
|
64
|
+
- **Templates are expanded into readable text by default**: dates (`{{birth date|1990|5|15}}` → "May 15, 1990"), unit conversions (`{{convert|100|km|mi}}` → "100 km (62 mi)"), coordinates, language tags (`{{lang|ja|日本語}}` → 日本語), quotes, and 20+ more, plus common parser functions (`{{#if:}}`, `{{#switch:}}`) and magic words (`{{PAGENAME}}`). Turn off with `--no-expand-templates`
|
|
65
|
+
- **Interrupted downloads resume**: a partial dump download is detected and, when the server supports it, only the remaining data is fetched. If the dump date has changed you are asked whether to resume, redownload, or keep the old file; files are validated before and after
|
|
66
|
+
- **Corrupt dump files are detected before processing** instead of failing partway through, with the specific reason reported (file truncated, not a bz2 file, failed decompression test)
|
|
67
|
+
- **Output files rotate at article boundaries**: an article is never split across two output files, and multi-byte characters are no longer corrupted at file boundaries. Verified against the full Japanese (1.49M articles) and English (24.2 GB) dumps
|
|
68
|
+
- **Downloads can no longer hang indefinitely**: every network operation now has a timeout (several previously had none)
|
|
69
|
+
- **Transient network errors retry automatically** (3 attempts with 2/4/8-second backoff) instead of failing the run, and failed category API requests are logged instead of silently returning nothing
|
|
70
|
+
- **Running out of disk space raises a clear error** instead of leaving corrupt output behind
|
|
71
|
+
- **Cache age is visible**: cache listings show the date and age of each cached dump and warn when it exceeds `dump_expiry_days` (default 30); `--update-cache` (`-U`) forces a refresh
|
|
72
|
+
- **`--from-category`**: extract every article in a Wikipedia category via the Wikipedia API, with `--depth` for subcategory levels, `--dry-run` to preview article counts before downloading, and `--yes` for unattended runs
|
|
73
|
+
- **`--config-init`**: writes a persistent configuration file to `~/.wp2txt/config.yml` (cache lifetimes, cache directory, output defaults). Command-line options still take precedence
|
|
74
|
+
- **`--markers=none` is deprecated**: removing inline content (a formula mid-sentence) leaves broken text, so the option now warns and behaves like `--markers=all`. Use `--markers=math,code` to keep only specific marker types
|
|
75
|
+
- **Markers are classified as inline or block**: inline markers (`[MATH]`, `[CODE]`, `[CHEM]`, `[IPA]`) stand in for content whose removal would break the sentence; block markers (`[TABLE]`, `[INFOBOX]`, …) replace standalone content. New `[CODEBLOCK]` marker for code blocks; `[CODE]` now covers only inline code
|
|
76
|
+
- **Conflicting options are rejected with a clear message**: `--category-only`, `--summary-only`, and `--metadata-only` are mutually exclusive, and `--sections` / `--section-stats` cannot be combined with them
|
|
77
|
+
- **Option fixes**: `--table`, `--multiline`, and `--pre` were missing from the CLI and are now available; `--ref` was accepted but had no effect — reference removal now respects it
|
|
78
|
+
- `--ractor` combined with turbo mode now warns explicitly (the combination is unsupported)
|
|
79
|
+
- **Security fixes**: file paths are no longer passed through a shell, so crafted path names cannot inject commands; TLS certificate verification is actually performed (it was previously disabled by an accept-everything callback); temporary files use unpredictable names instead of a fixed name in the working directory
|
|
80
|
+
- **Performance**: text cleanup is faster (fewer passes over each article), and buffer sizes adapt to the memory actually available on the machine
|
|
81
|
+
- **Ruby API**: articles can be extracted from a multistream dump in parallel (`extract_articles_parallel`, `each_article_parallel`), grouped by stream to avoid decompressing the same block twice
|
|
194
82
|
|
|
195
83
|
## [2.0.0] - 2026-01-08
|
|
196
84
|
|
|
197
85
|
### Added
|
|
198
86
|
|
|
199
|
-
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
- **
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
- **
|
|
210
|
-
|
|
211
|
-
- `--cache-clear` - Clear all cache
|
|
212
|
-
- `--cache-clear --lang=ja` - Clear cache for specific language
|
|
213
|
-
- `--cache-dir` - Custom cache directory
|
|
214
|
-
|
|
215
|
-
- **Content type markers**: New `--markers` option marks special content:
|
|
216
|
-
- Supported types: `[MATH]`, `[CODE]`, `[CHEM]`, `[TABLE]`, `[SCORE]`, `[TIMELINE]`, `[GRAPH]`, `[IPA]`, `[INFOBOX]`, `[NAVBOX]`, `[GALLERY]`, `[SIDEBAR]`, `[MAPFRAME]`, `[IMAGEMAP]`, `[REFERENCES]`
|
|
217
|
-
- `--markers=all` (default) - Enable all markers
|
|
218
|
-
- `--markers=none` - Disable markers (content removed)
|
|
219
|
-
- `--markers=math,code` - Enable specific markers only
|
|
220
|
-
|
|
221
|
-
- **Citation extraction**: New `--extract-citations` (`-C`) option for formatted bibliography output:
|
|
222
|
-
- Extracts author, title, and year from `{{cite book}}`, `{{cite web}}`, `{{Citation}}` templates
|
|
223
|
-
- Formats citations as "Author. \"Title\". Year."
|
|
224
|
-
- Available via CLI (`--extract-citations`) and Ruby API (`extract_citations: true`)
|
|
225
|
-
|
|
226
|
-
- **Multistream support**: New classes for efficient Wikipedia dump processing:
|
|
227
|
-
- `MultistreamIndex` - Parse multistream index files
|
|
228
|
-
- `MultistreamReader` - Extract articles from multistream dumps
|
|
229
|
-
- `DumpManager` - Download and cache dump files
|
|
230
|
-
- Enables targeted article extraction without downloading full dump
|
|
231
|
-
|
|
232
|
-
- **Validation framework**: New rake tasks for validating Wikipedia dump processing:
|
|
233
|
-
- `testdata:prepare[lang,level]` - Download and cache test data
|
|
234
|
-
- `validate:run[lang,level]` - Run validation on cached data
|
|
235
|
-
- `validate:full[lang]` - Full dump validation
|
|
236
|
-
|
|
237
|
-
- **HTML entity management**: Comprehensive entity support from authoritative sources:
|
|
238
|
-
- 2125 entities from WHATWG HTML specification (`html_entities.json`)
|
|
239
|
-
- Wikipedia-specific entities (`wikipedia_entities.json`): `∶`, `‐`, `&nbso;`
|
|
240
|
-
- New script `scripts/fetch_html_entities.rb` to update from WHATWG
|
|
241
|
-
- Replaces hardcoded entity list with data-driven approach
|
|
242
|
-
|
|
243
|
-
- **MediaWiki data auto-generation**: Magic words and namespace aliases fetched from all Wikipedia APIs:
|
|
244
|
-
- New script `scripts/fetch_mediawiki_data.rb` queries 350+ Wikipedia language editions
|
|
245
|
-
- Data stored in `lib/wp2txt/data/mediawiki_aliases.json`
|
|
246
|
-
- 176 redirect keywords, 231 category aliases, 313 file aliases
|
|
247
|
-
- Run `ruby scripts/fetch_mediawiki_data.rb` to update
|
|
248
|
-
|
|
249
|
-
- **JSON/JSONL output format**: New `--format json` option outputs articles as JSONL (one JSON object per line) with `title`, `categories`, `text`, and `redirect` fields. Ideal for data pipelines and machine learning workflows.
|
|
250
|
-
|
|
251
|
-
- **Streaming processing**: Complete rewrite of the processing architecture:
|
|
252
|
-
- No longer creates intermediate XML files
|
|
253
|
-
- Directly streams from bz2 compressed files
|
|
254
|
-
- Reduced disk I/O and storage requirements
|
|
255
|
-
- New `StreamProcessor` and `OutputWriter` classes for modular design
|
|
256
|
-
|
|
257
|
-
- **Regex cache**: Dynamic regex patterns are now cached to avoid repeated compilation
|
|
258
|
-
|
|
259
|
-
- **Multilingual category support**: Added support for category namespaces in 30+ languages (European, Cyrillic, Asian, Middle Eastern)
|
|
260
|
-
|
|
261
|
-
- **Multilingual redirect support**: Added support for redirect keywords in 25+ languages
|
|
262
|
-
|
|
263
|
-
- **Comprehensive test suite**: 395 tests covering:
|
|
264
|
-
- Unicode handling (CJK, Cyrillic, Arabic, emoji)
|
|
265
|
-
- Edge cases (deeply nested templates, malformed markup)
|
|
266
|
-
- Multilingual category and redirect extraction
|
|
267
|
-
- Text processing utilities
|
|
268
|
-
- Integration tests with real Wikipedia content
|
|
269
|
-
|
|
270
|
-
- **SimpleCov integration**: Added code coverage reporting for development
|
|
271
|
-
|
|
272
|
-
- **Ruby 4.0 compatibility**: Full support for Ruby 4.0
|
|
87
|
+
- **`--lang`**: downloads the Wikipedia dump for any language code and processes it in one step (`wp2txt --lang=ja -o ./output`). Downloads are cached under `~/.wp2txt/cache/` for reuse
|
|
88
|
+
- **`--articles`**: extract specific articles by title (`wp2txt --lang=en --articles="Tokyo,Kyoto,Osaka" -o ./articles`), downloading only the index and the data streams that contain them rather than the whole dump
|
|
89
|
+
- **Cache management**: `--cache-status` (per-language overview), `--cache-clear` (all languages, or one with `--lang`), and `--cache-dir` (custom location)
|
|
90
|
+
- **Content markers (`--markers`)**: special content is replaced by a marker instead of disappearing silently: `[MATH]`, `[CODE]`, `[CHEM]`, `[TABLE]`, `[SCORE]`, `[TIMELINE]`, `[GRAPH]`, `[IPA]`, `[INFOBOX]`, `[NAVBOX]`, `[GALLERY]`, `[SIDEBAR]`, `[MAPFRAME]`, `[IMAGEMAP]`, `[REFERENCES]`. `--markers=all` is the default; `--markers=math,code` keeps only the listed types
|
|
91
|
+
- **Citation extraction (`--extract-citations`, `-C`)**: outputs a formatted bibliography ("Author. \"Title\". Year.") from `{{cite book}}`, `{{cite web}}`, and `{{Citation}}` templates. Also available from the Ruby API (`extract_citations: true`)
|
|
92
|
+
- **JSON/JSONL output (`--format json`)**: one JSON object per line with `title`, `categories`, `text`, and `redirect` fields, for data pipelines and machine-learning workflows
|
|
93
|
+
- **Multistream dumps are handled natively**, which is what makes targeted extraction work without downloading a full dump. Ruby API: `MultistreamIndex`, `MultistreamReader`, `DumpManager`
|
|
94
|
+
- **HTML character entities are converted comprehensively**: 2,125 entities from the WHATWG HTML specification, plus Wikipedia-specific ones (`∶`, `‐`, `&nbso;`)
|
|
95
|
+
- **Redirect, category, and file keywords fetched from 350+ Wikipedia language editions** (176 redirect keywords, 231 category aliases, 313 file aliases), replacing a small hardcoded list
|
|
96
|
+
- **Categories are recognized in 30+ languages** (European, Cyrillic, Asian, and Middle Eastern scripts) and **redirects in 25+**
|
|
97
|
+
- **Fully streamed processing**: dumps are processed directly from the compressed file, with no intermediate XML files — far less disk space and I/O than before
|
|
98
|
+
- **Ruby 4.0 support**
|
|
273
99
|
|
|
274
100
|
### Changed
|
|
275
101
|
|
|
276
|
-
- **
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
- `remove_complex`, `make_reference`: Optimized with `gsub!`
|
|
280
|
-
- Category deduplication: Changed from O(n²) to O(n) by calling `uniq!` once at end instead of every line
|
|
281
|
-
- `correct_separator`: Uses `tr` instead of `gsub` for single character replacement
|
|
282
|
-
- `remove_inbetween`: Dynamic regex patterns are now cached
|
|
283
|
-
|
|
284
|
-
- **BREAKING**: `REMOVE_HR_REGEX` now matches 4 or more hyphens (previously 3+) to align with MediaWiki specification where `----` is the minimum for horizontal rules
|
|
285
|
-
|
|
286
|
-
- **`chrref_to_utf` function**: Completely rewritten to support all Unicode codepoints (U+0001 to U+10FFFF), including:
|
|
287
|
-
- Supplementary plane characters (emoji, CJK Extension B, etc.)
|
|
288
|
-
- Proper handling of invalid codepoints (returns empty string)
|
|
289
|
-
|
|
290
|
-
- **`convert_characters` function**: Now uses `String#scrub` for safe handling of invalid UTF-8 sequences instead of calling `exit`
|
|
291
|
-
|
|
292
|
-
- **`command_exist?` function**: Updated to use `IO.popen` instead of `open("| ...")` for Ruby 4.0 compatibility
|
|
102
|
+
- **BREAKING: horizontal rules now require 4 or more hyphens** (previously 3), matching the MediaWiki specification — a line of `---` in an article is now treated as content, not as a rule
|
|
103
|
+
- **Character references convert across the entire Unicode range** (U+0001–U+10FFFF), including emoji and supplementary-plane CJK; invalid codepoints become empty strings instead of garbage characters
|
|
104
|
+
- **Faster text processing**: category deduplication went from quadratic to linear time, and string handling allocates far less memory on large articles
|
|
293
105
|
|
|
294
106
|
### Fixed
|
|
295
107
|
|
|
296
|
-
- **
|
|
297
|
-
|
|
298
|
-
- **
|
|
299
|
-
|
|
300
|
-
- **Horizontal rule detection**: Fixed `REMOVE_HR_REGEX` to correctly match MediaWiki horizontal rules (4+ hyphens)
|
|
301
|
-
|
|
302
|
-
- **Heading regex**: Fixed `IN_HEADING_REGEX` to allow trailing whitespace after closing equal signs
|
|
303
|
-
|
|
304
|
-
- **Ruby 4.0 compatibility**: Fixed `open("| which cmd")` pattern which no longer works in Ruby 4.0
|
|
108
|
+
- **Character references beyond U+FFFF now produce the right character**: `😀` becomes 😀 instead of an invalid character
|
|
109
|
+
- **Invalid UTF-8 in a dump no longer aborts the run**: bad byte sequences are scrubbed and processing continues (the tool previously exited mid-run)
|
|
110
|
+
- **Headings with trailing whitespace** after the closing `==` are now recognized as headings
|
|
305
111
|
|
|
306
112
|
### Deprecated
|
|
307
113
|
|
|
308
|
-
- **`--convert` / `-c
|
|
309
|
-
- **`--del-interfile` / `-x
|
|
114
|
+
- **`--convert` / `-c`**: no longer needed — streaming processing always converts
|
|
115
|
+
- **`--del-interfile` / `-x`**: no longer needed — intermediate files are no longer created
|
|
310
116
|
|
|
311
117
|
### Removed
|
|
312
118
|
|
|
313
|
-
- **Intermediate XML
|
|
119
|
+
- **Intermediate XML files**: processing no longer writes intermediate XML next to the output at any point
|
|
314
120
|
|
|
315
121
|
### Security
|
|
316
122
|
|
data/DEVELOPMENT.md
CHANGED
|
@@ -383,6 +383,14 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
|
|
|
383
383
|
2. Redirect keywords: `data/language_redirects.json`
|
|
384
384
|
3. Scripts: `scripts/generate_language_data.rb`
|
|
385
385
|
|
|
386
|
+
## Documentation
|
|
387
|
+
|
|
388
|
+
User-facing documents (README, README_ja, CHANGELOG, docs/) describe what a user can
|
|
389
|
+
observe and what they need to do — not how the code works internally or why it is
|
|
390
|
+
designed the way it is. Keep implementation detail in this file and in code comments,
|
|
391
|
+
and keep design rationale out of the public repository entirely. The MCP tool table in
|
|
392
|
+
docs/INDEXES.md is checked against the actual server surface by spec/docs_sync_spec.rb.
|
|
393
|
+
|
|
386
394
|
## Code Style
|
|
387
395
|
|
|
388
396
|
- Ruby 2.6+ compatibility
|
|
@@ -395,7 +403,8 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
|
|
|
395
403
|
Build and push Docker images:
|
|
396
404
|
|
|
397
405
|
```bash
|
|
398
|
-
rake
|
|
406
|
+
rake check_image # Builds the image locally and verifies it carries no private files
|
|
407
|
+
rake push # Verifies, then builds multi-arch and pushes to GHCR
|
|
399
408
|
```
|
|
400
409
|
|
|
401
410
|
## Release Process
|
data/DEVELOPMENT_ja.md
CHANGED
|
@@ -383,6 +383,13 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
|
|
|
383
383
|
2. リダイレクトキーワード: `data/language_redirects.json`
|
|
384
384
|
3. スクリプト: `scripts/generate_language_data.rb`
|
|
385
385
|
|
|
386
|
+
## ドキュメンテーション
|
|
387
|
+
|
|
388
|
+
利用者向け文書（README, README_ja, CHANGELOG, docs/）には、利用者が観測できることと
|
|
389
|
+
利用者が取るべき行動だけを書く — コードが内部でどう動くか、なぜそう設計したかは書かない。
|
|
390
|
+
実装の詳細は本ファイルとコードコメントに、設計の理由づけは公開リポジトリの外に置く。
|
|
391
|
+
docs/INDEXES.md の MCP ツール表は spec/docs_sync_spec.rb が実際のサーバー表面と照合する。
|
|
392
|
+
|
|
386
393
|
## コードスタイル
|
|
387
394
|
|
|
388
395
|
- Ruby 2.6+互換性
|
|
@@ -395,7 +402,8 @@ info = Wp2txt::Bz2Validator.file_info("/path/to/file.bz2")
|
|
|
395
402
|
Dockerイメージのビルドとプッシュ：
|
|
396
403
|
|
|
397
404
|
```bash
|
|
398
|
-
rake
|
|
405
|
+
rake check_image # ローカルでイメージをビルドし、私的ファイルの混入がないか検証
|
|
406
|
+
rake push # 検証したうえでマルチアーキテクチャでビルドしGHCRにプッシュ
|
|
399
407
|
```
|
|
400
408
|
|
|
401
409
|
## リリースプロセス
|
data/Dockerfile
CHANGED
|
@@ -7,7 +7,12 @@ WORKDIR /wp2txt
|
|
|
7
7
|
COPY . ./
|
|
8
8
|
RUN rm -f Gemfile.lock
|
|
9
9
|
|
|
10
|
-
# Install dependencies (git is required by gemspec's `git ls-files`)
|
|
10
|
+
# Install dependencies (git is required by gemspec's `git ls-files`).
|
|
11
|
+
# The repository's .gitignore is copied in deliberately: `git add -A` must
|
|
12
|
+
# honour it so the file list here matches a local `gem build` — without it,
|
|
13
|
+
# ignored material (private notes, scratch files) would land in the image.
|
|
14
|
+
# The throwaway .git is removed afterwards: it holds a blob copy of every
|
|
15
|
+
# added file and is dead weight in the published image.
|
|
11
16
|
RUN apk update && \
|
|
12
17
|
apk upgrade && \
|
|
13
18
|
apk add --no-cache \
|
|
@@ -17,6 +22,7 @@ RUN apk update && \
|
|
|
17
22
|
build-base curl-dev wget && \
|
|
18
23
|
git init && git add -A && \
|
|
19
24
|
bundle install -j4 && \
|
|
25
|
+
rm -rf /wp2txt/.git && \
|
|
20
26
|
apk del .build-packages
|
|
21
27
|
|
|
22
28
|
# lbzip2 is not available as an Alpine package; build from source
|