mwoffliner 1.17.4 → 2.0.0

package/README.md

@@ -1,19 +1,10 @@
 # MWoffliner
 
-MWoffliner is a tool for making a local offline HTML snapshot of any
-online [MediaWiki](https://mediawiki.org) instance. It goes through
-all online articles (or a selection if specified) and create the
-corresponding [ZIM](https://openzim.org) file. It has mainly been
-tested against Wikimedia projects like
-[Wikipedia](https://wikipedia.org) and
-[Wiktionary](https://wiktionary.org) --- but it should also work for
-any recent MediaWiki.
+MWoffliner is a tool for creating a local offline HTML snapshot of any online [MediaWiki](https://mediawiki.org) instance. It scrapes all pages (or a selection if specified) and creates the corresponding [ZIM](https://openzim.org) file. While primarily targeted for Wikimedia projects like [Wikipedia](https://wikipedia.org) and [Wiktionary](https://wiktionary.org), MWoffliner also supports any recent MediaWiki instance (version 1.27+), though instances with custom skins or highly unusual configurations may have limitations.
 
-Read [CONTRIBUTING.md](./CONTRIBUTING.md) to know more about
-MWoffliner development.
+Read [CONTRIBUTING.md](./CONTRIBUTING.md) to learn more about MWoffliner development.
 
-User Help is available in the for a a
-[FAQ](https://github.com/openzim/mwoffliner/wiki/Frequently-Asked-Questions).
+User help is available in the [FAQ](https://github.com/openzim/mwoffliner/wiki/Frequently-Asked-Questions).
 
 [![NPM](https://nodei.co/npm/mwoffliner.png)](https://www.npmjs.com/package/mwoffliner)
 
@@ -28,53 +19,282 @@ User Help is available in the for a a
 
 ## Features
 
-- Scrape with or without image thumbnail
+- Scrape with or without image thumbnails
 - Scrape with or without audio/video multimedia content
 - S3 cache (optional)
-- Image size optimiser / Webp converter
-- Scrape all articles in namespaces or title list based
+- Image size optimization and WebP conversion
+- Scrape all pages in namespaces or title list based
 - Specify additional/non-main namespaces to scrape
 
-Run `mwoffliner --help` to get all the possible options.
+Run `mwoffliner --help` to see all available options.
 
-## Prerequisites
+## Quick Start
 
-- *NIX Operating System (GNU/Linux, macOS, ...)
-- [Redis](https://redis.io/)
-- [NodeJS](https://nodejs.org/en/) version 24 (we support only one single Node.JS version, other versions might work or not)
-- [Libzim](https://github.com/openzim/libzim) (On GNU/Linux & macOS we automatically download it)
-- Various build tools which are probably already installed on your
-  machine (packages `libjpeg-dev`, `libglu1`, `autoconf`, `automake`, `gcc` on
-  Debian/Ubuntu)
+### Prerequisites
 
-... and an online MediaWiki with its API available.
+- [Docker](https://docs.docker.com/engine/install/) (or Docker-based engine)
+- amd64 or arm64 architecture
 
-## Usage
+To turn the Bambara Wikipedia into an offline ZIM:
 
-To install latest released MWoffliner version from NPM repo (use `-g` to install globally, not only in current folder):
-```bash
-npm i -g mwoffliner
+```sh
+mkdir -p output
+docker run -v $(pwd)/output:/output ghcr.io/openzim/mwoffliner \
+  mwoffliner --mwUrl=https://bm.wikipedia.org --adminEmail=you@example.com \
+  --outputDirectory=/output
 ```
 
-> [!WARNING] 
-> Note that you might need to run this command with the `sudo` command, depending
-how your `npm` / OS is configured. `npm` permission checking can be a bit annoying for a
-newcomer. Please read the documentation carefully if you hit problems: https://docs.npmjs.com/cli/v7/using-npm/scripts#user
+**NOTE**: In order to avoid [429 responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429), be sure to use an
+actual email address as dummy addresses like `you@example.com` will be flagged by the Wiki servers.
 
-Then you can run the scraper:
-```bash
-mwoffliner --help
+That's it. When it finishes you'll find a `.zim` file in the `output/` directory. Open it with
+[Kiwix Reader](https://get.kiwix.org/en/solutions/applications/kiwix-reader/).
+
+### The Four Decisions
+
+Every scrape comes down to four choices:
+
+#### 1. Which wiki?
+
+```sh
+--mwUrl=https://en.wikipedia.org        # English Wikipedia
+--mwUrl=https://fr.wiktionary.org       # French Wiktionary
+--mwUrl=https://terraria.wiki.gg        # Terraria Wiki
+--mwUrl=https://proofwiki.org           # ProofWiki
+--mwUrl=https://wiki.my-org.com         # Any MediaWiki site
 ```
 
-To use MWoffliner with a S3 cache, you should provide a S3 URL like
-this:
-```bash
---optimisationCacheUrl="https://wasabisys.com/?bucketName=my-bucket&keyId=my-key-id&secretAccessKey=my-sac"
+#### 2. Which API path?
+
+Most MediaWiki sites use the default API path `/w/api.php`, but many
+don't. Check by visiting `<wiki-url>/w/api.php` in your browser. If that returns a 404, you need to set `--mwActionApiPath` to the correct path.
+
+> [!TIP]
+> Try `https://terraria.wiki.gg/api.php` to see a wiki where
+> `--mwActionApiPath=/api.php` is required.
+
+#### 3. What content?
+
+| If you want…                            | Add this                        |
+| --------------------------------------- | ------------------------------- |
+| Everything (text, images, audio, video) | (nothing — this is the default) |
+| Everything except video and audio       | `--format=novid:maxi`           |
+| No pictures, no video, no audio         | `--format=nopic:nopic`          |
+| Head paragraphs only, no media at all   | `--format=nodet,nopic:mini`     |
+
+#### 4. Where does it go?
+
+The ZIM is written to `/output` inside the container. Map that to a folder on your machine:
+
+```sh
+docker run -v /path/on/my/machine:/output ghcr.io/openzim/mwoffliner ...
+```
+
+**NOTE**: `--adminEmail=you@example.com` is also required. It is included in the HTTP
+User-Agent so wiki operators know who is scraping.
+
+### A Few More Things You Might Want
+
+#### Scrape only specific pages
+
+Pass a comma-separated list of page titles directly:
+
+```sh
+--pageList="Main Page,Earth,Albert Einstein"
+```
+
+Or put one page title per line in a text file and point to it:
+
+```sh
+--pageList=./my-pages.txt
+```
+
+#### Scrape a private wiki
+
+Provide credentials with `--mwUsername` and `--mwPassword`:
+
+```sh
+--mwUsername=jdoe --mwPassword=s3cret
+```
+
+Preferably use a [bot password](https://www.mediawiki.org/wiki/Manual:Bot_passwords)
+rather than a regular user account.
+
+If authentication requires a separate domain, also pass `--mwDomain`:
+
+```sh
+--mwDomain=corp --mwUsername=jdoe --mwPassword=s3cret
 ```
 
+#### Customising the Result
+
+Want your ZIM to have a specific title, description, or icon?
+
+```sh
+--customZimTitle="My Offline Wiki"
+--customZimDescription="A hand-picked selection of articles"
+--customZimFavicon=https://example.com/icon.png
+```
+
+#### Adjusting the scrape speed
+
+Since version 2.0.0, the default request rate (speed `1`) is fine for most
+wikis. The `--speed` option controls the climb rate — how aggressively
+mwoffliner ramps up its request concurrency:
+
+If you see lots of HTTP errors (e.g. 429 Too Many Requests) in the logs,
+try lowering the speed e.g `--speed=0.5` can help prevent
+the wiki from rate-limiting you.
+
+#### Going Further
+
+These and all other options are listed in `mwoffliner --help`.
+
+Also, see the [FAQ](https://github.com/openzim/mwoffliner/wiki/Frequently-Asked-Questions) for detailed explanations of
+the command line options and common issues.
+
+Need help? [![Join Slack](https://img.shields.io/badge/Join%20us%20on%20Slack%20%23mwoffliner-2EB67D)](https://slack.kiwix.org)
+
+## Installation
+
+The recommended way to install and run `mwoffliner` is using the pre-built Docker container:
+
+```sh
+docker pull ghcr.io/openzim/mwoffliner
+```
+
+<details>
+<summary>Run software locally / Build from source</summary>
+
+### Prerequisites for local execution
+
+- \*NIX Operating System (GNU/Linux, macOS, etc.)
+- [Redis](https://redis.io/) — in-memory data store
+- [Node.js](https://nodejs.org/en/) version 24 (we support only one single Node.js version; other versions might work or might not)
+- [Libzim](https://github.com/openzim/libzim) — C++ library for creating ZIM files (automatically downloaded on GNU/Linux & macOS)
+- Various build tools which are probably already installed on your machine:
+  - `libjpeg-dev` — JPEG image processing
+  - `libglu1` — OpenGL utility library
+  - `autoconf` — automatic configuration system
+  - `automake` — Makefile generator
+  - `gcc` — C compiler
+
+  (These packages are for Debian/Ubuntu systems)
+
+An online [MediaWiki](https://mediawiki.org) instance with its API available.
+
+### Installation methods
+
+#### Build your own container
+
+1. Clone the repository locally:
+
+   ```sh
+   git clone https://github.com/openzim/mwoffliner.git && cd mwoffliner
+   ```
+
+1. Build the image:
+
+   ```sh
+   docker build . -f docker/Dockerfile -t ghcr.io/openzim/mwoffliner
+   ```
+
+#### Run the software locally using NPM
+
+> [!WARNING]
+> Local installation requires several system dependencies (see above). Using the Docker image is strongly recommended to avoid setup issues.
+
+Setting up MWoffliner locally for development can be tricky due to several dependencies and version requirements. Follow these steps carefully to avoid common errors.
+
+##### 1. Node.js Version
+
+MWoffliner requires Node.js 24 (other versions may fail).
+
+Compatible Node 24 ranges: `>=24 <24.6` or `>=24.7 <25`.
+
+Check your version:
+
+```sh
+node -v
+```
+
+If your version does not match, use [nvm](https://github.com/nvm-sh/nvm) to install the correct Node.js version.
+
+##### 2. libzim Dependency
+
+MWoffliner depends on [`@openzim/libzim`](https://github.com/openzim/libzim), which requires the C++ libzim library.
+
+- On Linux/macOS, MWoffliner can download libzim automatically.
+- On Windows, you must install libzim manually because there are no prebuilt binaries. See the [libzim installation guide](https://github.com/openzim/libzim) for details.
+
+##### 3. Compiler Requirements (Windows)
+
+Node 24 on Windows officially supports [Visual Studio 2019 (v16)](https://visualstudio.microsoft.com/vs/older-downloads/) or [Visual Studio 2022 (v17)](https://visualstudio.microsoft.com/downloads/).
+
+Ensure C++ build tools are installed and environment variables are set correctly. See [Windows Setup for node-gyp](https://github.com/nodejs/node-gyp#on-windows) for detailed instructions.
+
+##### 4. Node-gyp
+
+MWoffliner uses [node-gyp](https://github.com/nodejs/node-gyp), which enforces strict checks for Node and compiler versions. Make sure you have:
+
+- Proper Visual Studio version (Windows) — see [Visual Studio versions](https://visualstudio.microsoft.com/downloads/)
+- Required C++ headers, e.g., `zim/archive.h` — see [libzim documentation](https://github.com/openzim/libzim)
+- [Python 3.10+](https://www.python.org/downloads/) (required by node-gyp; a recent version is preferred for compatibility)
+
+##### Additional troubleshooting steps if errors persist:
+
+1. **Clear npm cache** — a corrupted cache can cause cryptic install failures:
+
+   ```sh
+   npm cache clean --force
+   ```
+
+2. **Delete node_modules and reinstall** — stale or partially installed dependencies are a common source of errors:
+
+   ```sh
+   rm -rf node_modules package-lock.json
+   npm install
+   ```
+
+3. **Check that all environment variables are set** — especially on Windows, `PATH`, `INCLUDE`, and `LIB` must point to the correct Visual Studio and libzim directories. Reopen your terminal after installing new tools.
+
+4. **Verify Redis is running before starting MWoffliner** — MWoffliner will fail immediately if it cannot connect to Redis:
+
+   ```sh
+   redis-cli ping   # expected output: PONG
+   ```
+
+5. **Run npm install with verbose logging** to see exactly where it fails:
+   ```sh
+   npm install --verbose
+   ```
+
+##### 5. Common Errors & Troubleshooting
+
+| Error                              | Cause                          | Solution                                                                                                                                                                 |
+| ---------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Node.js version error              | Node.js version incompatible   | Install [Node 24 with nvm](https://github.com/nvm-sh/nvm)                                                                                                                |
+| Cannot find module @openzim/libzim | libzim not installed           | Follow [libzim installation guide](https://github.com/openzim/libzim); Windows users must install manually                                                               |
+| node-gyp rebuild failed            | Wrong Node or compiler version | Check [Node.js version](https://nodejs.org/en/), [Visual Studio version](https://visualstudio.microsoft.com/downloads/), [Python 3.x](https://www.python.org/downloads/) |
+| zim/archive.h not found            | C++ headers missing            | Install [libzim](https://github.com/openzim/libzim) system-wide, verify include paths                                                                                    |
+
+> [!NOTE]
+> Even with these steps, other setup errors may occur. Using Docker is strongly recommended for a smoother experience.
+
+##### Installation via NPM
+
+```sh
+npm i -g mwoffliner
+```
+
+> [!WARNING]
+> You might need to run this command with the `sudo` command, depending on how your `npm` / OS is configured. `npm` permission checking can be a bit annoying for newcomers. Please read the [npm script documentation](https://docs.npmjs.com/cli/v7/using-npm/scripts#user) if you encounter issues.
+
+</details>
+
 ## Contribute
 
-If you've retrieved mwoffliner source code (e.g. with a git clone of our repo), you can then install and run it locally (including with your local modifications):
+If you've retrieved the MWoffliner source code (e.g., via a git clone), you can install and run it locally with your modifications:
 
 ```bash
 npm i
@@ -85,35 +305,108 @@ Detailed [contribution documentation and guidelines](CONTRIBUTING.md) are availa
 
 ## API
 
-MWoffliner provides also an API and therefore can be used as a NodeJS
-library. Here a stub example that could go in your index.mjs file:
+MWoffliner provides an API and can be used as a Node.js library. Here's a stub example for your `index.mjs` file:
+
 ```javascript
-import * as mwoffliner from 'mwoffliner';
+import * as mwoffliner from 'mwoffliner'
 
 const parameters = {
-    mwUrl: "https://es.wikipedia.org",
-    adminEmail: "foo@bar.net",
-    verbose: true,
-    format: "nopic",
-    articleList: "./articleList"
-};
-mwoffliner.execute(parameters); // returns a Promise
+  mwUrl: 'https://es.wikipedia.org',
+  adminEmail: 'foo@bar.net',
+  verbose: true,
+  format: 'nopic',
+  pageList: './pageList',
+}
+
+mwoffliner.execute(parameters) // returns a Promise
+```
+
+## MathJax support
+
+> [!WARNING]
+> MathJax support is UNSTABLE: the `--mathJax*` CLI parameters described below may change, even in a minor release. Setting them up also requires wiki-specific technical preparation, so this is aimed at developers rather than end users.
+
+MWoffliner can bundle [MathJax](https://www.mathjax.org) into the ZIM so that math formulas keep rendering offline, on wikis that rely on it (e.g. via the `SimpleMathJax` extension).
+
+MathJax 2, 3 and 4 are very different beasts: each ships its own set of files and, more importantly, each requires its own incompatible configuration format (which defines things like math delimiters and custom macros). Because of this, MWoffliner cannot auto-detect and configure MathJax for you — you have to supply a matching MathJax build and configuration yourself, via three CLI parameters:
+
+- `--mathJaxSource`: local path or HTTP(S) URL to a ZIP archive of a compiled MathJax distribution. Its content is extracted and pushed to the ZIM (under an internal `_mathjax_/` namespace).
+- `--mathJaxConfig`: local path or HTTP(S) URL to an HTML file containing a single `<script>` tag with the MathJax configuration (this must be copied from the wiki, see below). Its content is injected inline, before the MathJax library, on every page that needs it. If the configuration needs to reference a path inside the MathJax archive (e.g. `MathJax.Ajax.config.path[...]`), it cannot use an absolute path since ZIMs have no fixed root URL. Write `__MATHJAX_ROOT__` instead of the leading slash and mwoffliner will replace it, on every page, with the correct relative path to the root of the MathJax archive you provided. For instance use `MathJax.Ajax.config.path["Contrib"] = "__MATHJAX_ROOT__/MathJaxExtensions/legacy";` if the resources are located in a `MathJaxExtensions/legacy` folder in the archive (inside the root folder of the archive if there is a single folder as usual).
+- `--mathJaxEntryPoint`: path, relative to the root of the extracted archive, to the MathJax script to load (e.g. `es5/tex-chtml.js` for MathJax 3). Defaults to `MathJax.js` (MathJax 2 entry point).
+- `--mathJaxAllPages`: inject the config/entry-point `<script>` tags on every page instead of only on pages detected to need MathJax (see below). Some wikis don't list MathJax in their page JS modules, which defeats the automatic detection; this flag works around that at the cost of adding the scripts to every page.
+
+`--mathJaxConfig`, `--mathJaxEntryPoint` and `--mathJaxAllPages` all require `--mathJaxSource` to also be set. A page is considered to "need" MathJax, and only then gets the config/entry-point `<script>` tags injected, when one of the JS modules MediaWiki reports for that page matches `mathjax` (case-insensitive), unless `--mathJaxAllPages` is set, in which case every page gets them. The extracted library files themselves are always pushed to the ZIM as soon as `--mathJaxSource` is set, regardless of which pages use them.
+
+### Preparing the parameters for a given wiki
+
+1. **Find the live MathJax version and configuration.** Open a wiki page that renders math formulas, open your browser's developer console and type `MathJax.version` to get the exact version. Then find the configuration, typically a `<script>` block setting `window.MathJax = {...}` (MathJax 3/4) or `MathJax.Hub.Config({...})` (MathJax 2) in the page source — copy it as-is into a local file, e.g. `mathjax-config.html`.
+2. **Find the exact entry point.** In your browser's network tab, find the request loading the MathJax library itself (typically named `MathJax.js` for MathJax 2, or a `tex-chtml.js`/`tex-svg.js`/... for MathJax 3/4) and note its full path, including any query string.
+3. **Build a matching MathJax ZIP**, the exact steps depend on the major version in use (see below).
+4. **Run the scraper** with the three parameters, e.g.:
+   ```sh
+   mwoffliner --mwUrl=https://your.wiki --adminEmail=foo@bar.net \
+     --mathJaxSource=./mathjax-source.zip \
+     --mathJaxConfig=./mathjax-config.html \
+     --mathJaxEntryPoint=es5/tex-chtml.js
+   ```
+
+### Building the ZIP — MathJax 2
+
+MathJax 2 is not published to npm as an installable package; it is only distributed as source on GitHub. Download a release archive directly from the [MathJax releases page](https://github.com/mathjax/MathJax/releases) (e.g. `2.7.9`) and use it as-is as `--mathJaxSource`, no re-zipping needed:
+
+```sh
+curl -Lo mathjax-source.zip https://github.com/mathjax/MathJax/archive/refs/tags/2.7.9.zip
 ```
 
+The archive has a single top-level folder (e.g. `MathJax-2.7.9/`); mwoffliner strips it automatically when extracting.
+
+Unlike MathJax 3/4, MathJax 2 loads its extensions/output-processor via a `config=` query parameter on the `MathJax.js` request itself (e.g. `MathJax.js?config=TeX-MML-AM_CHTML`) rather than solely through the injected configuration script — this is the URL you captured in step 2 above. Set `--mathJaxEntryPoint` to that same path and query string, e.g.:
+
+```sh
+--mathJaxEntryPoint="MathJax.js?config=TeX-MML-AM_CHTML"
+```
+
+The referenced combined-configuration file (here `config/TeX-MML-AM_CHTML.js`) is part of the standard MathJax 2 distribution, so it is already included in the ZIP from the release archive.
+
+### Building the ZIP — MathJax 3
+
+MathJax 3 is published to npm as `mathjax-full`, which bundles both the compiled runtime (under `es5/`) and the TypeScript sources (which mwoffliner automatically ignores when an `es5/` folder is present):
+
+```sh
+npm install mathjax-full@3.2.2
+cd node_modules/mathjax-full && zip -r ../../mathjax-source.zip . && cd ../..
+```
+
+Entry point example: `--mathJaxEntryPoint=es5/tex-chtml.js`.
+
+### Building the ZIP — MathJax 4
+
+MathJax 4 moved to scoped npm packages. Use the plain `mathjax` package (deployment-ready bundle), **not** `@mathjax/src` (the TypeScript source package meant for building MathJax itself, which requires separately installing font packages):
+
+```sh
+npm install mathjax@4
+cd node_modules/mathjax && zip -r ../../mathjax-source.zip . && cd ../..
+```
+
+Unlike MathJax 3, entry-point files live at the root of the package (no `es5/` folder), e.g. `--mathJaxEntryPoint=tex-chtml.js`.
+
 ## Background
 
 Complementary information about MWoffliner:
 
-* MediaWiki software is used by thousands of wikis, the most
-  famous ones being the Wikimedia ones, including [Wikipedia](https://wikipedia.org).
-* MediaWiki is a PHP wiki runtime engine.
-* Wikitext is the name of the markup language that MediaWiki uses.
-* MediaWiki includes a parser for WikiText into HTML, and this
-  parser creates the HTML pages displayed in your browser.
-* Have a look at the scraper [functional architecture](docs/functional_architecture.md)
+- **MediaWiki software** is used by thousands of wikis, the most famous ones being the Wikimedia ones, including [Wikipedia](https://wikipedia.org).
+- **MediaWiki** is a PHP wiki runtime engine.
+- **Wikitext** is the markup language that MediaWiki uses.
+- **MediaWiki parser** converts Wikitext to HTML, which displays in your browser.
+- Read the [scraper functional architecture](docs/functional_architecture.md) for more details.
+
+## License
+
+[GPLv3](https://www.gnu.org/licenses/gpl-3.0) or later, see [LICENSE](LICENSE) for more details.
+
+## Acknowledgements
 
-License
--------
+This project received funding through [NGI Zero Core](https://nlnet.nl/core), a fund established by [NLnet](https://nlnet.nl/) with financial support from the European Commission's [Next Generation Internet](https://ngi.eu/) program. Learn more at the [NLnet project page](https://nlnet.nl/project/MWOffliner).
 
-[GPLv3](https://www.gnu.org/licenses/gpl-3.0) or later, see
-[LICENSE](LICENSE) for more details.
+[<img width="20%" alt="NLnet foundation logo" src="https://github.com/user-attachments/assets/22233242-ec49-4540-a0af-b70725cedbee" />](https://nlnet.nl/)
+[<img width="20%" alt="NGI Zero Logo" src="https://github.com/user-attachments/assets/1bbbda57-dc6f-4902-ae29-236e5e89228f" />](https://nlnet.nl/core)

package/docs/functional_architecture.md

@@ -5,11 +5,11 @@ This document describes a high-level overview of how mwoffliner scraper works.
 At a high level, mwoffliner is divided into following sequence of actions.
 
 - retrieve Mediawiki info
-- retrieve list of articles to include and their metadata
-- for every article:
-  - retrieve its parsed HTML (Wikitext transformed into HTML) and JS/CSS dependencie
+- retrieve list of pages to include and their metadata
+- for every page:
+  - retrieve its parsed HTML (Wikitext transformed into HTML) and JS/CSS dependencies
   - adapt / render it for proper operation within the ZIM file (includes detection of media dependencies)
-  - save rendered article HTML into the ZIM
+  - save rendered page HTML into the ZIM
 - for every file dependency (JS/CSS/media)
   - if its an image, download it either from S3 cache (images only) or from online and recompress when possible
   - otherwise download it from online
@@ -17,7 +17,17 @@ At a high level, mwoffliner is divided into following sequence of actions.
 
 The scraper supports flavours, which are variants of the ZIM (e.g. without images, with images but without videos, ...).
 
-For now, retrieval of articles and files dependencies is repeated for every flavour requested (even if content probably didn't changed).
+For now, retrieval of pages and files dependencies is repeated for every flavour requested (even if content probably didn't changed).
+
+## Lexicography
+
+> [!NOTE]
+> All these concepts have been clarified in mwoffliner 2.0.0.
+
+- Page: Base object of Mediawikis. Could be an article. Since this scraper is capable to process non-main namespaces, it processes pages, not only articles (see [The_difference_between_articles_and_page](https://en.wikipedia.org/wiki/Wikipedia:The_difference_between_articles_and_pages)).
+- Page Title : as in [Mediawiki](https://www.mediawiki.org/wiki/Manual:Page_title), title of the page with spaces (not underscores) but namespace. E.g. 'Escherichia coli O157:H7' or 'Category:Escherichia coli'.
+- Page Display Title : as in [Mediawiki](https://www.mediawiki.org/wiki/Display_title), preferred title for display. May contain HTML code.
+- Page ZIM Title : Title of the page in the ZIM
 
 ## Retrieve Mediawiki info
 
@@ -28,68 +38,54 @@ Currently, scrape uses this call to retrieve:
 - from `siteinfo`: `general` info (language, title, mainPage, site name, logo, text direction, ...), `skins` (to detect default skin), `rightsinfo` (to extract the license), `namespaces` and `namespacealiases` to build the list of namespaces
 - from `allmessages`: the `tagline` (subtitle)
 
-## Retrive list of articles to include and their metadata
-
-First, the scraper needs a list of article IDs and their details (redirects, ...).
+## Retrieve list of pages to include and their metadata
 
-"Article ID" refers to the page title with underscores instead of spaces, not the numeric page ID.
+First, the scraper needs a list of page titles and their details (redirects, ...).
 
-If user specified the exact list of articles to retrieve, then scraper simply request details about every articles in the list, in batches of about 50 articles (with `action=query&titles=titleX|titleY|...` ; batch sizes may vary due to constraints on query parameters size).
+"Page title" refers to the page title with spaces (not underscores), even if it is possible to use underscores in many places since Mediawiki APIs + scraper are permissive when possible. It is not the numeric page ID.
 
-Otherwise, the scraper enumerates articles in given namespaces (by default, all content namespaces) with the `allpages` generator, requesting one namespace content at a time (`action=query&generator=allpages&gapnamespace=xx`).
+If user specified the exact list of pages to retrieve, then scraper simply request details about every pages in the list, in batches of about 50 pages (with `action=query&titles=titleX|titleY|...` ; batch sizes may vary due to constraints on query parameters size).
 
-The details we retrieve at this stage about every articles are their title, subtitle, revisions, redirects, thumbnail, categories, coordinates (when it applies), text language, text direction and contentmodel (to consider only on wikitext ones).
+Otherwise, the scraper enumerates pages in given namespaces (by default, all content namespaces) with the `allpages` generator, requesting one namespace content at a time (`action=query&generator=allpages&gapnamespace=xx`).
 
-## Retrieving article HTML and rendering
+The details we retrieve at this stage about every pages are their title, subtitle, revisions, redirects, thumbnail, categories, coordinates (when it applies), text language, text direction and contentmodel (to consider only on wikitext ones).
 
-In order to retrieve article HTML and render it, multiple solutions have been identified.
+## Retrieving page HTML and rendering
 
-As of today, 5 renderers (way to download article + render it to ZIM compatible HTML) are implemented:
+In order to retrieve page HTML and render it, multiple solutions have been identified.
 
-- WikimediaDesktop
-- WikimediaMobile
-- RestApi
-- VisualEditor
-- ActionParse
+As of today, 1 renderer (way to download page + render it to ZIM compatible HTML) is left implemented: ActionParse. Other renderers have been dropped in mwoffliner 2.0 because they were not used anymore, not implementing skin support and not providing added value compared to ActionParse render.
 
-WikimediaDesktop and WikimediaMobile are only available on Wikimedia Mediawikis.
+ActionParse API is available in Mediawiki since 1.16.0 (2010).
 
-Availability of RestApi and VisualEditor is subject to Mediawiki admin decision to support it or not. RestApi is available by default but might be blocked by admin. VisualEditor is an extension which might be installed or not.
+ActionParse renderer implements a thorough skin support (see below about skin) and needs only one HTTP query per page.
 
-ActionParse is available since 1.16.0 (2010) and is anyway a requirement for other APIs.
-
-Only ActionParse (most recent renderer at mwoffliner level) implements a thorough skin support (see below about skin).
-
-All renderers but ActionParse needs two HTTP queries: one to retrieve the article HTML and one to retrieve its metadata (to a 'simplified' ActionParse URL in fact).
-
-The article metadata we retrieve at this stage is the article are:
+The page metadata we retrieve at this stage are:
 
 - `displaytitle` and `subtitle`
 - `headhtml`: used to extract proper CSS classes we have to set on `<html>` and `<body>` tags + some JS configuration variables which are part of the HTML head
 - `jsconfigvars`: other JS configuration variables which comes from other parts of the Mediawiki codebase
-- `modules`: list of JS and CSS modules to apply on current article
-
-Renderer is automatically selected based on its availability and mwoffliner own preference. ActionParse is the preferred renderer since 1.15.0 due to its general availability and support of skins.
+- `modules`: list of JS and CSS modules to apply on current page
 
 ### ActionParse parser
 
-When using ActionParse renderer, we pass `usearticle=1`, which means that we ask the Mediawiki to use the parser configured for this article. This allows the scraper to retrieve article `text` (inner of the HTML containing the article itself) that is as close as possible to what is used online, either Parsoid or legacy parser (see https://www.mediawiki.org/wiki/Parsoid). This is mandatory because if we use a parser different than the one used online, we will de-facto get bugs which are "normal", either because parser still has a bug, or an extension is not compatible, or because Wikitext contains some workaround understandable only by a given parser. This should be avoided at all price if we want to have a versatile scraper capable of processing any Mediawiki. Even focusing only on Wikimedia wikis, not all of them have already transitioned to Parsoid for instance.
+When using ActionParse renderer, we pass `usearticle=1`, which means that we ask the Mediawiki to use the parser configured for this page. This allows the scraper to retrieve page `text` (inner of the HTML containing the page itself) that is as close as possible to what is used online, either Parsoid or legacy parser (see https://www.mediawiki.org/wiki/Parsoid). This is mandatory because if we use a parser different than the one used online, we will de-facto get bugs which are "normal", either because parser still has a bug, or an extension is not compatible, or because Wikitext contains some workaround understandable only by a given parser. This should be avoided at all price if we want to have a versatile scraper capable of processing any Mediawiki. Even focusing only on Wikimedia wikis, not all of them have already transitioned to Parsoid for instance.
 
 ### Skins
 
-In Mediawikis, rendering of Wikitext into HTML works around a concept of skin. A skin is a mix of HTML template and CSS+JS dependencies. It defines both the visual appareance of the rendered Wikitext but also everything "around it".
+In Mediawikis, rendering of Wikitext into HTML works around a concept of skin. A skin is a mix of HTML template and CSS+JS dependencies. It defines both the visual appearance of the rendered Wikitext but also everything "around it".
 
-Since most wikis have adapted their content to their skin (and vice versa), it is mostly mandatory to use the skin inside the ZIM, both for proper rendering and for a visual appareance similar to online website (users don't mind about technical details, they want the wiki to be the same inside the ZIM than online).
+Since most wikis have adapted their content to their skin (and vice versa), it is mostly mandatory to use the skin inside the ZIM, both for proper rendering and for a visual appearance similar to online website (users don't mind about technical details, they want the wiki to be the same inside the ZIM than online).
 
 Skin detection is automated in mwoffliner for now (see https://github.com/openzim/mwoffliner/issues/2213).
 
-For now, only `vector` (legacy) and `vector-2022` are supported, and only with ActionParse renderer. Only `vector-2022` is the truely responsive skin, providing ultimate rendering on mostly all screen sizes.
+For now, only `vector` (legacy) and `vector-2022` are supported, and only with ActionParse renderer. Only `vector-2022` is the truly responsive skin, providing ultimate rendering on mostly all screen sizes.
 
-With ActionParse renderer, other skins have a `fallback` skin implemented in the scraper. This means that the scraper will render the article HTML inside an HTML structure which looks like an expected structure (which comes from `vector` legacy). The consequence is that if the skins uses only article HTML structure to attach CSS rules or JS code, then everything will render fine. If the skins uses HTML structures coming from the "surrounding HTML" (headers, ...) then things will not apply 100% correctly.
+With ActionParse renderer, other skins have a `fallback` skin implemented in the scraper. This means that the scraper will render the page HTML inside an HTML structure which looks like an expected structure (which comes from `vector` legacy). The consequence is that if the skins uses only page HTML structure to attach CSS rules or JS code, then everything will render fine. If the skins uses HTML structures coming from the "surrounding HTML" (headers, ...) then things will not apply 100% correctly.
 
 ### JS / CSS dependencies
 
-ActionParse API is returning the list of JS and CSS dependencies for a given article, by inspecting what Wikitext is using.
+ActionParse API is returning the list of JS and CSS dependencies for a given page, by inspecting what Wikitext is using.
 
 The special `startup` JS module is missing from results because always used anyway.
 

package/extensions/wiktionary_fr.js

@@ -1,21 +1,22 @@
-module.exports = class WiktionaryFR { // implements CustomProcessor
-    async shouldKeepArticle(articleId, doc) {
-        const frenchTitle = doc.querySelector(`#fr.sectionlangue`);
-        return !!frenchTitle;
+export default class WiktionaryFR {
+  // implements CustomProcessor
+  async shouldKeepPage(pageTitle, doc) {
+    const frenchTitle = doc.querySelector(`#fr.sectionlangue`)
+    return !!frenchTitle
+  }
+  async preProcessPage(pageTitle, doc) {
+    const nonFrenchTitles = Array.from(doc.querySelectorAll(`.sectionlangue:not(#fr)`))
+    for (const title of nonFrenchTitles) {
+      title.closest('details').remove()
     }
-    async preProcessArticle(articleId, doc) {
-        const nonFrenchTitles = Array.from(doc.querySelectorAll(`.sectionlangue:not(#fr)`));
-        for (const title of nonFrenchTitles) {
-            title.closest('details').remove();
-        }
 
-        const h4titles = Array.from(doc.querySelectorAll(`h4`));
-        for (const h4title of h4titles) {
-            h4title.closest('details').remove();
-        }
-        //Remove h2 summary title
-        doc.querySelector('h2').closest('summary').setAttribute('style', 'display:none! important')
-
-        return doc;
+    const h4titles = Array.from(doc.querySelectorAll(`h4`))
+    for (const h4title of h4titles) {
+      h4title.closest('details').remove()
     }
+    //Remove h2 summary title
+    doc.querySelector('h2').closest('summary').setAttribute('style', 'display:none! important')
+
+    return doc
+  }
 }