mwoffliner 1.14.2 → 1.15.0

package/README.md

@@ -41,7 +41,7 @@ Run `mwoffliner --help` to get all the possible options.
 
 - *NIX Operating System (GNU/Linux, macOS, ...)
 - [Redis](https://redis.io/)
-- [NodeJS](https://nodejs.org/en/) version 16 or greater
+- [NodeJS](https://nodejs.org/en/) version 22 (we support only one single Node.JS version, other versions might work or not)
 - [Libzim](https://github.com/openzim/libzim) (On GNU/Linux & macOS we automatically download it)
 - Various build tools which are probably already installed on your
   machine (packages `libjpeg-dev`, `libglu1`, `autoconf`, `automake`, `gcc` on
@@ -51,35 +51,38 @@ Run `mwoffliner --help` to get all the possible options.
 
 ## Usage
 
-To install MWoffliner globally:
+To install latest released MWoffliner version from NPM repo (use `-g` to install globally, not only in current folder):
 ```bash
 npm i -g mwoffliner
 ```
 
-You might need to run this command with the `sudo` command, depending
-how your `npm` is configured.
+> [!WARNING] 
+> Note that you might need to run this command with the `sudo` command, depending
+how your `npm` / OS is configured. `npm` permission checking can be a bit annoying for a
+newcomer. Please read the documentation carefully if you hit problems: https://docs.npmjs.com/cli/v7/using-npm/scripts#user
 
-`npm` permission checking can be a bit annoying for a
-newcomer. Please read the documentation carefully if you hit
-problems: https://docs.npmjs.com/cli/v7/using-npm/scripts#user
-
-Then to run it:
+Then you can run the scraper:
 ```bash
 mwoffliner --help
 ```
 
-To install and run it locally:
-```bash
-npm i
-npm run mwoffliner -- --help
-```
-
 To use MWoffliner with a S3 cache, you should provide a S3 URL like
 this:
 ```bash
 --optimisationCacheUrl="https://wasabisys.com/?bucketName=my-bucket&keyId=my-key-id&secretAccessKey=my-sac"
 ```
 
+## Contribute
+
+If you've retrieved mwoffliner source code (e.g. with a git clone of our repo), you can then install and run it locally (including with your local modifications):
+
+```bash
+npm i
+npm run mwoffliner -- --help
+```
+
+Detailed [contribution documentation and guidelines](CONTRIBUTING.md) are available.
+
 ## API
 
 MWoffliner provides also an API and therefore can be used as a NodeJS
@@ -107,33 +110,7 @@ Complementary information about MWoffliner:
 * Wikitext is the name of the markup language that MediaWiki uses.
 * MediaWiki includes a parser for WikiText into HTML, and this
   parser creates the HTML pages displayed in your browser.
-
-### GNU/Linux - Debian based distributions
-
-Install NodeJS:
-Read https://nodejs.org/en/download/current/
-
-Install Redis:
-```bash
-sudo apt-get install redis-server
-```
-
-## Troubleshooting
-
-Older GNU/Linux distributions and/or versions of Node.js might be
-shipped with a deprecated version of `npm`. Older versions of `npm`
-have incompatbilities with certain versions of Node.js and might
-simply fail to install `mwoffliner` package.
-
-We recommend to use a recent version of `npm`. Recent versions can
-perfectly deal with older Node.js 10. Do install the packaged
-version of `npm` and then use it to install a newer version like:
-
-```bash
-sudo npm install --unsafe-perm -g npm
-```
-
-Don't forget to remove the packaged version of `npm` afterward.
+* Have a look at the scraper [functional architecture](docs/functional_architecture.md)
 
 License
 -------

package/docs/functional_architecture.md

@@ -0,0 +1,62 @@
+# Functional Architecture
+
+This document describes a high-level overview of how mwoffliner scraper works.
+
+At a high level, mwoffliner is divided into following sequence of actions.
+
+- retrieve list of articles to include and their metadata
+- for every article:
+  - retrieve its parsed HTML (Wikitext transformed into HTML) and JS/CSS dependencie
+  - adapt / render it for proper operation within the ZIM file (includes detection of media dependencies)
+  - save rendered article HTML into the ZIM
+- for every file dependency (JS/CSS/media)
+  - if its an image, download it either from S3 cache (images only) or from online and recompress when possible
+  - otherwise download it from online
+  - save them into the ZIM file
+
+The scraper supports flavours, which are variants of the ZIM (e.g. without images, with images but without videos, ...).
+
+The process above is repeated for every flavour requested.
+
+## Retrieving article HTML and rendering
+
+In order to retrieve article HTML and render it, multiple solutions have been identified.
+
+As of today, 5 renderers (way to download article + render it to ZIM compatible HTML) are implemented:
+
+- WikimediaDesktop
+- WikimediaMobile
+- RestApi
+- VisualEditor
+- ActionParse
+
+WikimediaDesktop and WikimediaMobile are only available on Wikimedia Mediawikis.
+
+Availability of RestApi and VisualEditor is subject to Mediawiki admin decision to support it or not. RestApi is available by default but might be blocked by admin. VisualEditor is an extension which might be installed or not.
+
+ActionParse is available since 1.16.0 (2010) and is anyway a requirement for other APIs.
+
+Only ActionParse (most recent renderer) implements a thorough skin support (see below about skin).
+
+All renderers but ActionParse needs two HTTP queries: one to retrieve the article HTML and one to retrieve its metadata (to a 'simplified' ActionParse URL in fact).
+
+Renderer is automatically selected based on its availability and mwoffliner own preference. ActionParse is the preferred renderer since 1.15.0 due to its general availability and support of skins.
+
+### Skins
+
+In Mediwikis, rendering of Wikitext into HTML works around a concept of skin. A skin is a mix of HTML template and CSS+JS dependencies. It defines both the visual appareance of the rendered Wikitext but also everything "around it".
+
+Since most wikis have adapted their content to their skin (and vice versa), it is mostly mandatory to use the skin inside the ZIM, both for proper rendering and for a visual appareance similar to online website (users don't mind about technical details, they want the wiki to be the same inside the ZIM than online).
+
+Skin detection is automated in mwoffliner for now (see https://github.com/openzim/mwoffliner/issues/2213).
+
+For now, only `vector` (legacy) and `vector-2022` are supported, and only with ActionParse renderer. Only `vector-2022` is the truely responsive skin, providing ultimate rendering on mostly all screen sizes.
+
+
+### JS / CSS dependencies
+
+ActionParse API is returning the list of JS and CSS dependencies for a given article, by inspecting what Wikitext is using.
+
+The special `startup` JS module is missing from results because always used anyway.
+
+There are still some known hiccups around this (see https://github.com/openzim/mwoffliner/issues/2212 and https://github.com/openzim/mwoffliner/issues/2215 for instance), and probably more to come, this is a complex area of Mediawiki.

package/eslint.config.js

@@ -0,0 +1,24 @@
+import js from '@eslint/js'
+import eslint from '@eslint/js'
+import tseslint from 'typescript-eslint'
+import tsparser from '@typescript-eslint/parser'
+import { globalIgnores } from 'eslint/config'
+
+export default [
+  js.configs.recommended,
+  globalIgnores(['lib/**', 'res/**', '**/*.js', '**/*.cjs']),
+  eslint.configs.recommended,
+  ...tseslint.configs.recommended,
+  {
+    languageOptions: {
+      parser: tsparser,
+      parserOptions: {
+        project: 'tsconfig.json',
+        sourceType: 'module',
+      },
+    },
+    rules: {
+      '@typescript-eslint/no-explicit-any': 'off',
+    },
+  },
+]

package/lib/DOMUtils.js

@@ -1,5 +1,8 @@
 const DOMUtils = {
     deleteNode(node) {
+        if (!node) {
+            return;
+        }
         if (node.parentNode) {
             node.parentNode.removeChild(node);
         }

package/lib/DOMUtils.js.map

@@ -1 +1 @@
-{"version":3,"file":"DOMUtils.js","sourceRoot":"","sources":["../src/DOMUtils.ts"],"names":[],"mappings":"AAAA,MAAM,QAAQ,GAAG;IACf,UAAU,CAAC,IAAmB;QAC5B,IAAI,IAAI,CAAC,UAAU,EAAE;YACnB,IAAI,CAAC,UAAU,CAAC,WAAW,CAAC,IAAI,CAAC,CAAA;SAClC;aAAM;YACL,IAAI,CAAC,SAAS,GAAG,EAAE,CAAA;SACpB;QACD,IAAI,GAAG,SAAS,CAAA;IAClB,CAAC;IAED,YAAY,CAAC,IAAmB,EAAE,IAAY,EAAE,GAAQ;QACtD,MAAM,MAAM,GAAG,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,CAAA;QACtC,MAAM,QAAQ,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,MAAM,IAAI,GAAG,EAAE,CAAC,CAAC,CAAC,GAAG,CAAA;QAClD,IAAI,CAAC,YAAY,CAAC,IAAI,EAAE,QAAe,CAAC,CAAA;IAC1C,CAAC;IAED,kBAAkB,CAAC,IAAmB;QACpC,IAAI,OAAO,GAAG,IAAI,CAAC,WAAW,CAAA;QAC9B,OAAO,OAAO,IAAI,OAAO,CAAC,QAAQ,KAAK,CAAC,CAAC,kBAAkB,EAAE;YAC3D,OAAO,GAAG,OAAO,CAAC,WAAW,CAAA;SAC9B;QACD,OAAO,OAAO,CAAA;IAChB,CAAC;CACF,CAAA;AAED,eAAe,QAAQ,CAAA"}
+{"version":3,"file":"DOMUtils.js","sourceRoot":"","sources":["../src/DOMUtils.ts"],"names":[],"mappings":"AAAA,MAAM,QAAQ,GAAG;IACf,UAAU,CAAC,IAAmB;QAC5B,IAAI,CAAC,IAAI,EAAE,CAAC;YACV,OAAM;QACR,CAAC;QACD,IAAI,IAAI,CAAC,UAAU,EAAE,CAAC;YACpB,IAAI,CAAC,UAAU,CAAC,WAAW,CAAC,IAAI,CAAC,CAAA;QACnC,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,SAAS,GAAG,EAAE,CAAA;QACrB,CAAC;QACD,IAAI,GAAG,SAAS,CAAA;IAClB,CAAC;IAED,YAAY,CAAC,IAAmB,EAAE,IAAY,EAAE,GAAQ;QACtD,MAAM,MAAM,GAAG,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,CAAA;QACtC,MAAM,QAAQ,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,MAAM,IAAI,GAAG,EAAE,CAAC,CAAC,CAAC,GAAG,CAAA;QAClD,IAAI,CAAC,YAAY,CAAC,IAAI,EAAE,QAAe,CAAC,CAAA;IAC1C,CAAC;IAED,kBAAkB,CAAC,IAAmB;QACpC,IAAI,OAAO,GAAG,IAAI,CAAC,WAAW,CAAA;QAC9B,OAAO,OAAO,IAAI,OAAO,CAAC,QAAQ,KAAK,CAAC,CAAC,kBAAkB,EAAE,CAAC;YAC5D,OAAO,GAAG,OAAO,CAAC,WAAW,CAAA;QAC/B,CAAC;QACD,OAAO,OAAO,CAAA;IAChB,CAAC;CACF,CAAA;AAED,eAAe,QAAQ,CAAA"}

package/lib/Downloader.d.ts

@@ -1,8 +1,8 @@
-/// <reference types="node" resolution-mode="require"/>
 import type { BackoffStrategy } from 'backoff';
 import { AxiosRequestConfig, AxiosResponse } from 'axios';
 import S3 from './S3.js';
 import { Dump } from './Dump.js';
+import { Renderer } from './renderers/abstract.renderer.js';
 interface DownloaderOpts {
     uaString: string;
     speed: number;
@@ -20,34 +20,58 @@ interface BackoffOptions {
     retryIf: (error?: any) => boolean;
     backoffHandler: (number: number, delay: number, error?: any) => void;
 }
+export declare class DownloadError extends Error {
+    urlCalled: string | null;
+    httpReturnCode: number | null;
+    responseContentType: string | null;
+    responseData: any;
+    constructor(message: string, urlCalled: string | null, httpReturnCode: number | null, responseContentType: string | null, responseData: any);
+}
+export interface DownloadErrorContext {
+    urlCalled: string;
+    errorCode: string | null;
+    httpReturnCode: number | null;
+    responseContentType: string | null;
+    responseData: any;
+}
 /**
  * Downloader is a class providing content retrieval functionalities for both Mediawiki and S3 remote instances.
  */
 declare class Downloader {
+    private static instance;
+    static getInstance(): Downloader;
     loginCookie: string;
-    readonly speed: number;
+    private _speed;
     cssDependenceUrls: KVS<boolean>;
-    readonly webp: boolean;
-    readonly requestTimeout: number;
-    readonly basicRequestOptions: AxiosRequestConfig;
-    readonly arrayBufferRequestOptions: AxiosRequestConfig;
-    readonly jsonRequestOptions: AxiosRequestConfig;
-    readonly streamRequestOptions: AxiosRequestConfig;
+    private _webp;
+    private _requestTimeout;
+    private _basicRequestOptions;
+    private _arrayBufferRequestOptions;
+    private _jsonRequestOptions;
+    private _streamRequestOptions;
     wikimediaMobileJsDependenciesList: string[];
     wikimediaMobileStyleDependenciesList: string[];
-    private readonly uaString;
+    private uaString;
     private activeRequests;
     private maxActiveRequests;
-    private readonly backoffOptions;
-    private readonly optimisationCacheUrl;
+    private backoffOptions;
+    private optimisationCacheUrl;
     private s3;
     private apiUrlDirector;
     private articleUrlDirector;
     private mainPageUrlDirector;
-    private readonly insecure;
-    constructor({ uaString, speed, reqTimeout, optimisationCacheUrl, s3, webp, backoffOptions, insecure }: DownloaderOpts);
+    private insecure;
+    get speed(): number;
+    get webp(): boolean;
+    get requestTimeout(): number;
+    get basicRequestOptions(): AxiosRequestConfig<any>;
+    get arrayBufferRequestOptions(): AxiosRequestConfig<any>;
+    get jsonRequestOptions(): AxiosRequestConfig<any>;
+    get streamRequestOptions(): AxiosRequestConfig<any>;
+    set init({ uaString, speed, reqTimeout, optimisationCacheUrl, s3, webp, backoffOptions, insecure }: DownloaderOpts);
+    private reset;
     private getUrlDirector;
-    setUrlsDirectors(mainPageRenderer: any, articlesRenderer: any): void;
+    setUrlsDirectors(mainPageRenderer: Renderer, articlesRenderer: Renderer): void;
     getArticleUrl(articleId: string): string;
     getMainPageUrl(articleId: string): string;
     removeEtagWeakPrefix(etag: string): string;
@@ -57,7 +81,8 @@ declare class Downloader {
         gapContinue: string;
         articleDetails: QueryMwRet;
     }>;
-    getArticle(webp: boolean, _moduleDependencies: any, articleId: string, articleDetailXId: RKVS<ArticleDetail>, articleRenderer: any, articleUrl: any, dump: Dump, articleDetail?: ArticleDetail, isMainPage?: boolean): Promise<any>;
+    getLogEvents(letype: string, articleId: string): Promise<any>;
+    getArticle(articleId: string, articleDetailXId: RKVS<ArticleDetail>, articleRenderer: Renderer, articleUrl: any, dump: Dump, articleDetail?: ArticleDetail, isMainPage?: boolean): Promise<any>;
     getJSON<T>(_url: string): Promise<T>;
     request<T = any, R extends AxiosResponse<T> = AxiosResponse<T>, D = any>(config: AxiosRequestConfig<D>): Promise<R>;
     downloadContent(_url: string, kind: DonwloadKind, retry?: boolean): Promise<{
@@ -82,8 +107,11 @@ declare class Downloader {
     getModuleDependencies(title: string): Promise<{
         jsConfigVars: string;
         jsDependenciesList: string[];
-        styleDependenciesList: string[];
+        styleDependenciesList: any[];
     }>;
     private streamToBuffer;
+    static extractJsConfigVars(headhtml: string): string;
 }
-export default Downloader;
+export { Downloader as DownloaderClass };
+declare const _default: Downloader;
+export default _default;