mvw-search-index 2.3.7 → 3.0.1

package/README.md

@@ -8,9 +8,12 @@
 
 ## About
 
-**mvw-search-index** is a lightweight tool for generating a [lunr](http://lunrjs.com/) search index with a result store.
+**mvw-search-index** generates a [lunr](https://lunrjs.com/) search index plus a result store from the HTML
+files of a static website - built with Hugo, Jekyll, Gatsby, or by hand. The generated JSON file is loaded by
+the browser, which then searches entirely client-side.
 
-It is ideal for adding fast client-side search capabilities to static websites — whether built with Hugo, Jekyll, Gatsby, or even manually.
+> **Upgrading from 2.x?** Version 3 changes the API, the CLI and what ends up in the index.
+> See the [upgrade guide](UPGRADING.md).
 
 ---
 
@@ -18,9 +21,12 @@ It is ideal for adding fast client-side search capabilities to static websites
 
 - [Installation](#installation)
 - [Usage](#usage)
-  - [From CLI](#from-cli)
-  - [From TypeScript](#from-typescript)
-  - [From Node.js](#from-nodejs)
+  - [CLI](#cli)
+  - [Node.js / TypeScript](#nodejs--typescript)
+  - [Options](#options)
+- [Searching in the browser](#searching-in-the-browser)
+  - [Content in other languages](#content-in-other-languages)
+- [What gets indexed](#what-gets-indexed)
 - [Demo](#demo)
 - [Releases](#releases)
 
@@ -28,7 +34,7 @@ It is ideal for adding fast client-side search capabilities to static websites
 
 ## Installation
 
-Add the package to your project:
+Requires Node.js 22.12 or newer.
 
 ```bash
 npm install --save-dev mvw-search-index
@@ -38,64 +44,164 @@ npm install --save-dev mvw-search-index
 
 ## Usage
 
-You can create an index using the command-line interface, directly from TypeScript, or through Node.js.
+Run the indexer **after** your site has been built, on the generated HTML.
 
-### From CLI
+### CLI
 
 ```bash
-mvw-search-index <glob> <destination> [css-selector]
+mvw-search-index [options] <glob> <dest> [bodySelector]
 ```
 
-Example:
+Example - index everything in `public/`, but only the content of `<main>`, with German stemming and
+root-relative links:
 
 ```bash
-mvw-search-index ./build/**/*.html ./build/index.json
+mvw-search-index '**/*.html' public/suche/index.json main --cwd public --base-url / --language de
 ```
 
-Or from an npm script:
+Quote the glob so your shell doesn't expand it. From an npm script:
 
 ```json
 {
   "scripts": {
-    "index": "mvw-search-index './public/**/*.html' './static/suche/index.json' 'main'"
+    "index": "mvw-search-index '**/*.html' public/suche/index.json main --cwd public --base-url /"
   }
 }
 ```
 
-> Ensure the indexing step runs after the site is built.
+| Flag                         | Option            | Description                                                                 |
+|------------------------------|-------------------|-----------------------------------------------------------------------------|
+| `[bodySelector]`             | `bodySelector`    | CSS selector of the content to index (default `body`)                       |
+| `--cwd <dir>`                | `cwd`             | Directory the glob is resolved in; hrefs are relative to it                 |
+| `-e, --exclude <selector>`   | `excludeSelector` | Content to leave out (default `"nav, footer"`, `""` for none)               |
+| `-l, --language <code>`      | `language`        | Content language: stop words and stemmer (default `en`)                     |
+| `--base-url <url>`           | `baseUrl`         | Prefix for every href, e.g. `/`                                             |
+| `--strip-index-html`         | `stripIndexHtml`  | Link to `dir/` instead of `dir/index.html`                                  |
+| `--no-noindex`               | `respectNoindex`  | Also index pages marked `<meta name="robots" content="noindex">`            |
+| `--allow-empty`              | `allowEmpty`      | Write an empty index instead of failing when the glob matches nothing       |
+| `-b, --boost <field=number>` | `boosts`          | Field weight, repeatable, e.g. `-b title=10 -b body=1`                      |
+| `-v, --verbose`              | `logger`          | List every indexed file                                                     |
 
-### From TypeScript
+The CLI exits with code 1 and prints `Error: …` if indexing fails, including when the glob matches no files.
+
+`<dest>` must be inside the directory you run the command from (relative or absolute); anything outside, such as
+`../index.json`, is rejected so a wrong argument can't overwrite unrelated files. `--cwd` only affects where the
+HTML files are read from.
+
+### Node.js / TypeScript
 
 ```ts
-import { SearchIndex } from "mvw-search-index";
-import * as fs from "fs";
+import {writeFile} from "fs/promises";
+import {SearchIndex} from "mvw-search-index";
+
+const result = await SearchIndex.createFromGlob("**/*.html", {
+  cwd: "public",
+  bodySelector: "main",
+  language: "de",
+  baseUrl: "/",
+});
+await writeFile("public/suche/index.json", JSON.stringify(result));
+```
+
+CommonJS works the same way: `const {SearchIndex} = require("mvw-search-index");`.
+
+There are three entry points:
 
-const index = SearchIndex.createFromHtml(files, bodySelector);
-// Alternatives:
-// const index = SearchIndex.createFromInfo(info);
-// const index = SearchIndex.createFromGlob(glob, bodySelector, callback);
+| Method                                   | Input                                                            | Returns                       |
+|------------------------------------------|------------------------------------------------------------------|-------------------------------|
+| `createFromGlob(pattern, options?)`      | A glob pattern; files are read from disk                         | `Promise<ISearchIndexResult>` |
+| `createFromHtml(files, options?)`        | `HtmlFile[]` - `{relative: string, contents: string \| Buffer}` | `ISearchIndexResult`          |
+| `createFromInfo(files, options?)`        | `IFileInformation[]` - already extracted title/body/…            | `ISearchIndexResult`          |
 
-fs.writeFileSync("index.json", JSON.stringify(index));
+`ISearchIndexResult` is `{index: lunr.Index, store: {[href]: {title, description?}}}`. `JSON.stringify()` it to
+get the file the browser loads.
+
+### Options
+
+| Option            | Default                                          | Applies to         | Description                                                                                                                   |
+|-------------------|--------------------------------------------------|--------------------|-------------------------------------------------------------------------------------------------------------------------------|
+| `bodySelector`    | `"body"`                                         | Glob, HTML         | CSS selector of the element(s) whose text is indexed                                                                          |
+| `excludeSelector` | `"nav, footer"`                                  | Glob, HTML         | Elements inside the body to leave out; `""` for none                                                                          |
+| `respectNoindex`  | `true`                                           | Glob, HTML         | Skip pages with `<meta name="robots" content="noindex">`                                                                      |
+| `language`        | `"en"`                                           | all                | Two-letter language code; other than `en` uses [lunr-languages](https://github.com/MihaiValentin/lunr-languages) (see below) |
+| `boosts`          | `{title: 5, keywords: 3, description: 2, body: 1}` | all              | Relative weight of matches per field                                                                                          |
+| `cwd`             | `process.cwd()`                                  | Glob               | Directory the pattern is resolved in; hrefs are relative to it                                                                |
+| `allowEmpty`      | `false`                                          | Glob               | Resolve with an empty index instead of rejecting when nothing matches                                                         |
+| `baseUrl`         | `""`                                             | Glob, HTML         | Prefix for every href                                                                                                         |
+| `stripIndexHtml`  | `false`                                          | Glob, HTML         | `foo/index.html` → `foo/`                                                                                                     |
+| `logger`          | silent                                           | Glob, HTML         | Receives progress (`info`) and warnings (`warn`); `console` works                                                             |
+
+---
+
+## Searching in the browser
+
+Load lunr (2.3.x) and the generated file, then query it. Building the query with lunr's query API - rather than
+passing user input to `index.search()` - means no input can cause a `QueryParseError`, and every word is matched
+both as a whole (stemmed) and as a prefix, so results appear while typing:
+
+```html
+<script src="https://cdnjs.cloudflare.com/ajax/libs/lunr.js/2.3.9/lunr.min.js"></script>
+<script type="module">
+  const {index: serializedIndex, store} = await (await fetch("/suche/index.json")).json();
+  const index = lunr.Index.load(serializedIndex);
+
+  function search(input) {
+    const terms = lunr.tokenizer(input)
+      .map((token) => lunr.trimmer(token).toString())
+      .filter((term) => term.length > 0);
+    if (terms.length === 0) {
+      return [];
+    }
+    return index.query((query) => {
+      for (const term of terms) {
+        query.term(term, {boost: 10}); // whole word, stemmed like the index
+        query.term(term, {usePipeline: false, wildcard: lunr.Query.wildcard.TRAILING}); // prefix
+      }
+    }).map((result) => ({href: result.ref, ...store[result.ref]}));
+  }
+</script>
 ```
 
-### From Node.js
+Render the `title` and `description` with `textContent` (not `innerHTML`). [docs/index.html](docs/index.html) is a
+complete, working example.
 
-```js
-"use strict";
+### Content in other languages
 
-const { SearchIndex } = require("mvw-search-index");
-const fs = require("fs");
+With `language` set to anything other than `en`, the index references that language's lunr-languages pipeline
+functions. Load the matching scripts after lunr and **before** `lunr.Index.load()` - otherwise lunr throws
+`Cannot load unregistered function: trimmer-de`:
 
-SearchIndex.createFromGlob("./build/**/*.html", "main", (index) => {
-  fs.writeFileSync("./static/suche/index.json", JSON.stringify(index));
-});
+```html
+<script src="https://cdnjs.cloudflare.com/ajax/libs/lunr.js/2.3.9/lunr.min.js"></script>
+<script src="https://cdn.jsdelivr.net/npm/lunr-languages@1.22.0/lunr.stemmer.support.js"></script>
+<script src="https://cdn.jsdelivr.net/npm/lunr-languages@1.22.0/lunr.de.js"></script>
 ```
 
+Use `lunr.de.trimmer` instead of `lunr.trimmer` in the `search()` function above, so umlauts at word boundaries are
+kept.
+
+---
+
+## What gets indexed
+
+For every page:
+
+- **title**: `<title>`, falling back to `og:title`, then the first `<h1>`
+- **description**: `<meta name="description">`, falling back to `og:description`
+- **keywords**: `<meta name="keywords">` (comma separated)
+- **body**: the text of `bodySelector`, without `excludeSelector` matches, `<script>`, `<style>`, `<noscript>` and
+  `<template>`; words in separate block elements stay separate words
+
+Title and description also go into the result store, keyed by href. Text runs through lunr's pipeline for the
+configured language: punctuation is trimmed, stop words are dropped and words are stemmed.
+
+Pages with `<meta name="robots" content="noindex">` are skipped.
+
 ---
 
 ## Demo
 
-A basic sample site is included and served from [GitHub Pages](https://www.tiliavir.github.io/mvw-search-index).
+A basic sample site is included and served from [GitHub Pages](https://tiliavir.github.io/mvw-search-index/).
 
 Start it locally with:
 
@@ -103,13 +209,18 @@ Start it locally with:
 npm run serve
 ```
 
-This serves the content from [./docs](docs), featuring a simple static site with a search form on `index.html`.
+This rebuilds `docs/index.json` and serves [./docs](docs): a simple static site with a search form on `index.html`.
 
 ---
 
 ## Releases
 
-- **2.3.2 – 2.3.6**: Dependency updates.
+- **3.0.1**: Fixes all open SonarCloud issues - `node:` imports, `export … from` re-exports, `Object.hasOwn`,
+  `replaceAll`; CI installs with `--ignore-scripts` and pins third-party actions to commit SHAs; the demo page can be
+  zoomed and its search field has a label. No API or behaviour changes.
+- **3.0.0**: Major overhaul - Promise based API, correct text extraction and lunr pipeline (stemming, stop words),
+  language support, field boosts, href options, a full-featured CLI. **Breaking** - see [UPGRADING.md](UPGRADING.md).
+- **2.3.2 – 2.3.7**: Dependency updates.
 - **2.3.0**: Added attribute support for metadata extraction.
 - **2.2.10 - 2.2.16**: Dependency updates.
 - **2.2.9**: Removed `vinyl`; introduced demo application.
@@ -130,5 +241,3 @@ This project is licensed under the [MIT License](LICENSE).
 ## Author
 
 Maintained by [Tiliavir](https://github.com/Tiliavir).
-
----

package/js/cli.js

@@ -35,13 +35,76 @@ var __importStar = (this && this.__importStar) || (function () {
 })();
 Object.defineProperty(exports, "__esModule", { value: true });
 const commander_1 = require("commander");
-const fs = __importStar(require("fs"));
+const fs = __importStar(require("node:fs"));
+const path = __importStar(require("node:path"));
 const index_1 = require("./index");
+// read at runtime: package.json is outside of the compiled sources
+const { version } = JSON.parse(fs.readFileSync(path.join(__dirname, "..", "package.json"), "utf8"));
+const FIELDS = ["title", "keywords", "description", "body"];
+function parseBoost(value, previous = {}) {
+    const match = /^(\w+)=(\d+(?:\.\d+)?)$/.exec(value);
+    if (!match || !FIELDS.includes(match[1])) {
+        throw new commander_1.InvalidArgumentError(`Expected <field>=<number> with field one of ${FIELDS.join(", ")}.`);
+    }
+    return { ...previous, [match[1]]: Number(match[2]) };
+}
+/**
+ * Resolves <dest> and makes sure it lies inside the current working directory, so a
+ * mistyped or injected argument (e.g. "../../somewhere") cannot overwrite arbitrary files.
+ */
+function resolveDestination(dest) {
+    const root = process.cwd();
+    const resolved = path.resolve(root, dest);
+    const prefix = root.endsWith(path.sep) ? root : root + path.sep;
+    if (!resolved.startsWith(prefix)) {
+        throw new Error(`<dest> must be a file inside the current directory (${root}), got "${dest}".`);
+    }
+    return resolved;
+}
 commander_1.program
-    .version("2.2.8")
-    .arguments("<glob> <dest> [bodySelector]")
-    .action((glob, dest, bodySelector) => {
-    index_1.SearchIndex.createFromGlob(glob, bodySelector, (index) => fs.writeFileSync(dest || "./index.json", JSON.stringify(index)));
+    .name("mvw-search-index")
+    .description("Generates a lunr search index and result store from HTML files.")
+    .version(version)
+    .argument("<glob>", "glob pattern of the HTML files to index (quote it, so your shell doesn't expand it)")
+    .argument("<dest>", "path of the JSON file to write; must be inside the current directory")
+    .argument("[bodySelector]", "CSS selector of the content to index", "body")
+    .option("--cwd <dir>", "directory to resolve <glob> in; hrefs are relative to it (use your site's root)")
+    .option("-e, --exclude <selector>", "CSS selector of content to leave out (\"\" for none)", index_1.DEFAULT_EXCLUDE_SELECTOR)
+    .option("-l, --language <code>", "two-letter content language, selects stemmer and stop words", index_1.DEFAULT_LANGUAGE)
+    .option("--base-url <url>", "prefix for all hrefs, e.g. \"/\"")
+    .option("--strip-index-html", "link to \"dir/\" instead of \"dir/index.html\"")
+    .option("--no-noindex", "also index pages marked <meta name=\"robots\" content=\"noindex\">")
+    .option("--allow-empty", "write an empty index instead of failing if <glob> matches no files")
+    .option("-b, --boost <field=number>", "weight of a field, repeatable (default title=5 keywords=3 description=2 body=1)", parseBoost)
+    .option("-v, --verbose", "list every indexed file")
+    .showHelpAfterError()
+    .action(async (glob, dest, bodySelector, cli) => {
+    const logger = {
+        info: (message) => cli.verbose && console.log(message),
+        warn: (message) => console.warn(`Warning: ${message}`),
+    };
+    const options = {
+        bodySelector,
+        excludeSelector: cli.exclude,
+        language: cli.language,
+        cwd: cli.cwd,
+        baseUrl: cli.baseUrl,
+        stripIndexHtml: cli.stripIndexHtml,
+        respectNoindex: cli.noindex,
+        allowEmpty: cli.allowEmpty,
+        boosts: cli.boost,
+        logger,
+    };
+    try {
+        const destination = resolveDestination(dest);
+        const index = await index_1.SearchIndex.createFromGlob(glob, options);
+        await fs.promises.mkdir(path.dirname(destination), { recursive: true });
+        await fs.promises.writeFile(destination, JSON.stringify(index));
+        console.log(`Indexed ${Object.keys(index.store).length} page(s) into ${dest}`);
+    }
+    catch (err) {
+        console.error(`Error: ${err instanceof Error ? err.message : err}`);
+        process.exitCode = 1;
+    }
 })
-    .parse(process.argv);
-//# sourceMappingURL=cli.js.map
+    .parseAsync(process.argv);

package/js/href.d.ts

@@ -0,0 +1,8 @@
+export declare interface HrefOptions {
+    /** Prefix for every href, e.g. `"/"` or `"https://example.org/docs/"`. Default: `""`. */
+    baseUrl?: string;
+    /** Turn `foo/index.html` into `foo/` (and `index.html` into the base URL). Default: `false`. */
+    stripIndexHtml?: boolean;
+}
+/** Turns the path of an indexed file into the `href` of its search result. */
+export declare function toHref(relativePath: string, options?: HrefOptions): string;

package/js/href.js

@@ -0,0 +1,24 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.toHref = toHref;
+/** Turns the path of an indexed file into the `href` of its search result. */
+function toHref(relativePath, options = {}) {
+    // Windows paths use backslashes, URLs never do.
+    let href = relativePath.replaceAll("\\", "/").replace(/^\.\//, "");
+    if (options.stripIndexHtml) {
+        href = href.replace(/(^|\/)index\.html?$/, "$1");
+    }
+    const baseUrl = trimTrailingSlashes(options.baseUrl ?? "");
+    if (options.baseUrl) {
+        href = baseUrl + "/" + href;
+    }
+    // an empty href would link to the search page itself
+    return href || "./";
+}
+function trimTrailingSlashes(value) {
+    let end = value.length;
+    while (end > 0 && value[end - 1] === "/") {
+        end--;
+    }
+    return value.slice(0, end);
+}

package/js/html.d.ts

@@ -0,0 +1,26 @@
+import type { CheerioAPI } from "cheerio";
+/** Collapses all whitespace runs (including single newlines and tabs) into one space. */
+export declare function normalizeWhitespace(text: string): string;
+export declare interface PageMetadata {
+    title: string;
+    description?: string;
+    keywords?: string;
+}
+/**
+ * Reads title, description and keywords, falling back to Open Graph tags and the
+ * first `<h1>` where the regular tags are missing or empty. Call this before
+ * {@link extractText}, which modifies the document.
+ */
+export declare function extractMetadata($: CheerioAPI): PageMetadata;
+/** Whether the page asks search engines not to index it (`<meta name="robots" content="noindex">`). */
+export declare function isNoindex($: CheerioAPI): boolean;
+/** Default for the `excludeSelector` option: site-wide navigation and footers. */
+export declare const DEFAULT_EXCLUDE_SELECTOR = "nav, footer";
+/**
+ * Returns the visible text of all elements matching `selector`, with block-level
+ * elements and separate matches delimited by spaces. Descendants matching
+ * `excludeSelector` are left out.
+ *
+ * Note: modifies the document.
+ */
+export declare function extractText($: CheerioAPI, selector: string, excludeSelector?: string): string;

package/js/html.js

@@ -0,0 +1,71 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.DEFAULT_EXCLUDE_SELECTOR = void 0;
+exports.normalizeWhitespace = normalizeWhitespace;
+exports.extractMetadata = extractMetadata;
+exports.isNoindex = isNoindex;
+exports.extractText = extractText;
+/**
+ * Elements that browsers render on their own line (plus table cells). Their text
+ * must be separated from the surrounding text, otherwise `<li>a</li><li>b</li>`
+ * becomes the single word "ab".
+ */
+const BLOCK_ELEMENTS = [
+    "address", "article", "aside", "blockquote", "caption", "dd", "details", "dialog", "div", "dl", "dt",
+    "fieldset", "figcaption", "figure", "footer", "form", "h1", "h2", "h3", "h4", "h5", "h6", "header",
+    "hgroup", "hr", "li", "main", "nav", "ol", "option", "p", "pre", "section", "summary", "table",
+    "tbody", "td", "tfoot", "th", "thead", "tr", "ul",
+].join(",");
+/** Elements whose content is never visible page text. */
+const NON_CONTENT_ELEMENTS = "script, style, noscript, template";
+/** Collapses all whitespace runs (including single newlines and tabs) into one space. */
+function normalizeWhitespace(text) {
+    return text.replace(/\s+/g, " ").trim();
+}
+function nonEmpty(value) {
+    const normalized = normalizeWhitespace(value ?? "");
+    return normalized === "" ? undefined : normalized;
+}
+/**
+ * Reads title, description and keywords, falling back to Open Graph tags and the
+ * first `<h1>` where the regular tags are missing or empty. Call this before
+ * {@link extractText}, which modifies the document.
+ */
+function extractMetadata($) {
+    const meta = (selector) => nonEmpty($(selector).first().attr("content"));
+    return {
+        title: nonEmpty($("title").first().text())
+            ?? meta("meta[property='og:title']")
+            ?? nonEmpty($("h1").first().text())
+            ?? "",
+        description: meta("meta[name='description' i]") ?? meta("meta[property='og:description']"),
+        keywords: meta("meta[name='keywords' i]"),
+    };
+}
+/** Whether the page asks search engines not to index it (`<meta name="robots" content="noindex">`). */
+function isNoindex($) {
+    return $("meta[name='robots' i]").toArray()
+        .some((meta) => /\b(noindex|none)\b/i.test($(meta).attr("content") ?? ""));
+}
+/** Default for the `excludeSelector` option: site-wide navigation and footers. */
+exports.DEFAULT_EXCLUDE_SELECTOR = "nav, footer";
+/**
+ * Returns the visible text of all elements matching `selector`, with block-level
+ * elements and separate matches delimited by spaces. Descendants matching
+ * `excludeSelector` are left out.
+ *
+ * Note: modifies the document.
+ */
+function extractText($, selector, excludeSelector = exports.DEFAULT_EXCLUDE_SELECTOR) {
+    $(NON_CONTENT_ELEMENTS).remove();
+    $("br").replaceWith(" ");
+    $(BLOCK_ELEMENTS).prepend(" ").append(" ");
+    // If matches are nested (e.g. selector "div"), only take the outermost ones -
+    // otherwise the inner text would be indexed twice.
+    const roots = $(selector).filter((_, el) => $(el).parents(selector).length === 0);
+    if (excludeSelector.trim()) {
+        // only descendants: a body selector that itself matches the exclusion still works
+        roots.find(excludeSelector).remove();
+    }
+    return normalizeWhitespace(roots.map((_, el) => $(el).text()).get().join(" "));
+}

package/js/index.d.ts

@@ -1,32 +1,105 @@
 import * as lunr from "lunr";
+import { type HrefOptions } from "./href";
+export { DEFAULT_EXCLUDE_SELECTOR } from "./html";
+export type { HrefOptions } from "./href";
+export { DEFAULT_LANGUAGE } from "./language";
 export declare interface IResultStore {
     [key: string]: {
         title: string;
-        description: string | undefined;
+        description?: string;
     };
 }
 export declare interface IFileInformation {
     body: string;
-    description: string | undefined;
+    description?: string;
     href: string;
-    keywords: string | undefined;
+    keywords?: string;
     title: string;
 }
 export declare interface ISearchIndexResult {
     index: lunr.Index;
     store: IResultStore;
 }
-declare interface ReadFileWithContents {
-    contents: Buffer;
+/** An HTML document to index. */
+export declare interface HtmlFile {
+    /** The raw HTML. */
+    contents: Buffer | string;
+    /** Path of the file; used as the `href` of the search result. */
     relative: string;
 }
+/** @deprecated Use {@link HtmlFile}. */
+export type ReadFileWithContents = HtmlFile;
+/** Receives progress and diagnostic messages. `console` satisfies this interface. */
+export declare interface Logger {
+    info(message: string): void;
+    warn(message: string): void;
+}
+/** The indexed fields. */
+export type SearchField = "title" | "keywords" | "description" | "body";
+/** Default for the `boosts` option. */
+export declare const DEFAULT_BOOSTS: Readonly<Record<SearchField, number>>;
+export declare interface SearchIndexOptions extends HrefOptions {
+    /** CSS selector of the element(s) whose text is indexed as body. Default: `"body"`. */
+    bodySelector?: string;
+    /**
+     * CSS selector of elements inside the body to leave out, e.g. navigation and footers
+     * that repeat on every page. Use `""` to exclude nothing. Default: `"nav, footer"`.
+     */
+    excludeSelector?: string;
+    /**
+     * Skip pages with `<meta name="robots" content="noindex">` (or `none`), just like
+     * search engines do. Default: `true`.
+     */
+    respectNoindex?: boolean;
+    /**
+     * Language of the content, as two-letter code. Selects stop words and stemmer.
+     * Anything other than `"en"` uses the matching plugin of
+     * [lunr-languages](https://github.com/MihaiValentin/lunr-languages), which the
+     * client must then load as well before calling `lunr.Index.load()`. Default: `"en"`.
+     */
+    language?: string;
+    /**
+     * Relative weight of a match per field, so that a match in the title ranks above a
+     * match somewhere in the body text. Missing fields use the defaults. Default:
+     * `{title: 5, keywords: 3, description: 2, body: 1}`.
+     */
+    boosts?: Partial<Record<SearchField, number>>;
+    /**
+     * `createFromGlob` only: resolve with an empty index instead of rejecting when the
+     * pattern matches no files. Default: `false`.
+     */
+    allowEmpty?: boolean;
+    /**
+     * `createFromGlob` only: directory the pattern is resolved against. The paths of
+     * the matched files relative to it become the hrefs, so point it at the root of
+     * the built site. Default: `process.cwd()`.
+     */
+    cwd?: string;
+    /** Where to report progress (one message per indexed file). Default: silent. */
+    logger?: Logger;
+}
 export declare class SearchIndex {
     private readonly store;
     private readonly index;
     private constructor();
-    static createFromInfo(files: IFileInformation[]): ISearchIndexResult;
-    static createFromHtml(files: ReadFileWithContents[], bodySelector?: string): ISearchIndexResult;
-    static createFromGlob(pattern: string, bodySelector: string, cb: (index: ISearchIndexResult) => void): void;
+    /**
+     * @param files Already extracted page information to index.
+     * @param options Only `language` and `boosts` apply here; the other options concern HTML parsing.
+     */
+    static createFromInfo(files: IFileInformation[], options?: SearchIndexOptions): ISearchIndexResult;
+    /**
+     * @param files HTML documents to index.
+     * @param options Options, or - for backwards compatibility - just the body selector.
+     */
+    static createFromHtml(files: HtmlFile[], options?: string | SearchIndexOptions): ISearchIndexResult;
+    /**
+     * Indexes all HTML files matching a glob pattern.
+     *
+     * @param pattern Glob pattern of the HTML files to index.
+     * @param options Options, or - for backwards compatibility - just the body selector.
+     * @returns The index and result store. Rejects if a file cannot be read.
+     */
+    static createFromGlob(pattern: string, options?: string | SearchIndexOptions, ...legacyCallback: never[]): Promise<ISearchIndexResult>;
+    private static createFromGlobAsync;
     private getResult;
 }
-export {};

package/js/index.js

@@ -33,60 +33,131 @@ var __importStar = (this && this.__importStar) || (function () {
     };
 })();
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.SearchIndex = void 0;
+exports.SearchIndex = exports.DEFAULT_BOOSTS = exports.DEFAULT_LANGUAGE = exports.DEFAULT_EXCLUDE_SELECTOR = void 0;
 const cheerio = __importStar(require("cheerio"));
 const glob_1 = require("glob");
-const fs = __importStar(require("fs"));
+const fs = __importStar(require("node:fs"));
+const path = __importStar(require("node:path"));
 const lunr = __importStar(require("lunr"));
+const html_1 = require("./html");
+const href_1 = require("./href");
+const language_1 = require("./language");
+var html_2 = require("./html");
+Object.defineProperty(exports, "DEFAULT_EXCLUDE_SELECTOR", { enumerable: true, get: function () { return html_2.DEFAULT_EXCLUDE_SELECTOR; } });
+var language_2 = require("./language");
+Object.defineProperty(exports, "DEFAULT_LANGUAGE", { enumerable: true, get: function () { return language_2.DEFAULT_LANGUAGE; } });
+const silentLogger = {
+    info: () => undefined,
+    warn: () => undefined,
+};
+/** Default for the `boosts` option. */
+exports.DEFAULT_BOOSTS = Object.freeze({
+    title: 5,
+    keywords: 3,
+    description: 2,
+    body: 1,
+});
+function normalizeOptions(options) {
+    return typeof options === "string" ? { bodySelector: options } : { ...options };
+}
 class SearchIndex {
     store;
     index;
-    constructor(files) {
+    constructor(files, options) {
         this.store = {};
         const builder = new lunr.Builder();
-        builder.field("title");
-        builder.field("keywords");
-        builder.field("description");
-        builder.field("body");
+        // The same text processing lunr() sets up by default. A bare Builder has empty
+        // pipelines, which meant no stemming, no stop word removal and punctuation
+        // sticking to words ("konzert," / "page:"). The search pipeline is serialized
+        // into the index, so lunr applies the stemmer to queries on the client as well.
+        builder.pipeline.add(lunr.trimmer, lunr.stopWordFilter, lunr.stemmer);
+        builder.searchPipeline.add(lunr.stemmer);
+        const plugin = (0, language_1.languagePlugin)(options.language ?? language_1.DEFAULT_LANGUAGE);
+        if (plugin) {
+            builder.use(plugin); // replaces both pipelines with the language specific ones
+        }
+        const boosts = { ...exports.DEFAULT_BOOSTS, ...options.boosts };
+        for (const field of Object.keys(exports.DEFAULT_BOOSTS)) {
+            builder.field(field, { boost: boosts[field] });
+        }
         builder.ref("href");
         files.forEach((info) => {
+            if (Object.hasOwn(this.store, info.href)) {
+                throw new Error(`Duplicate href "${info.href}": every document needs a unique href.`);
+            }
             this.store[info.href] = {
                 description: info.description,
                 title: info.title,
             };
-            builder.add(info);
-        }, builder);
+            // keywords are a comma separated list, but lunr only splits on whitespace and hyphens
+            builder.add({ ...info, keywords: info.keywords?.replace(/[,;]/g, " ") });
+        });
         this.index = builder.build();
     }
-    static createFromInfo(files) {
-        return new SearchIndex(files).getResult();
+    /**
+     * @param files Already extracted page information to index.
+     * @param options Only `language` and `boosts` apply here; the other options concern HTML parsing.
+     */
+    static createFromInfo(files, options) {
+        return new SearchIndex(files, normalizeOptions(options)).getResult();
     }
-    static createFromHtml(files, bodySelector = "body") {
-        const infos = files.map((file) => {
-            console.info(file.relative);
+    /**
+     * @param files HTML documents to index.
+     * @param options Options, or - for backwards compatibility - just the body selector.
+     */
+    static createFromHtml(files, options) {
+        const normalized = normalizeOptions(options);
+        const { bodySelector, excludeSelector, respectNoindex = true, logger = silentLogger } = normalized;
+        const infos = [];
+        for (const file of files) {
             const dom = cheerio.load(file.contents.toString());
-            return {
-                body: dom(bodySelector || "body").text().replace(/\s\s+/g, " "),
-                href: file.relative,
-                description: dom("meta[name='description']").attr("content"),
-                keywords: dom("meta[name='keywords']").attr("content"),
-                title: dom("head title").text(),
-            };
-        });
-        return SearchIndex.createFromInfo(infos);
+            if (respectNoindex && (0, html_1.isNoindex)(dom)) {
+                logger.info(`Skipping ${file.relative} (robots noindex)`);
+                continue;
+            }
+            logger.info(`Indexing ${file.relative}`);
+            const metadata = (0, html_1.extractMetadata)(dom);
+            if (!metadata.title) {
+                logger.warn(`${file.relative} has no <title>, og:title or <h1> - its search result will have an empty title`);
+            }
+            infos.push({
+                ...metadata,
+                body: (0, html_1.extractText)(dom, bodySelector || "body", excludeSelector),
+                href: (0, href_1.toHref)(file.relative, normalized),
+            });
+        }
+        return SearchIndex.createFromInfo(infos, normalized);
     }
-    static createFromGlob(pattern, bodySelector, cb) {
-        (0, glob_1.glob)(pattern, {
-            dotRelative: false
-        }).then(files => {
-            const readFiles = files.map((file) => ({
-                relative: file,
-                contents: fs.readFileSync(file)
-            }));
-            cb(SearchIndex.createFromHtml(readFiles, bodySelector));
-        }).catch(err => {
-            throw err;
-        });
+    /**
+     * Indexes all HTML files matching a glob pattern.
+     *
+     * @param pattern Glob pattern of the HTML files to index.
+     * @param options Options, or - for backwards compatibility - just the body selector.
+     * @returns The index and result store. Rejects if a file cannot be read.
+     */
+    static createFromGlob(pattern, options, ...legacyCallback) {
+        if (legacyCallback.length > 0) {
+            // 2.x took a callback as third argument. Silently ignoring it would mean the
+            // caller's index is simply never written - fail loudly instead.
+            throw new TypeError("SearchIndex.createFromGlob() no longer accepts a callback; it returns a Promise. "
+                + "See https://github.com/Tiliavir/mvw-search-index/blob/main/UPGRADING.md");
+        }
+        return SearchIndex.createFromGlobAsync(pattern, normalizeOptions(options));
+    }
+    static async createFromGlobAsync(pattern, options) {
+        const cwd = path.resolve(options.cwd ?? ".");
+        // glob's result order depends on the file system - sort for reproducible output
+        const files = (await (0, glob_1.glob)(pattern, { cwd, dotRelative: false, nodir: true, posix: true }))
+            .sort((a, b) => a.localeCompare(b, "en"));
+        if (files.length === 0 && !options.allowEmpty) {
+            throw new Error(`No files match "${pattern}" in ${cwd}. `
+                + "Set the allowEmpty option to create an empty index anyway.");
+        }
+        const readFiles = await Promise.all(files.map(async (file) => ({
+            relative: file,
+            contents: await fs.promises.readFile(path.join(cwd, file)),
+        })));
+        return SearchIndex.createFromHtml(readFiles, options);
     }
     getResult() {
         return {
@@ -96,4 +167,3 @@ class SearchIndex {
     }
 }
 exports.SearchIndex = SearchIndex;
-//# sourceMappingURL=index.js.map

package/js/language.d.ts

@@ -0,0 +1,8 @@
+import type * as Lunr from "lunr";
+/** lunr's built-in language; needs no plugin. */
+export declare const DEFAULT_LANGUAGE = "en";
+/**
+ * Returns the lunr-languages plugin for `language` (an ISO 639-1 code such as
+ * "de"), registering it with lunr on first use, or `undefined` for English.
+ */
+export declare function languagePlugin(language: string): Lunr.Builder.Plugin | undefined;

package/js/language.js

@@ -0,0 +1,45 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.DEFAULT_LANGUAGE = void 0;
+exports.languagePlugin = languagePlugin;
+// lunr-languages plugins register themselves by mutating the lunr module object, so
+// they need the actual CommonJS export - an ES namespace wrapper (as created by
+// bundlers and test runners for `import * as`) would silently drop the additions.
+// eslint-disable-next-line @typescript-eslint/no-require-imports
+const lunr = require("lunr");
+/** lunr's built-in language; needs no plugin. */
+exports.DEFAULT_LANGUAGE = "en";
+let stemmerSupportLoaded = false;
+/**
+ * Returns the lunr-languages plugin for `language` (an ISO 639-1 code such as
+ * "de"), registering it with lunr on first use, or `undefined` for English.
+ */
+function languagePlugin(language) {
+    if (language === exports.DEFAULT_LANGUAGE) {
+        return undefined;
+    }
+    if (!/^[a-z]{2}$/.test(language)) {
+        throw new Error(`Invalid language "${language}": expected a two-letter code such as "de" or "fr".`);
+    }
+    const registry = lunr;
+    if (!registry[language]) {
+        try {
+            if (!stemmerSupportLoaded) {
+                // eslint-disable-next-line @typescript-eslint/no-require-imports -- loaded on demand, synchronously
+                require("lunr-languages/lunr.stemmer.support")(lunr);
+                stemmerSupportLoaded = true;
+            }
+            // eslint-disable-next-line @typescript-eslint/no-require-imports -- loaded on demand, synchronously
+            require(`lunr-languages/lunr.${language}`)(lunr);
+        }
+        catch (err) {
+            throw new Error(`Unsupported language "${language}": ${err instanceof Error ? err.message : err}. `
+                + "See https://github.com/MihaiValentin/lunr-languages for the supported languages.", { cause: err });
+        }
+    }
+    const plugin = registry[language];
+    if (!plugin) {
+        throw new Error(`Unsupported language "${language}".`);
+    }
+    return plugin;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mvw-search-index",
-  "version": "2.3.7",
+  "version": "3.0.1",
   "description": "Module to generate the search index using lunr.",
   "keywords": [
     "website",
@@ -22,27 +22,36 @@
   },
   "main": "js/index.js",
   "types": "js/index.d.ts",
+  "files": [
+    "js/**/*.js",
+    "js/**/*.d.ts"
+  ],
+  "engines": {
+    "node": ">=22.12.0"
+  },
   "devDependencies": {
-    "@types/cheerio": "1.0.0",
+    "@eslint/js": "10.0.1",
     "@types/lunr": "2.3.7",
-    "@types/node": "25.5.2",
-    "@typescript-eslint/eslint-plugin": "8.58.0",
-    "@typescript-eslint/parser": "8.58.0",
-    "eslint": "10.2.0",
+    "@types/node": "26.6.4",
+    "eslint": "10.12.0",
     "serve": "14.2.6",
-    "typescript": "6.0.2",
-    "vitest": "4.1.2"
+    "typescript": "6.0.3",
+    "typescript-eslint": "8.71.1",
+    "vitest": "5.0.3"
   },
   "dependencies": {
     "cheerio": "1.2.0",
-    "commander": "14.0.3",
+    "commander": "15.0.0",
     "glob": "13.0.6",
-    "lunr": "2.3.9"
+    "lunr": "2.3.9",
+    "lunr-languages": "1.22.0"
   },
   "scripts": {
+    "prepare": "npm run build",
     "build": "tsc -p ./ts",
-    "test": "npx eslint . && npm run build && npx vitest run",
-    "preserve": "cd docs && node ../js/cli.js './**/*.html' index.json 'body.to-be-indexed'",
+    "lint": "eslint . && tsc -p spec",
+    "test": "npm run lint && npm run build && vitest run",
+    "preserve": "node js/cli.js '**/*.html' docs/index.json 'body.to-be-indexed' --cwd docs",
     "serve": "serve docs"
   },
   "repository": {

package/README.Release.md

@@ -1,5 +0,0 @@
-# Release process
-
-- update version in `package.json`
-- commit with `Release <version>` in commit message
-- the publish action will add a tag and publish to npm automatically

package/eslint.config.mjs

@@ -1,15 +0,0 @@
-import tsEsLint from "@typescript-eslint/eslint-plugin";
-import tsParser from "@typescript-eslint/parser";
-
-export default [
-  {
-    files: ["**/*.ts"],
-    plugins: {
-      "@typescript-eslint": tsEsLint,
-    },
-
-    languageOptions: {
-      parser: tsParser,
-    }
-  },
-];

package/js/cli.js.map

@@ -1 +0,0 @@
-{"version":3,"file":"cli.js","sourceRoot":"","sources":["../ts/cli.ts"],"names":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAEA,yCAAoC;AACpC,uCAAyB;AAEzB,mCAAsC;AAEtC,mBAAO;KACJ,OAAO,CAAC,OAAO,CAAC;KAChB,SAAS,CAAC,8BAA8B,CAAC;KACzC,MAAM,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,YAAY,EAAE,EAAE;IACnC,mBAAW,CAAC,cAAc,CAAC,IAAI,EAAE,YAAY,EAAE,CAAC,KAAK,EAAE,EAAE,CACvD,EAAE,CAAC,aAAa,CAAC,IAAI,IAAI,cAAc,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,CAChE,CAAC;AACJ,CAAC,CAAC;KACD,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC"}

package/js/index.js.map

@@ -1 +0,0 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../ts/index.ts"],"names":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAAA,iDAAmC;AACnC,+BAA0B;AAC1B,uCAAyB;AACzB,2CAA6B;AA4B7B,MAAa,WAAW;IACL,KAAK,CAAe;IACpB,KAAK,CAAa;IAEnC,YAAoB,KAAyB;QAC3C,IAAI,CAAC,KAAK,GAAG,EAAE,CAAC;QAChB,MAAM,OAAO,GAAiB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;QACjD,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;QACvB,OAAO,CAAC,KAAK,CAAC,UAAU,CAAC,CAAC;QAC1B,OAAO,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC;QAC7B,OAAO,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;QACtB,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QAEpB,KAAK,CAAC,OAAO,CAAC,CAAC,IAAsB,EAAQ,EAAE;YAC7C,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG;gBACtB,WAAW,EAAE,IAAI,CAAC,WAAW;gBAC7B,KAAK,EAAE,IAAI,CAAC,KAAK;aAClB,CAAC;YACF,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACpB,CAAC,EAAE,OAAO,CAAC,CAAC;QACZ,IAAI,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,EAAE,CAAC;IAC/B,CAAC;IAEM,MAAM,CAAC,cAAc,CAAC,KAAyB;QACpD,OAAO,IAAI,WAAW,CAAC,KAAK,CAAC,CAAC,SAAS,EAAE,CAAC;IAC5C,CAAC;IAEM,MAAM,CAAC,cAAc,CAAC,KAA6B,EAAE,eAAuB,MAAM;QACvF,MAAM,KAAK,GAAuB,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE;YACnD,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;YAC5B,MAAM,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,QAAQ,EAAE,CAAC,CAAC;YACnD,OAAO;gBACL,IAAI,EAAE,GAAG,CAAC,YAAY,IAAI,MAAM,CAAC,CAAC,IAAI,EAAE,CAAC,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC;gBAC/D,IAAI,EAAE,IAAI,CAAC,QAAQ;gBACnB,WAAW,EAAE,GAAG,CAAC,0BAA0B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC;gBAC5D,QAAQ,EAAE,GAAG,CAAC,uBAAuB,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC;gBACtD,KAAK,EAAE,GAAG,CAAC,YAAY,CAAC,CAAC,IAAI,EAAE;aAChC,CAAC;QACJ,CAAC,CAAC,CAAC;QAEH,OAAO,WAAW,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;IAC3C,CAAC;IAEM,MAAM,CAAC,cAAc,CAAC,OAAe,EACf,YAAoB,EACpB,EAAuC;QAClE,IAAA,WAAI,EAAC,OAAO,EAAE;YACZ,WAAW,EAAE,KAAK;SACnB,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE;YACZ,MAAM,SAAS,GAA2B,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;gBAC7D,QAAQ,EAAE,IAAI;gBACd,QAAQ,EAAE,EAAE,CAAC,YAAY,CAAC,IAAI,CAAC;aAChC,CAAC,CAAC,CAAC;YACJ,EAAE,CAAC,WAAW,CAAC,cAAc,CAAC,SAAS,EAAE,YAAY,CAAC,CAAC,CAAC;QAC1D,CAAC,CACF,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE;YACZ,MAAM,GAAG,CAAC;QACZ,CAAC,CAAC,CAAC;IACL,CAAC;IAEO,SAAS;QACf,OAAO;YACL,KAAK,EAAE,IAAI,CAAC,KAAK;YACjB,KAAK,EAAE,IAAI,CAAC,KAAK;SAClB,CAAC;IACJ,CAAC;CACF;AAlED,kCAkEC"}

package/vitest.config.ts

@@ -1,8 +0,0 @@
-import { defineConfig } from "vitest/config";
-
-export default defineConfig({
-  test: {
-    globals: true,
-    include: ["spec/**/*.spec.ts"],
-  },
-});