wiki-search-index 0.1.0

package/INDEX-FORMAT.md

@@ -0,0 +1,74 @@
+# Index format (v1)
+
+A wiki-search index is a single, self-describing JSON document. A client
+**assumes nothing** beyond this contract: it validates the version and required
+fields, then builds result links purely from the index's own metadata. Any site
+that emits this shape is searchable — wiki-search is not GitHub-specific.
+
+```json
+{
+  "v": 1,
+  "site": {
+    "name": "wiki-search wiki",
+    "urlTemplate": "https://github.com/uhop/wiki-search/wiki/{page}",
+    "fragments": true
+  },
+  "docs": [
+    {
+      "id": 0,
+      "page": "Index-Format",
+      "title": "Index format",
+      "heading": "Validation",
+      "anchor": "validation",
+      "text": "full plain-text of the section…"
+    }
+  ]
+}
+```
+
+## Fields
+
+| Field | Meaning |
+|-------|---------|
+| `v` | Format version. This document is `1`. Clients reject versions they don't understand. |
+| `site.name` | Human label for the corpus (shown in the UI). |
+| `site.urlTemplate` | Result-URL template; **must contain `{page}`**. No hardcoded host. |
+| `site.fragments` | `true` if the target renders [Text Fragments](https://developer.mozilla.org/docs/Web/Text_fragments). When `false`, clients omit the `:~:text=` directive. |
+| `docs[]` | One entry per indexed section. |
+| `doc.id` | Stable integer, sequential in build order. |
+| `doc.page` | The `{page}` substitution — for GitHub wikis, the page's URL segment (`Foo-Bar`). |
+| `doc.title` | Page display title. |
+| `doc.heading` | Section heading (falls back to the page title for a page's preamble). |
+| `doc.anchor` | In-page anchor slug; `""` means the page top. For GitHub, the heading's slug. |
+| `doc.text` | Plain-text body of the section (markdown stripped), for the search engine. |
+
+## Building a result URL
+
+```
+base   = urlTemplate.replace("{page}", encodeURIComponent(doc.page))
+hash   = doc.anchor || ""              (omit if empty)
+text   = ":~:text=" + <matched phrase> (only if site.fragments and a phrase)
+result = base + ("#" + hash + text  if hash or text)
+```
+
+So a hit links to e.g.
+`https://github.com/uhop/wiki-search/wiki/Index-Format#validation:~:text=clients%20reject`.
+
+## Validation (verify-or-explain)
+
+A client must check, and on any failure show a specific message (never a blank
+result box):
+
+1. the index is **fetchable** (else: 404 / network / no CORS);
+2. it is **valid JSON**;
+3. `v` is **supported** (else: format vN unsupported — app or index out of date);
+4. `site.urlTemplate` is present and contains `{page}`;
+5. `docs` is a non-empty array, each entry having `page`, `title`, `text`.
+
+## Versioning
+
+`v` increases only on a breaking change to this shape. Additive, optional fields
+do **not** bump `v`; clients ignore unknown fields. A client that meets a higher
+`v` than it knows stops and says so rather than guessing.
+
+The reference builder that emits this is `builder/wiki-index.mjs`.

package/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, Eugene Lazutkin
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

package/README.md

@@ -0,0 +1,54 @@
+# wiki-search
+
+GitHub wikis are great for docs but have no real search. **wiki-search adds it:**
+a bookmarklet (plus a hosted search page) that searches a wiki and takes you
+straight to the matching section — without moving your docs off the wiki.
+
+**▶ [Try the live demo & install the bookmarklet](https://uhop.github.io/wiki-search/)**
+
+## Use it
+
+- **Search a wiki.** Drag the bookmarklet to your bookmarks bar, then open any
+  GitHub wiki that has a wiki-search index and search — each result jumps you to
+  the exact section. ([Try it now](https://uhop.github.io/wiki-search/app/?wiki=uhop/wiki-search)
+  on this project's own wiki — no install needed.)
+- **Add search to your own wiki.** Build an index from your Markdown — needs
+  [Node](https://nodejs.org), nothing to install:
+
+  ```bash
+  npx wiki-search-index --wiki ./your-wiki   # → your-wiki/search-index.json
+  ```
+
+  Commit that `search-index.json` into your wiki, then add a one-line Search
+  section. Full guide — including keeping the index fresh —
+  [Add search to your wiki](https://github.com/uhop/wiki-search/wiki/Add-Search).
+
+## How it works
+
+The bookmarklet opens the search page on its own GitHub Pages origin, so the
+wiki page's security policy can't block it. There it loads a small JSON index
+built from the wiki's Markdown and links each result with a
+[text fragment](https://developer.mozilla.org/docs/Web/Text_fragments), so your
+browser scrolls to — and, where supported, highlights — the matched phrase. By
+default a result re-uses your current tab; Back returns you. The index carries
+its own URL template, so the same app works for any site with no hardcoded host.
+
+<details>
+<summary>Repo layout & local run</summary>
+
+| Path | What |
+|------|------|
+| `index.html` | Landing + bookmarklet-install page (the Pages root). |
+| `app/` | The search page (loads + validates an index, searches, links out). |
+| `builder/` | `wiki-index` CLI: Markdown → the JSON index. |
+| `engine/` | Search core: MiniSearch (vendored), with a zero-dep fallback. |
+| `bookmarklet/` | The `window.open` stub + builder. |
+
+Run locally: `python3 -m http.server` from the repo root, then open
+<http://localhost:8000/app/?wiki=uhop/wiki-search>.
+
+</details>
+
+## License
+
+[BSD-3-Clause](LICENSE)

package/builder/README.md

@@ -0,0 +1,30 @@
+# builder — `wiki-index`
+
+Turns a directory of GitHub-wiki Markdown into a self-describing
+[v1 index](../INDEX-FORMAT.md). Dependency-free; deterministic output so a CI
+`git diff --exit-code` can gate a stale committed index.
+
+```bash
+# Build the index for our own wiki (owner/repo + template inferred from git origin)
+node builder/wiki-index.mjs --wiki ./wiki                 # → wiki/search-index.json
+node builder/wiki-index.mjs --wiki ./wiki --stdout        # print instead of writing
+
+# Any other site: give the template explicitly
+node builder/wiki-index.mjs --wiki ./docs --url-template 'https://example.com/d/{page}' --name 'Example docs'
+```
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `--wiki <dir>` | `./wiki` | Markdown source directory. |
+| `--out <path>` | `<wiki>/search-index.json` | Where to write. |
+| `--stdout` | — | Print the index instead of writing a file. |
+| `--url-template <tpl>` | inferred | Result-URL template; must contain `{page}`. |
+| `--repo <owner/repo>` | inferred | Build the GitHub template from this. |
+| `--name <str>` | `<repo> wiki` | `site.name`. |
+
+What it does: one section per ATX heading (plus a page-top preamble section),
+GitHub-style heading anchors with `-1`/`-2` disambiguation, `#`-in-code-fence
+ignored, markdown reduced to plain text for the engine. Pages named `_*.md`
+(`_Sidebar`, `_Footer`) are treated as chrome and skipped.
+
+Tests: `node builder/test/run.mjs`.

package/builder/lib/build-index.mjs

@@ -0,0 +1,45 @@
+// builder/lib/build-index.mjs — assemble a self-describing v1 index from a
+// directory of GitHub-wiki Markdown files.
+//
+// Deterministic by construction: pages sorted by filename, sections in document
+// order, sequential ids, no timestamps — so a CI `git diff --exit-code` can gate
+// a stale committed index.
+
+import { readdir, readFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import { splitSections } from './markdown.mjs';
+import { createSlugger } from './slug.mjs';
+
+// GitHub stores page "Foo Bar" as Foo-Bar.md and special pages (_Sidebar,
+// _Footer, …) start with an underscore — those are chrome, not content.
+const isContentPage = name => name.endsWith('.md') && !name.startsWith('_');
+
+const firstH1 = md => {
+  const m = /^#\s+(.+?)\s*#*\s*$/m.exec(md);
+  return m ? m[1].trim() : null;
+};
+
+export const buildIndex = async ({ wikiDir, urlTemplate, siteName, fragments = true }) => {
+  const files = (await readdir(wikiDir)).filter(isContentPage).sort();
+  const docs = [];
+  let id = 0;
+
+  for (const file of files) {
+    const page = file.slice(0, -3); // URL segment: Foo-Bar.md → {page}=Foo-Bar → /wiki/Foo-Bar
+    const md = await readFile(join(wikiDir, file), 'utf8');
+    const title = firstH1(md) || page.replace(/-/g, ' ');
+    const slug = createSlugger();
+    for (const sec of splitSections(md)) {
+      docs.push({
+        id: id++,
+        page,
+        title,
+        heading: sec.heading ?? title,
+        anchor: sec.heading ? slug(sec.heading) : '', // preamble → page top (no anchor)
+        text: sec.text || sec.heading || title,
+      });
+    }
+  }
+
+  return { v: 1, site: { name: siteName, urlTemplate, fragments }, docs };
+};

package/builder/lib/markdown.mjs

@@ -0,0 +1,42 @@
+// builder/lib/markdown.mjs — split a Markdown page into heading-delimited
+// sections and reduce each to searchable plain text. Dependency-free and
+// intentionally approximate: it serves retrieval, not faithful rendering.
+
+const FENCE = /^[ \t]*(```|~~~)/;
+const ATX = /^(#{1,6})\s+(.*?)\s*#*\s*$/;
+
+// Split markdown into sections. Text before the first heading becomes a
+// preamble section with heading=null, level=0. `#` inside fenced code is
+// ignored so code comments don't masquerade as headings.
+export const splitSections = md => {
+  const sections = [{ level: 0, heading: null, lines: [] }];
+  let inFence = false;
+  for (const line of md.split(/\r?\n/)) {
+    if (FENCE.test(line)) inFence = !inFence;
+    const m = inFence ? null : ATX.exec(line);
+    if (m) sections.push({ level: m[1].length, heading: m[2].trim(), lines: [] });
+    else sections.at(-1).lines.push(line);
+  }
+  return sections
+    .map(s => ({ level: s.level, heading: s.heading, text: toPlainText(s.lines.join('\n')) }))
+    .filter(s => s.heading || s.text); // drop an empty preamble
+};
+
+// Reduce Markdown to plain, collapsed text. Code *text* is kept (API names are
+// worth searching) — only the fence delimiters are removed.
+export const toPlainText = md =>
+  md
+    .replace(/^[ \t]*(```|~~~).*$/gm, ' ') // fence delimiters (keep the code text)
+    .replace(/`([^`]*)`/g, '$1')           // inline code → its text
+    .replace(/\[\[([^\]|]+)\|([^\]]+)\]\]/g, '$1') // [[Display|Page]] wiki link → Display
+    .replace(/\[\[([^\]]+)\]\]/g, '$1')    // [[Page]] wiki link → Page
+    .replace(/!\[([^\]]*)\]\([^)]*\)/g, '$1') // image → alt
+    .replace(/\[([^\]]*)\]\([^)]*\)/g, '$1')  // link → text
+    .replace(/<[^>]+>/g, ' ')              // strip HTML tags
+    .replace(/^[ \t>]*>+/gm, ' ')          // blockquote markers
+    .replace(/^\s{0,3}([-*+]|\d+\.)\s+/gm, ' ') // list markers
+    .replace(/[*_~]+/g, '')                // emphasis
+    .replace(/^\s*\|.*$/gm, m => m.replace(/\|/g, ' ')) // table pipes
+    .replace(/^#{1,6}\s+/gm, '')           // stray heading marks
+    .replace(/\s+/g, ' ')
+    .trim();

package/builder/lib/slug.mjs

@@ -0,0 +1,27 @@
+// builder/lib/slug.mjs — GitHub-style heading slugs (approximates github-slugger).
+//
+// GitHub lowercases a heading, strips most punctuation, turns spaces into
+// hyphens, and disambiguates repeats *within a page* as -1, -2, …. This is a
+// close approximation, good enough for English docs; S2/S3 verify the anchors
+// against real rendered pages and flag any divergence.
+
+export const slugify = text =>
+  text
+    .trim()
+    .toLowerCase()
+    .replace(/\s+/g, ' ')                // collapse source whitespace, as HTML rendering does
+    .replace(/[^\p{L}\p{N}_ -]+/gu, '')  // drop punctuation/symbols (em dash, quotes, colon, parens…)
+    .replace(/ /g, '-');                 // each remaining space → one hyphen, so a removed char
+                                         // between two spaces yields "--", matching github-slugger
+
+// A per-page deduping slugger: call the returned fn on each heading in document
+// order so duplicates get GitHub's -1 / -2 / … suffixes.
+export const createSlugger = () => {
+  const seen = new Map();
+  return text => {
+    const base = slugify(text);
+    const n = seen.get(base) ?? 0;
+    seen.set(base, n + 1);
+    return n === 0 ? base : `${base}-${n}`;
+  };
+};

package/builder/wiki-index.mjs

@@ -0,0 +1,75 @@
+#!/usr/bin/env node
+// builder/wiki-index.mjs — CLI: GitHub-wiki Markdown → self-describing v1 index.
+//
+//   node builder/wiki-index.mjs [--wiki ./wiki] [--out <path>]
+//        [--url-template <tpl>] [--name "<site name>"]
+//        [--repo owner/repo] [--stdout]
+//
+// With neither --url-template nor --repo, it infers owner/repo from the wiki
+// dir's git origin (…/<owner>/<repo>.wiki.git) and builds the GitHub template.
+// Default --out is <wiki>/search-index.json (the index is hosted from the wiki).
+
+import { writeFile } from 'node:fs/promises';
+import { join, resolve } from 'node:path';
+import { execFile } from 'node:child_process';
+import { promisify } from 'node:util';
+import { buildIndex } from './lib/build-index.mjs';
+
+const run = promisify(execFile);
+
+const BOOLEAN_FLAGS = new Set(['stdout']);
+
+// Accept both --key=value and --key value; flags in BOOLEAN_FLAGS (and a --key
+// followed by another --flag or nothing) are valueless booleans.
+const parseArgs = argv => {
+  const args = {};
+  for (let i = 0; i < argv.length; ++i) {
+    const m = /^--([^=]+)(?:=(.*))?$/.exec(argv[i]);
+    if (!m) continue;
+    const key = m[1];
+    if (m[2] !== undefined) { args[key] = m[2]; continue; }
+    const next = argv[i + 1];
+    if (!BOOLEAN_FLAGS.has(key) && next !== undefined && !next.startsWith('--')) args[key] = argv[++i];
+    else args[key] = true;
+  }
+  return args;
+};
+
+// owner/repo from the wiki clone's origin, tolerating the …/.wiki.git suffix.
+const inferRepo = async wikiDir => {
+  try {
+    const { stdout } = await run('git', ['-C', wikiDir, 'remote', 'get-url', 'origin']);
+    const m = /[/:]([^/]+)\/([^/]+?)(?:\.wiki)?\.git$/.exec(stdout.trim());
+    return m ? `${m[1]}/${m[2]}` : null;
+  } catch {
+    return null;
+  }
+};
+
+const main = async () => {
+  const args = parseArgs(process.argv.slice(2));
+  const wikiDir = resolve(args.wiki || './wiki');
+
+  let repo = args.repo || null;
+  if (!args['url-template'] && !repo) repo = await inferRepo(wikiDir);
+
+  const urlTemplate = args['url-template'] || (repo && `https://github.com/${repo}/wiki/{page}`);
+  if (!urlTemplate) {
+    console.error('wiki-index: need --url-template or --repo owner/repo (could not infer from git origin).');
+    process.exit(2);
+  }
+  const siteName = args.name || (repo ? `${repo.split('/')[1]} wiki` : 'wiki');
+
+  const index = await buildIndex({ wikiDir, urlTemplate, siteName });
+  const json = JSON.stringify(index, null, 2) + '\n';
+
+  if (args.stdout) { process.stdout.write(json); return; }
+
+  const out = resolve(args.out || join(wikiDir, 'search-index.json'));
+  await writeFile(out, json);
+  const pages = new Set(index.docs.map(d => d.page)).size;
+  console.error(`wiki-index: ${index.docs.length} sections from ${pages} page(s) → ${out}`);
+  console.error(`            site "${siteName}" · ${urlTemplate}`);
+};
+
+main();

package/package.json

@@ -0,0 +1,38 @@
+{
+  "name": "wiki-search-index",
+  "version": "0.1.0",
+  "description": "Build a self-describing search index from a GitHub wiki (or any Markdown docs) — the indexer for wiki-search.",
+  "type": "module",
+  "bin": {
+    "wiki-search-index": "builder/wiki-index.mjs"
+  },
+  "files": [
+    "builder/wiki-index.mjs",
+    "builder/lib",
+    "builder/README.md",
+    "INDEX-FORMAT.md"
+  ],
+  "engines": {
+    "node": ">=18"
+  },
+  "keywords": [
+    "wiki",
+    "search",
+    "github-wiki",
+    "search-index",
+    "bookmarklet",
+    "documentation",
+    "wiki-search",
+    "static-search"
+  ],
+  "homepage": "https://uhop.github.io/wiki-search/",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/uhop/wiki-search.git"
+  },
+  "bugs": {
+    "url": "https://github.com/uhop/wiki-search/issues"
+  },
+  "author": "Eugene Lazutkin",
+  "license": "BSD-3-Clause"
+}