wacrawl-dashboard 0.1.1

package/README.md

@@ -0,0 +1,133 @@
+# WaCrawl Dashboard
+
+Local WhatsApp analytics dashboard for the SQLite archive generated by `wacrawl`. A small monorepo runs a **read-only Express API** (`apps/api`) and a **React + Vite** UI (`apps/web`) that talks to it over CORS on localhost only.
+
+## One-line install
+
+```bash
+npx wacrawl-dashboard@latest
+```
+
+Opens the dashboard on `http://127.0.0.1:3001`. Requires Node 20+ and a WaCrawl archive at `~/.wacrawl/wacrawl.db`.
+Point it at a different archive:
+
+```bash
+WACRAWL_DB=/path/to/wacrawl.db npx wacrawl-dashboard@latest
+```
+
+![WaCrawl Dashboard screenshot](docs/dashboard-2.png)
+
+## Features
+
+### Dashboard
+
+- **Overview cards** — total messages, chats, contacts, media files, and archive date span; each card can jump to Search, Chats, People, or Media as a shortcut.
+- **Time window** — period filter in the top bar (**Day / Week / Month / Year / All**) drives chart stats (default **Year**). The **activity heatmap** uses the current calendar year.
+- **Charts and lists** — message volume over time, top contacts (bar chart; click a contact to open **Chats** filtered to that JID), hour-of-day and day-of-week activity, monthly sent vs received vs ratio, media type breakdown (count and bytes), top media senders, response-time averages, busiest groups, messaging streaks (current and longest).
+- **Word cloud** — top terms with an **All words** vs **Useful words** (stopword-filtered) toggle.
+
+### People
+
+- Contact-style list ranked by activity, with message, media, sent, and received counts and last message time. Privacy-oriented display for LID-style identifiers (JID hidden when appropriate).
+
+### Chats
+
+- **Two-pane browser** — conversation list (with kind: direct vs group, preview line, counts, last activity) and the latest messages for the selected chat.
+- **Read-only inspection** — message bubbles, timestamps, optional **copy message** (including media-aware clipboard text), and inline **images / videos / audio** when paths resolve under your configured media root.
+- **Deep links** — `?contact=<jid>` (for example from the dashboard top-contacts chart) selects the matching chat when present.
+
+### Media
+
+- **Paged grid** of indexed media with **infinite scroll** (intersection-based loading).
+- Thumbnails for **images** and **videos**, affordances for **audio**, and a full-screen **lightbox** for preview and playback where the type supports it.
+
+### Search
+
+- **FTS-backed** full-text search over message content (plus chat and sender names) against the local archive; debounced queries after two or more characters.
+- Result rows show context, optional **image/video thumbnails** and **inline audio**, link detection in snippets, **copy**, and media type badges. Optimized DB (below) is recommended for best performance.
+
+### Settings
+
+- Edit **archive paths** (WhatsApp shared container, primary WaCrawl DB, chat DB, contacts DB, media root) with **save** and **reset to `.env` defaults**.
+- **Health** banner shows whether the primary database opens successfully and the resolved path.
+- Effective paths combine **environment variables** with optional overrides persisted to `~/.wacrawl/dashboard-paths.json` so `<img>` / `<video>` URLs work without custom browser headers.
+
+### App experience
+
+- **Welcome / help** modal on first visit (stored in `localStorage`); reopen from **Help** in the sidebar or top bar.
+- **Dark and light** themes (toggle in the top bar).
+- **Keyboard shortcuts** use the **Meta** key (**⌘** on macOS, **Win** on many Windows keyboards): **Meta+K** opens Search; **Meta+1** Dashboard, **Meta+2** People, **Meta+3** Chats, **Meta+4** Media, **Meta+5** Search, **Meta+6** Settings.
+
+### API and safety
+
+- Listens on **`127.0.0.1`** with host and CORS restricted to local dev/preview origins.
+- Opens the WaCrawl SQLite database in **readonly** mode where applicable; `/api/health` reports DB readability and resolved paths.
+
+## Requirements
+
+- Node.js 20+
+- A WaCrawl archive at `~/.wacrawl/wacrawl.db` (or another path you configure)
+
+## Development
+
+```bash
+npm install
+npm run dev
+```
+
+The API runs on `http://127.0.0.1:3001` and the Vite app runs on `http://localhost:5173`.
+
+Other root scripts:
+
+- **`npm run build`** — build API then web for production.
+- **`npm run preview`** — serve the built web app (Vite preview, localhost).
+- **`npm run test`** — API unit tests (Vitest).
+- **`npm run typecheck`** — TypeScript checks for both workspaces.
+
+## Usage (npx one-liner)
+
+The simplest way to run the dashboard is a single command:
+
+```bash
+npx wacrawl-dashboard@latest
+```
+
+This installs and launches the latest published version directly from npm, no cloning or setup required. Point it at a different WaCrawl archive with the `WACRAWL_DB` environment variable:
+
+```bash
+WACRAWL_DB=/path/to/wacrawl.db npx wacrawl-dashboard@latest
+```
+
+The dashboard opens on `http://127.0.0.1:3001` and requires Node 20+.
+
+## Configuration
+
+Copy `apps/api/.env.example` to `apps/api/.env` (and optionally `apps/web/.env.example` to `apps/web/.env.local`) and edit paths for your WhatsApp container and WaCrawl database.
+
+Supported variables:
+
+- **`WACRAWL_WHATSAPP_CONTAINER`** — folder such as `Library/Group Containers/group.net.whatsapp.WhatsApp.shared`; used to derive default chat, contacts, and media paths when those variables are omitted.
+- **`WACRAWL_CHAT_DB`** / **`WACRAWL_CONTACTS_DB`** — explicit paths to `ChatStorage.sqlite` and `ContactsV2.sqlite`.
+- **`WACRAWL_MEDIA_ROOT`** — folder containing downloaded media (often `…/Message/Media`); relative paths in the archive resolve against this directory.
+- **`WACRAWL_DB`** — primary SQLite file the dashboard queries (WaCrawl export schema from `wacrawl sync`). Defaults to `~/.wacrawl/wacrawl.db`.
+- **`WACRAWL_PATHS_FILE`** (optional) — JSON file for UI-saved path overrides; defaults to `~/.wacrawl/dashboard-paths.json`.
+
+On the web side, **`VITE_API_URL`** (default `http://127.0.0.1:3001`) sets the API base URL if your dev setup differs.
+
+You can also open **Settings** in the sidebar (or **Meta+6**): overrides are saved to `~/.wacrawl/dashboard-paths.json` so the API picks them up without relying on browser-only headers (needed for `<img>` / `<video>` URLs).
+
+To run the API once with env overrides:
+
+```bash
+WACRAWL_DB=/path/to/wacrawl.db npm run dev -w @wacrawl/api
+```
+
+## Search optimization
+
+Prepare the SQLite archive with supporting indexes and an FTS5 search table:
+
+```bash
+npm run optimize-db -w @wacrawl/api
+```
+
+Use `WACRAWL_DB=/path/to/wacrawl.db` with the same command to optimize a non-default archive. Re-run this after `wacrawl sync` so the FTS table includes newly synced messages.

package/apps/api/dist/db.js

@@ -0,0 +1,29 @@
+import Database from 'better-sqlite3';
+import { getResolvedPaths } from './runtimePaths.js';
+let db = null;
+let cachedPrimaryDb = '';
+export function ensurePrimaryDatabase() {
+    const paths = getResolvedPaths();
+    const key = paths.primaryDb;
+    if (db && cachedPrimaryDb === key) {
+        return db;
+    }
+    db?.close();
+    db = new Database(paths.primaryDb, {
+        fileMustExist: true,
+        readonly: true,
+    });
+    cachedPrimaryDb = key;
+    return db;
+}
+export function getDb() {
+    return ensurePrimaryDatabase();
+}
+export function closeDb() {
+    db?.close();
+    db = null;
+    cachedPrimaryDb = '';
+}
+export function getDbPath() {
+    return getResolvedPaths().primaryDb;
+}

package/apps/api/dist/index.js

@@ -0,0 +1,115 @@
+import cors from 'cors';
+import 'dotenv/config';
+import express from 'express';
+import { ensurePrimaryDatabase } from './db.js';
+import { isMediaRootAccessible } from './lib/mediaFsPath.js';
+import { dataRouter } from './routes/data.js';
+import { settingsRouter } from './routes/settings.js';
+import { statsRouter } from './routes/stats.js';
+import { getResolvedPaths, pathsMiddleware } from './runtimePaths.js';
+const PORT = Number(process.env.PORT ?? 3001);
+const HOST = '127.0.0.1';
+const ALLOWED_ORIGINS = new Set([
+    'http://localhost:5173',
+    'http://127.0.0.1:5173',
+    'http://localhost:4173',
+    'http://127.0.0.1:4173',
+    // Also allow same-origin requests when API serves the web frontend directly
+    `http://localhost:${PORT}`,
+    `http://127.0.0.1:${PORT}`,
+]);
+const ALLOWED_HOSTS = new Set([
+    `localhost:${PORT}`,
+    `127.0.0.1:${PORT}`,
+]);
+export function createApp() {
+    const app = express();
+    app.disable('x-powered-by');
+    app.use(express.json({ limit: '64kb' }));
+    app.use((req, res, next) => {
+        const host = req.headers.host;
+        if (host && !ALLOWED_HOSTS.has(host)) {
+            res.status(403).json({
+                error: {
+                    code: 'FORBIDDEN_HOST',
+                    message: 'Requests must target localhost.',
+                },
+            });
+            return;
+        }
+        next();
+    });
+    app.use(cors({
+        origin(origin, callback) {
+            if (!origin || ALLOWED_ORIGINS.has(origin)) {
+                callback(null, true);
+                return;
+            }
+            callback(new Error('Origin is not allowed by CORS.'));
+        },
+    }));
+    app.use(pathsMiddleware);
+    app.get('/api/health', (_req, res) => {
+        const paths = getResolvedPaths();
+        let primaryDbReadable = false;
+        try {
+            ensurePrimaryDatabase();
+            primaryDbReadable = true;
+        }
+        catch {
+            primaryDbReadable = false;
+        }
+        const mediaStatus = isMediaRootAccessible();
+        res.json({
+            ok: primaryDbReadable,
+            dbPath: paths.primaryDb,
+            paths,
+            mediaAccessible: mediaStatus.accessible,
+            mediaError: mediaStatus.error ?? null,
+        });
+    });
+    app.use('/api/settings', settingsRouter);
+    app.use('/api/stats', statsRouter);
+    app.use('/api', dataRouter);
+    // JSON 404 only for /api routes; non-API routes fall through to static serving below
+    app.use('/api', (_req, res) => {
+        res.status(404).json({
+            error: {
+                code: 'NOT_FOUND',
+                message: 'Route not found.',
+            },
+        });
+    });
+    // Production static file serving — set WACRAWL_STATIC_DIR to the web dist folder
+    const staticDir = process.env.WACRAWL_STATIC_DIR;
+    if (staticDir) {
+        app.use(express.static(staticDir));
+        // SPA fallback: unmatched non-API GET requests return index.html
+        app.use((req, res, next) => {
+            if (req.method !== 'GET' || req.path.startsWith('/api')) {
+                next();
+                return;
+            }
+            res.sendFile('index.html', { root: staticDir });
+        });
+    }
+    const errorHandler = (error, _req, res, _next) => {
+        const code = error?.code === 'SQLITE_CANTOPEN' ? 'DATABASE_UNAVAILABLE' : 'INTERNAL_ERROR';
+        const message = code === 'DATABASE_UNAVAILABLE'
+            ? 'Cannot open the WaCrawl database. Check Settings → Archive paths, WACRAWL_DB in .env, or run wacrawl sync.'
+            : 'Unexpected server error.';
+        res.status(code === 'DATABASE_UNAVAILABLE' ? 503 : 500).json({
+            error: {
+                code,
+                message,
+            },
+        });
+    };
+    app.use(errorHandler);
+    return app;
+}
+if (process.env.NODE_ENV !== 'test') {
+    createApp().listen(PORT, HOST, () => {
+        console.info(`WaCrawl API listening on http://${HOST}:${PORT}`);
+    });
+}

package/apps/api/dist/lib/displayName.js

@@ -0,0 +1,28 @@
+export function cleanDisplayNameSql(expression) {
+    return `
+    CASE
+      WHEN ${expression} IS NOT NULL
+        AND TRIM(${expression}) <> ''
+        AND TRIM(${expression}) NOT LIKE '%=%'
+        AND TRIM(${expression}) NOT LIKE '%GIAB%'
+        AND TRIM(${expression}) NOT LIKE '%@lid'
+      THEN TRIM(${expression})
+      ELSE NULL
+    END
+  `;
+}
+export function contactDisplayNameSql(alias) {
+    return `
+    COALESCE(
+      ${cleanDisplayNameSql(`${alias}.full_name`)},
+      ${cleanDisplayNameSql(`${alias}.business_name`)},
+      ${cleanDisplayNameSql(`COALESCE(${alias}.first_name, '') || ' ' || COALESCE(${alias}.last_name, '')`)},
+      ${cleanDisplayNameSql(`${alias}.first_name`)},
+      ${cleanDisplayNameSql(`${alias}.username`)},
+      ${cleanDisplayNameSql(`${alias}.phone`)}
+    )
+  `;
+}
+export function contactMatchSql(alias, jidExpression) {
+    return `(${alias}.jid = ${jidExpression} OR ${alias}.lid = ${jidExpression})`;
+}

package/apps/api/dist/lib/mediaFsPath.js

@@ -0,0 +1,80 @@
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { getResolvedPaths } from '../runtimePaths.js';
+/** Effective media directory (env + dashboard overrides + optional request headers). */
+export function getArchiveMediaRoot() {
+    return getResolvedPaths().mediaRoot;
+}
+/**
+ * WhatsApp Desktop on macOS stores media under
+ *   ~/Library/Group Containers/…/Message/Media/<jid>/…
+ * but some DB rows store the path with just …/Media/<jid>/… (missing the
+ * "Message/" segment). When the stored absolute path doesn't exist, we try
+ * inserting "Message/" before the last "Media/" segment as a fallback.
+ */
+function tryWhatsAppMessageMediaFallback(abspath) {
+    const idx = abspath.lastIndexOf('/Media/');
+    if (idx === -1)
+        return null;
+    const prefix = abspath.slice(0, idx);
+    if (prefix.endsWith('/Message'))
+        return null;
+    const candidate = `${prefix}/Message${abspath.slice(idx)}`;
+    try {
+        fs.accessSync(candidate, fs.constants.R_OK);
+        return candidate;
+    }
+    catch {
+        return null;
+    }
+}
+/**
+ * Turn a stored media_path from the SQLite archive into an absolute filesystem path
+ * suitable for sendFile / fs access. Supports file:// URLs, platform-native separators,
+ * and paths relative to WACRAWL_MEDIA_ROOT or the WaCrawl DB directory.
+ *
+ * When the resolved absolute path doesn't exist, the function checks common WhatsApp
+ * directory layout variations (e.g. …/Message/Media vs …/Media) before giving up.
+ */
+export function resolveArchiveMediaPath(storedPath) {
+    let p = storedPath.trim();
+    if (!p) {
+        throw new Error('Empty media path.');
+    }
+    if (p.startsWith('file:')) {
+        p = fileURLToPath(p);
+    }
+    p = path.normalize(p);
+    const resolved = path.isAbsolute(p) ? p : path.resolve(getArchiveMediaRoot(), p);
+    try {
+        fs.accessSync(resolved, fs.constants.R_OK);
+        return resolved;
+    }
+    catch {
+        // Primary path not readable — try WhatsApp layout fallback.
+    }
+    const fallback = tryWhatsAppMessageMediaFallback(resolved);
+    if (fallback)
+        return fallback;
+    return resolved;
+}
+/** Check whether the configured media root is readable. */
+export function isMediaRootAccessible() {
+    const root = getArchiveMediaRoot();
+    try {
+        fs.accessSync(root, fs.constants.R_OK);
+        return { accessible: true };
+    }
+    catch (err) {
+        const code = err.code;
+        if (code === 'EACCES' || code === 'EPERM') {
+            return {
+                accessible: false,
+                error: 'Permission denied. On macOS, grant Full Disk Access to the app running the API server ' +
+                    '(Terminal, iTerm, etc.) via System Settings \u2192 Privacy & Security \u2192 Full Disk Access.',
+            };
+        }
+        return { accessible: false, error: `Media root is not readable (${code ?? 'unknown error'}).` };
+    }
+}

package/apps/api/dist/lib/query.js

@@ -0,0 +1,71 @@
+import { getUnixTime } from 'date-fns';
+const PERIOD_SECONDS = {
+    day: 86_400,
+    week: 604_800,
+    month: 2_592_000,
+    year: 31_536_000,
+};
+export function parsePeriod(value) {
+    if (value === 'day' || value === 'week' || value === 'month' || value === 'year' || value === 'all') {
+        return value;
+    }
+    return 'all';
+}
+export function parseGranularity(value, fallback) {
+    if (value === 'day' || value === 'week' || value === 'month') {
+        return value;
+    }
+    return fallback;
+}
+export function parseLimit(value, fallback = 10, max = 100) {
+    const parsed = Number(value);
+    if (!Number.isFinite(parsed) || parsed <= 0) {
+        return fallback;
+    }
+    return Math.min(Math.floor(parsed), max);
+}
+export function parseOffset(value) {
+    const parsed = Number(value);
+    if (!Number.isFinite(parsed) || parsed < 0) {
+        return 0;
+    }
+    return Math.floor(parsed);
+}
+export function parseYear(value) {
+    const parsed = Number(value);
+    const currentYear = new Date().getFullYear();
+    if (!Number.isInteger(parsed) || parsed < 1970 || parsed > currentYear + 1) {
+        return currentYear;
+    }
+    return parsed;
+}
+export function sinceTimestamp(period, now = Date.now()) {
+    if (period === 'all') {
+        return 0;
+    }
+    return Math.floor(now / 1000) - PERIOD_SECONDS[period];
+}
+export function unixSecondsToIso(seconds) {
+    if (seconds == null) {
+        return null;
+    }
+    return new Date(seconds * 1000).toISOString();
+}
+export function bucketDateSql(granularity) {
+    if (granularity === 'month') {
+        return "strftime('%Y-%m-01', ts, 'unixepoch')";
+    }
+    if (granularity === 'week') {
+        return "date(ts, 'unixepoch', 'weekday 1', '-7 days')";
+    }
+    return "date(ts, 'unixepoch')";
+}
+export function yearBounds(year) {
+    const start = getUnixTime(new Date(Date.UTC(year, 0, 1)));
+    const end = getUnixTime(new Date(Date.UTC(year + 1, 0, 1)));
+    return { start, end };
+}
+export function normalizeBucketDate(date, granularity) {
+    void granularity;
+    return date;
+}

package/apps/api/dist/lib/searchIndex.js

@@ -0,0 +1,57 @@
+import { cleanDisplayNameSql, contactDisplayNameSql, contactMatchSql } from './displayName.js';
+export const MESSAGE_SEARCH_FTS_TABLE = 'messages_fts';
+export function hasMessageSearchIndex(db) {
+    const row = db.prepare(`
+    SELECT COUNT(*) AS count
+    FROM sqlite_master
+    WHERE type = 'table' AND name = @tableName
+  `).get({ tableName: MESSAGE_SEARCH_FTS_TABLE });
+    return row.count > 0;
+}
+export function createSearchIndexes(db) {
+    db.exec(`
+    CREATE INDEX IF NOT EXISTS idx_messages_chat_jid ON messages(chat_jid);
+    CREATE INDEX IF NOT EXISTS idx_messages_ts ON messages(ts DESC);
+    CREATE INDEX IF NOT EXISTS idx_messages_chat_ts ON messages(chat_jid, ts DESC);
+    CREATE INDEX IF NOT EXISTS idx_contacts_jid ON contacts(jid);
+    CREATE INDEX IF NOT EXISTS idx_contacts_lid ON contacts(lid);
+
+    DROP TABLE IF EXISTS ${MESSAGE_SEARCH_FTS_TABLE};
+    CREATE VIRTUAL TABLE ${MESSAGE_SEARCH_FTS_TABLE} USING fts5(
+      text,
+      chat_name,
+      sender_name,
+      chat_contact_name,
+      sender_contact_name
+    );
+
+    INSERT INTO ${MESSAGE_SEARCH_FTS_TABLE} (
+      rowid,
+      text,
+      chat_name,
+      sender_name,
+      chat_contact_name,
+      sender_contact_name
+    )
+    SELECT
+      messages.rowid,
+      COALESCE(messages.text, ''),
+      COALESCE(
+        ${cleanDisplayNameSql('chats.name')},
+        ${cleanDisplayNameSql('messages.chat_name')},
+        ${cleanDisplayNameSql('messages.chat_jid')},
+        ''
+      ),
+      COALESCE(
+        ${cleanDisplayNameSql('messages.sender_name')},
+        ${cleanDisplayNameSql('messages.sender_jid')},
+        ''
+      ),
+      COALESCE(${contactDisplayNameSql('chat_contacts')}, ''),
+      COALESCE(${contactDisplayNameSql('sender_contacts')}, '')
+    FROM messages
+    LEFT JOIN chats ON chats.jid = messages.chat_jid
+    LEFT JOIN contacts AS chat_contacts ON ${contactMatchSql('chat_contacts', 'messages.chat_jid')}
+    LEFT JOIN contacts AS sender_contacts ON ${contactMatchSql('sender_contacts', 'messages.sender_jid')};
+  `);
+}

package/apps/api/dist/lib/timezone.js

@@ -0,0 +1,38 @@
+/**
+ * First string value from Express queryparams (handles string | string[]).
+ */
+export function firstQueryString(value) {
+    if (typeof value === 'string' && value.trim())
+        return value.trim();
+    if (Array.isArray(value) && typeof value[0] === 'string' && value[0].trim())
+        return value[0].trim();
+    return undefined;
+}
+/**
+ * IANA timezone for stats bucketing when the client does not send one.
+ * UTC matches the legacy SQL strftime('%H', ts, 'unixepoch') behavior.
+ */
+export function resolveStatsTimeZone(raw) {
+    const candidate = firstQueryString(raw);
+    if (!candidate)
+        return 'UTC';
+    try {
+        Intl.DateTimeFormat('en-US', { timeZone: candidate }).format(new Date());
+        return candidate;
+    }
+    catch {
+        return 'UTC';
+    }
+}
+/** Calendar hour 0–23 for Unix epoch seconds in the given IANA time zone. */
+export function hourInTimezone(epochSeconds, timeZone) {
+    const parts = new Intl.DateTimeFormat('en-GB', {
+        timeZone,
+        hour: 'numeric',
+        hourCycle: 'h23',
+    }).formatToParts(new Date(epochSeconds * 1000));
+    const value = Number(parts.find((p) => p.type === 'hour')?.value);
+    if (!Number.isFinite(value))
+        return 0;
+    return Math.min(23, Math.max(0, value));
+}

package/apps/api/dist/lib/wordCloudUsefulStopWords.js

@@ -0,0 +1,118 @@
+import { eng } from 'stopword';
+/** Messaging / link / attachment noise beyond standard English stopwords. */
+const CHAT_TECH_URL_EXTRA = [
+    'apk',
+    'asp',
+    'au',
+    'auth',
+    'be',
+    'bin',
+    'bmp',
+    'br',
+    'ca',
+    'caption',
+    'captioned',
+    'captioning',
+    'cn',
+    'co',
+    'com',
+    'de',
+    'doc',
+    'docx',
+    'dmg',
+    'epub',
+    'exe',
+    'facebook',
+    'fr',
+    'ftp',
+    'gif',
+    'gmail',
+    'google',
+    'htm',
+    'html',
+    'https',
+    'http',
+    'img',
+    'instagram',
+    'io',
+    'it',
+    'jp',
+    'jpeg',
+    'jpg',
+    'mobi',
+    'mov',
+    'mp3',
+    'mp4',
+    'msg',
+    'net',
+    'nl',
+    'omit',
+    'omitted',
+    'org',
+    'pdf',
+    'php',
+    'png',
+    'ppt',
+    'pptx',
+    'rar',
+    'telegram',
+    'tif',
+    'tiff',
+    'ttf',
+    'twitter',
+    'txt',
+    'uk',
+    'us',
+    'vosotras',
+    'vosotros',
+    'wav',
+    'webp',
+    'webm',
+    'webpage',
+    'website',
+    'whatsapp',
+    'www',
+    'xls',
+    'xlsx',
+    'youtu',
+    'youtube',
+    'zip',
+];
+/** Casual chat fillers not in `eng`. */
+const CHAT_FILLER_EXTRA = ['btw', 'hello', 'hey', 'hi', 'hiya', 'hmm', 'idk', 'imo', 'lmao', 'lol', 'nope', 'okay', 'omg', 'pls', 'please', 'smh', 'thanx', 'thanks', 'thx', 'yep', 'yeah', 'yo'];
+/** Porter “English” list gaps and other very common tokens (>=3 chars for our tokenizer). */
+const EXTRA_EN_STOP = [
+    'anything',
+    'arent',
+    'cant',
+    'couldnt',
+    'didnt',
+    'does',
+    'doesnt',
+    'done',
+    'dont',
+    'every',
+    'everything',
+    'hadnt',
+    'havent',
+    'isnt',
+    'lets',
+    'nothing',
+    'not',
+    'nor',
+    'no',
+    'shall',
+    'shouldnt',
+    'something',
+    'wasnt',
+    'werent',
+    'will',
+    'wont',
+    'wouldnt',
+    'yes',
+    'yet',
+];
+/**
+ * Tokens dropped in "useful" word-cloud mode (lowercase; message tokens are `[a-z0-9]{3,}`).
+ */
+export const USEFUL_WORD_STOP_SET = new Set([...eng, ...CHAT_TECH_URL_EXTRA, ...CHAT_FILLER_EXTRA, ...EXTRA_EN_STOP]);