wacrawl-dashboard 0.1.1 → 0.1.2

package/README.md

@@ -1,6 +1,6 @@
 # WaCrawl Dashboard
 
-Local WhatsApp analytics dashboard for the SQLite archive generated by `wacrawl`. A small monorepo runs a **read-only Express API** (`apps/api`) and a **React + Vite** UI (`apps/web`) that talks to it over CORS on localhost only.
+Local WhatsApp analytics dashboard for the SQLite archive generated by [wacrawl](https://github.com/steipete/wacrawl). A small monorepo runs a **read-only Express API** (`apps/api`) and a **React + Vite** UI (`apps/web`) that talks to it over CORS on localhost only.
 
 ## One-line install
 
@@ -24,7 +24,15 @@ WACRAWL_DB=/path/to/wacrawl.db npx wacrawl-dashboard@latest
 - **Overview cards** — total messages, chats, contacts, media files, and archive date span; each card can jump to Search, Chats, People, or Media as a shortcut.
 - **Time window** — period filter in the top bar (**Day / Week / Month / Year / All**) drives chart stats (default **Year**). The **activity heatmap** uses the current calendar year.
 - **Charts and lists** — message volume over time, top contacts (bar chart; click a contact to open **Chats** filtered to that JID), hour-of-day and day-of-week activity, monthly sent vs received vs ratio, media type breakdown (count and bytes), top media senders, response-time averages, busiest groups, messaging streaks (current and longest).
-- **Word cloud** — top terms with an **All words** vs **Useful words** (stopword-filtered) toggle.
+- **Word cloud** — top terms with an **All words** vs **Useful words** (stopword-filtered) toggle; click any term to search your archive for it.
+- **Emoji analytics** — total and unique emoji counts, top emojis by **All / Sent / Received**, and a leaderboard of top emoji users with their favourite emoji.
+- **Conversation dynamics** — a dedicated section with relationship-level insights:
+  - **Who starts conversations** — initiation ratio per contact with a visual split bar (sessions separated by 4h+ gaps).
+  - **Conversation depth** — average messages per session, showing which chats go deep.
+  - **Ghost score** — percentage of your messages that went unanswered within 24 hours, colour-coded by severity.
+  - **Late-night texters** — who you talk to between midnight and 5 am, with hour-bucket breakdowns.
+  - **Relationship trajectory** — monthly sparkline per contact showing whether message frequency is *growing*, *fading*, or *stable*.
+- **Loading overlay** — an animated full-screen splash with rotating quotes while the dashboard fetches data.
 
 ### People
 
@@ -36,10 +44,17 @@ WACRAWL_DB=/path/to/wacrawl.db npx wacrawl-dashboard@latest
 - **Read-only inspection** — message bubbles, timestamps, optional **copy message** (including media-aware clipboard text), and inline **images / videos / audio** when paths resolve under your configured media root.
 - **Deep links** — `?contact=<jid>` (for example from the dashboard top-contacts chart) selects the matching chat when present.
 
-### Media
+### Media & Links
 
-- **Paged grid** of indexed media with **infinite scroll** (intersection-based loading).
-- Thumbnails for **images** and **videos**, affordances for **audio**, and a full-screen **lightbox** for preview and playback where the type supports it.
+- **Two-tab layout** — switch between **Media Grid** and **Link Intelligence** from the page header.
+- **Media Grid** — paged grid of indexed media with **infinite scroll** (intersection-based loading). Thumbnails for **images** and **videos**, affordances for **audio**, and a full-screen **lightbox** for preview and playback where the type supports it.
+- **Link Intelligence** (new) — a rich analytics panel for URLs shared in your conversations:
+  - **Overview pills** — total links shared and unique domains at a glance.
+  - **Domain leaderboard** — horizontal bar chart of the most-shared domains, colour-coded by category.
+  - **Content categories** — pie chart breaking links into *Video, Social, News, Shopping, Music, Developer, Reference,* and *Other*, with the top domain per category.
+  - **Media timeline density** — stacked area chart showing daily counts of images, videos, audio files, and links over time.
+  - **Sharing asymmetry** — per-contact breakdown of media and link volume in each direction (sent vs received).
+  - **First shared** — a timeline of the earliest message and earliest media item for each conversation.
 
 ### Search
 
@@ -62,6 +77,7 @@ WACRAWL_DB=/path/to/wacrawl.db npx wacrawl-dashboard@latest
 
 - Listens on **`127.0.0.1`** with host and CORS restricted to local dev/preview origins.
 - Opens the WaCrawl SQLite database in **readonly** mode where applicable; `/api/health` reports DB readability and resolved paths.
+- **Stats endpoints** — `/api/stats/emoji-analytics`, `/api/stats/conversation-dynamics`, and `/api/stats/link-intelligence` power the new dashboard and media analytics panels.
 
 ## Requirements
 

package/apps/api/dist/lib/displayName.js

@@ -11,7 +11,7 @@ export function cleanDisplayNameSql(expression) {
     END
   `;
 }
-export function contactDisplayNameSql(alias) {
+export function contactDisplayNameSingleSql(alias) {
     return `
     COALESCE(
       ${cleanDisplayNameSql(`${alias}.full_name`)},
@@ -23,6 +23,20 @@ export function contactDisplayNameSql(alias) {
     )
   `;
 }
-export function contactMatchSql(alias, jidExpression) {
-    return `(${alias}.jid = ${jidExpression} OR ${alias}.lid = ${jidExpression})`;
+export function contactDisplayNameSql(alias, isSplit = true) {
+    if (!isSplit) {
+        return contactDisplayNameSingleSql(alias);
+    }
+    return `
+    COALESCE(
+      ${contactDisplayNameSingleSql(`${alias}_jid`)},
+      ${contactDisplayNameSingleSql(`${alias}_lid`)}
+    )
+  `;
+}
+export function contactLeftJoins(alias, jidExpression) {
+    return `
+    LEFT JOIN contacts AS ${alias}_jid ON ${alias}_jid.jid = ${jidExpression}
+    LEFT JOIN contacts AS ${alias}_lid ON ${alias}_lid.lid = ${jidExpression}
+  `;
 }

package/apps/api/dist/lib/searchIndex.js

@@ -1,4 +1,4 @@
-import { cleanDisplayNameSql, contactDisplayNameSql, contactMatchSql } from './displayName.js';
+import { cleanDisplayNameSql, contactDisplayNameSql, contactLeftJoins } from './displayName.js';
 export const MESSAGE_SEARCH_FTS_TABLE = 'messages_fts';
 export function hasMessageSearchIndex(db) {
     const row = db.prepare(`
@@ -51,7 +51,7 @@ export function createSearchIndexes(db) {
       COALESCE(${contactDisplayNameSql('sender_contacts')}, '')
     FROM messages
     LEFT JOIN chats ON chats.jid = messages.chat_jid
-    LEFT JOIN contacts AS chat_contacts ON ${contactMatchSql('chat_contacts', 'messages.chat_jid')}
-    LEFT JOIN contacts AS sender_contacts ON ${contactMatchSql('sender_contacts', 'messages.sender_jid')};
+    ${contactLeftJoins('chat_contacts', 'messages.chat_jid')}
+    ${contactLeftJoins('sender_contacts', 'messages.sender_jid')};
   `);
 }

package/apps/api/dist/lib/timezone.js

@@ -24,13 +24,26 @@ export function resolveStatsTimeZone(raw) {
         return 'UTC';
     }
 }
+const formattersCache = new Map();
+function getFormatter(timeZone) {
+    let formatter = formattersCache.get(timeZone);
+    if (!formatter) {
+        formatter = new Intl.DateTimeFormat('en-GB', {
+            timeZone,
+            hour: 'numeric',
+            hourCycle: 'h23',
+        });
+        formattersCache.set(timeZone, formatter);
+    }
+    return formatter;
+}
 /** Calendar hour 0–23 for Unix epoch seconds in the given IANA time zone. */
 export function hourInTimezone(epochSeconds, timeZone) {
-    const parts = new Intl.DateTimeFormat('en-GB', {
-        timeZone,
-        hour: 'numeric',
-        hourCycle: 'h23',
-    }).formatToParts(new Date(epochSeconds * 1000));
+    if (timeZone === 'UTC') {
+        return new Date(epochSeconds * 1000).getUTCHours();
+    }
+    const formatter = getFormatter(timeZone);
+    const parts = formatter.formatToParts(new Date(epochSeconds * 1000));
     const value = Number(parts.find((p) => p.type === 'hour')?.value);
     if (!Number.isFinite(value))
         return 0;

package/apps/api/dist/lib/urlExtract.js

@@ -0,0 +1,111 @@
+const URL_RE = /https?:\/\/[^\s<>"')\]]+/gi;
+const TRAILING_PUNCT_RE = /[.,;:!?)}\]]+$/;
+const DOMAIN_CATEGORIES = {
+    'youtube.com': 'video',
+    'youtu.be': 'video',
+    'vimeo.com': 'video',
+    'dailymotion.com': 'video',
+    'twitch.tv': 'video',
+    'tiktok.com': 'video',
+    'vm.tiktok.com': 'video',
+    'twitter.com': 'social',
+    'x.com': 'social',
+    'facebook.com': 'social',
+    'fb.com': 'social',
+    'instagram.com': 'social',
+    'linkedin.com': 'social',
+    'reddit.com': 'social',
+    'threads.net': 'social',
+    'mastodon.social': 'social',
+    'bsky.app': 'social',
+    'pinterest.com': 'social',
+    'snapchat.com': 'social',
+    'tumblr.com': 'social',
+    'wa.me': 'social',
+    'cnn.com': 'news',
+    'bbc.com': 'news',
+    'bbc.co.uk': 'news',
+    'nytimes.com': 'news',
+    'theguardian.com': 'news',
+    'reuters.com': 'news',
+    'apnews.com': 'news',
+    'washingtonpost.com': 'news',
+    'aljazeera.com': 'news',
+    'news.google.com': 'news',
+    'news.yahoo.com': 'news',
+    'techcrunch.com': 'news',
+    'theverge.com': 'news',
+    'arstechnica.com': 'news',
+    'wired.com': 'news',
+    'medium.com': 'news',
+    'substack.com': 'news',
+    'bloomberg.com': 'news',
+    'cnbc.com': 'news',
+    'foxnews.com': 'news',
+    'nbcnews.com': 'news',
+    'ynet.co.il': 'news',
+    'haaretz.com': 'news',
+    'amazon.com': 'shopping',
+    'amazon.co.uk': 'shopping',
+    'amazon.de': 'shopping',
+    'ebay.com': 'shopping',
+    'etsy.com': 'shopping',
+    'aliexpress.com': 'shopping',
+    'walmart.com': 'shopping',
+    'target.com': 'shopping',
+    'shopify.com': 'shopping',
+    'wish.com': 'shopping',
+    'spotify.com': 'music',
+    'open.spotify.com': 'music',
+    'soundcloud.com': 'music',
+    'music.apple.com': 'music',
+    'music.youtube.com': 'music',
+    'deezer.com': 'music',
+    'bandcamp.com': 'music',
+    'github.com': 'dev',
+    'gitlab.com': 'dev',
+    'stackoverflow.com': 'dev',
+    'npmjs.com': 'dev',
+    'pypi.org': 'dev',
+    'dev.to': 'dev',
+    'codepen.io': 'dev',
+    'jsfiddle.net': 'dev',
+    'codesandbox.io': 'dev',
+    'hackernews.com': 'dev',
+    'news.ycombinator.com': 'dev',
+    'wikipedia.org': 'reference',
+    'en.wikipedia.org': 'reference',
+    'docs.google.com': 'reference',
+    'drive.google.com': 'reference',
+    'maps.google.com': 'reference',
+    'translate.google.com': 'reference',
+    'google.com': 'reference',
+    'bing.com': 'reference',
+    'duckduckgo.com': 'reference',
+};
+export function extractUrls(text) {
+    if (!text)
+        return [];
+    const matches = text.match(URL_RE);
+    if (!matches)
+        return [];
+    return matches.map((url) => url.replace(TRAILING_PUNCT_RE, ''));
+}
+export function parseDomain(url) {
+    try {
+        const hostname = new URL(url).hostname.toLowerCase();
+        return hostname.replace(/^www\./, '');
+    }
+    catch {
+        return null;
+    }
+}
+export function categorizeDomain(domain) {
+    if (DOMAIN_CATEGORIES[domain])
+        return DOMAIN_CATEGORIES[domain];
+    for (const [known, category] of Object.entries(DOMAIN_CATEGORIES)) {
+        if (domain.endsWith(`.${known}`))
+            return category;
+    }
+    return 'other';
+}

package/apps/api/dist/routes/data.js

@@ -1,6 +1,6 @@
 import { Router } from 'express';
 import { getDb } from '../db.js';
-import { cleanDisplayNameSql, contactDisplayNameSql, contactMatchSql } from '../lib/displayName.js';
+import { cleanDisplayNameSql, contactDisplayNameSql, contactLeftJoins } from '../lib/displayName.js';
 import { resolveArchiveMediaPath } from '../lib/mediaFsPath.js';
 import { parseLimit, parseOffset, unixSecondsToIso } from '../lib/query.js';
 import { hasMessageSearchIndex, MESSAGE_SEARCH_FTS_TABLE } from '../lib/searchIndex.js';
@@ -85,10 +85,10 @@ export function getPeople(params, db = getDb()) {
         media_type
       FROM messages
       LEFT JOIN chats ON chats.jid = CASE WHEN messages.from_me = 1 THEN messages.chat_jid ELSE COALESCE(messages.sender_jid, messages.chat_jid) END
-      LEFT JOIN contacts ON ${contactMatchSql('contacts', 'CASE WHEN messages.from_me = 1 THEN messages.chat_jid ELSE COALESCE(messages.sender_jid, messages.chat_jid) END')}
+      ${contactLeftJoins('contacts', 'CASE WHEN messages.from_me = 1 THEN messages.chat_jid ELSE COALESCE(messages.sender_jid, messages.chat_jid) END')}
     ),
     all_people AS (
-      SELECT jid, ${contactDisplayNameSql('contacts')} AS contact_name, NULL AS message_name, NULL AS direct_chat_name, NULL AS from_me, NULL AS ts, NULL AS media_type
+      SELECT jid, ${contactDisplayNameSql('contacts', false)} AS contact_name, NULL AS message_name, NULL AS direct_chat_name, NULL AS from_me, NULL AS ts, NULL AS media_type
       FROM contacts
       UNION ALL
       SELECT jid, contact_name, message_name, direct_chat_name, from_me, ts, media_type FROM normalized_messages
@@ -121,8 +121,12 @@ export function getChats(params, db = getDb()) {
     const limit = parseLimit(params.limit, 50, 200);
     const offset = parseOffset(params.offset);
     const kind = params.kind === 'direct' || params.kind === 'group' ? params.kind : undefined;
-    const where = kind ? 'WHERE chats.kind = @kind' : 'WHERE chats.kind IN (\'direct\', \'group\')';
-    const total = db.prepare(`SELECT COUNT(*) AS count FROM chats ${where}`).get({ kind }).count;
+    const targetJid = typeof params.jid === 'string' && params.jid.trim() ? params.jid.trim() : undefined;
+    let where = kind ? 'WHERE chats.kind = @kind' : 'WHERE chats.kind IN (\'direct\', \'group\')';
+    if (targetJid) {
+        where = 'WHERE chats.jid = @targetJid';
+    }
+    const total = db.prepare(`SELECT COUNT(*) AS count FROM chats ${where}`).get({ kind, targetJid }).count;
     const rows = db.prepare(`
     WITH chat_page AS (
       SELECT
@@ -172,9 +176,9 @@ export function getChats(params, db = getDb()) {
     FROM chat_page
     LEFT JOIN message_stats ON message_stats.chat_jid = chat_page.jid
     LEFT JOIN latest ON latest.chat_jid = chat_page.jid
-    LEFT JOIN contacts AS chat_contacts ON ${contactMatchSql('chat_contacts', 'chat_page.jid')}
+    ${contactLeftJoins('chat_contacts', 'chat_page.jid')}
     ORDER BY COALESCE(message_stats.lastMessageTs, chat_page.last_message_at, 0) DESC, name COLLATE NOCASE ASC
-  `).all({ kind, limit, offset });
+  `).all({ kind, limit, offset, targetJid });
     return asPagination(rows.map((row) => ({
         jid: row.jid,
         kind: row.kind,
@@ -207,8 +211,8 @@ export function getChatMessages(chatJid, params, db = getDb()) {
       media_path AS mediaPath
     FROM messages
     LEFT JOIN chats ON chats.jid = messages.chat_jid
-    LEFT JOIN contacts AS chat_contacts ON ${contactMatchSql('chat_contacts', 'messages.chat_jid')}
-    LEFT JOIN contacts AS sender_contacts ON ${contactMatchSql('sender_contacts', 'messages.sender_jid')}
+    ${contactLeftJoins('chat_contacts', 'messages.chat_jid')}
+    ${contactLeftJoins('sender_contacts', 'messages.sender_jid')}
     WHERE chat_jid = @chatJid
     ORDER BY ts DESC, messages.rowid DESC
     LIMIT @limit OFFSET @offset
@@ -240,8 +244,8 @@ export function getMediaItems(params, db = getDb()) {
       media_size AS mediaSize
     FROM messages
     LEFT JOIN chats ON chats.jid = messages.chat_jid
-    LEFT JOIN contacts AS chat_contacts ON ${contactMatchSql('chat_contacts', 'messages.chat_jid')}
-    LEFT JOIN contacts AS sender_contacts ON ${contactMatchSql('sender_contacts', 'messages.sender_jid')}
+    ${contactLeftJoins('chat_contacts', 'messages.chat_jid')}
+    ${contactLeftJoins('sender_contacts', 'messages.sender_jid')}
     WHERE ${where}
     ORDER BY ts DESC, messages.rowid DESC
     LIMIT @limit OFFSET @offset
@@ -301,8 +305,8 @@ export function searchMessages(params, db = getDb()) {
         FROM matched
         JOIN messages ON messages.rowid = matched.rowid
         LEFT JOIN chats ON chats.jid = messages.chat_jid
-        LEFT JOIN contacts AS chat_contacts ON ${contactMatchSql('chat_contacts', 'messages.chat_jid')}
-        LEFT JOIN contacts AS sender_contacts ON ${contactMatchSql('sender_contacts', 'messages.sender_jid')}
+        ${contactLeftJoins('chat_contacts', 'messages.chat_jid')}
+        ${contactLeftJoins('sender_contacts', 'messages.sender_jid')}
         ORDER BY ts DESC, messages.rowid DESC
         LIMIT @limit OFFSET @offset
       `).all({ ftsQuery, limit, offset });
@@ -325,8 +329,8 @@ export function searchMessages(params, db = getDb()) {
     SELECT COUNT(*) AS count
     FROM messages
     LEFT JOIN chats ON chats.jid = messages.chat_jid
-    LEFT JOIN contacts AS chat_contacts ON ${contactMatchSql('chat_contacts', 'messages.chat_jid')}
-    LEFT JOIN contacts AS sender_contacts ON ${contactMatchSql('sender_contacts', 'messages.sender_jid')}
+    ${contactLeftJoins('chat_contacts', 'messages.chat_jid')}
+    ${contactLeftJoins('sender_contacts', 'messages.sender_jid')}
     WHERE ${where}
   `).get({ like }).count;
     const rows = db.prepare(`
@@ -345,8 +349,8 @@ export function searchMessages(params, db = getDb()) {
       media_path AS mediaPath
     FROM messages
     LEFT JOIN chats ON chats.jid = messages.chat_jid
-    LEFT JOIN contacts AS chat_contacts ON ${contactMatchSql('chat_contacts', 'messages.chat_jid')}
-    LEFT JOIN contacts AS sender_contacts ON ${contactMatchSql('sender_contacts', 'messages.sender_jid')}
+    ${contactLeftJoins('chat_contacts', 'messages.chat_jid')}
+    ${contactLeftJoins('sender_contacts', 'messages.sender_jid')}
     WHERE ${where}
     ORDER BY ts DESC, messages.rowid DESC
     LIMIT @limit OFFSET @offset
@@ -415,3 +419,25 @@ dataRouter.get('/media/file', (req, res, next) => {
 dataRouter.get('/search', (req, res) => {
     res.json(searchMessages(req.query));
 });
+dataRouter.get('/messages/:id/offset', (req, res) => {
+    const id = Number(req.params.id);
+    if (isNaN(id)) {
+        res.status(400).json({ error: 'Invalid message ID' });
+        return;
+    }
+    const db = getDb();
+    const msg = db.prepare('SELECT chat_jid, ts FROM messages WHERE rowid = ?').get(id);
+    if (!msg) {
+        res.status(404).json({ error: 'Message not found' });
+        return;
+    }
+    const countRow = db.prepare(`
+    SELECT COUNT(*) AS count
+    FROM messages
+    WHERE chat_jid = ? AND (ts > ? OR (ts = ? AND rowid > ?))
+  `).get(msg.chat_jid, msg.ts, msg.ts, id);
+    res.json({
+        chatJid: msg.chat_jid,
+        offset: countRow.count
+    });
+});