@rmdes/indiekit-endpoint-webmention-io 1.0.3 → 1.0.5

package/README.md

@@ -0,0 +1,270 @@
+# @rmdes/indiekit-endpoint-webmention-io
+
+Webmention moderation endpoint for [Indiekit](https://getindiekit.com). Syncs webmentions from webmention.io into MongoDB with delete, block, and privacy removal capabilities.
+
+## Features
+
+- **Background Sync**: Automatically fetches webmentions from webmention.io every 15 minutes (configurable)
+- **Moderation Dashboard**: Admin UI for hiding/unhiding webmentions
+- **Domain Blocking**: Block spam domains (hides all mentions, blocks future ones)
+- **Privacy Removal**: GDPR-compliant permanent deletion with domain blocking
+- **Public JSON API**: Drop-in replacement for webmention.io API with server-side caching
+- **MongoDB Storage**: Persistent storage with indexes for fast queries
+- **Incremental Sync**: Only fetches new webmentions since last sync (efficient)
+- **Full Re-sync**: Option to clear and re-import all webmentions
+
+## Installation
+
+```bash
+npm install @rmdes/indiekit-endpoint-webmention-io
+```
+
+## Configuration
+
+```javascript
+// indiekit.config.js
+export default {
+  plugins: [
+    "@rmdes/indiekit-endpoint-webmention-io",
+  ],
+
+  "@rmdes/indiekit-endpoint-webmention-io": {
+    mountPath: "/webmentions",            // Optional, default "/webmentions"
+    token: process.env.WEBMENTION_IO_TOKEN,  // REQUIRED: webmention.io API token
+    domain: "example.com",                // REQUIRED: domain to fetch webmentions for
+    syncInterval: 900_000,                // Optional, default 15 minutes (in ms)
+    cacheTtl: 60,                         // Optional, default 60 seconds (public API cache)
+  },
+};
+```
+
+### Getting Your Webmention.io Token
+
+1. Sign up at [webmention.io](https://webmention.io)
+2. Add your domain
+3. Find your token in the dashboard
+4. Store it in your `.env` file:
+   ```bash
+   WEBMENTION_IO_TOKEN=your_token_here
+   ```
+
+## Usage
+
+### Admin Dashboard
+
+Visit `/webmentions` in your Indiekit admin panel to:
+
+- View all webmentions (paginated)
+- Filter by visibility (all/visible/hidden)
+- Filter by type (likes/replies/reposts/mentions)
+- Hide/unhide individual webmentions
+- Block spam domains
+- Remove mentions for privacy requests (GDPR)
+
+### Manual Sync
+
+Trigger sync via admin dashboard buttons:
+- **Sync Now**: Incremental sync (fetch only new mentions since last sync)
+- **Full Re-sync**: Delete all and re-import everything from webmention.io
+
+Or via POST requests:
+```bash
+# Incremental sync
+curl -X POST https://your-site.com/webmentions/sync
+
+# Full re-sync (destructive!)
+curl -X POST https://your-site.com/webmentions/sync/full
+```
+
+### Public JSON API
+
+The plugin exposes a public JSON API at `/webmentions/api/mentions` that can replace direct calls to webmention.io:
+
+**Fetch all webmentions:**
+```javascript
+fetch('/webmentions/api/mentions?page=0&per-page=50')
+```
+
+**Filter by target URL:**
+```javascript
+fetch('/webmentions/api/mentions?target=https://example.com/post')
+```
+
+**Filter by type:**
+```javascript
+fetch('/webmentions/api/mentions?wm-property=like-of')
+```
+
+**Response format (JF2):**
+```json
+{
+  "type": "feed",
+  "name": "Webmentions",
+  "children": [
+    {
+      "type": "entry",
+      "wm-id": 12345,
+      "wm-received": "2025-02-13T10:00:00.000Z",
+      "wm-property": "in-reply-to",
+      "wm-target": "https://example.com/post",
+      "author": {
+        "type": "card",
+        "name": "Author Name",
+        "url": "https://author.site/",
+        "photo": "https://author.site/photo.jpg"
+      },
+      "url": "https://source.site/post",
+      "published": "2025-02-13T09:00:00.000Z",
+      "content": {
+        "html": "<p>Reply text...</p>",
+        "text": "Reply text..."
+      }
+    }
+  ]
+}
+```
+
+### Moderation Workflows
+
+#### Hide a webmention
+```bash
+POST /webmentions/:wmId/hide
+```
+Marks the webmention as hidden (won't appear in public API).
+
+#### Unhide a webmention
+```bash
+POST /webmentions/:wmId/unhide
+```
+Restores a hidden webmention.
+
+#### Block a domain
+```bash
+POST /webmentions/block
+Body: domain=spam.example.com
+```
+- Hides all existing mentions from the domain
+- Adds domain to blocklist
+- Future mentions from this domain are filtered during sync
+
+#### Privacy removal (GDPR)
+```bash
+POST /webmentions/privacy-remove
+Body: domain=user-request.example.com
+```
+- **Permanently deletes** all mentions from the domain
+- Adds domain to blocklist with reason="privacy"
+- Irreversible - use for GDPR/privacy requests only
+
+#### Unblock a domain
+```bash
+POST /webmentions/blocklist/:domain/delete
+```
+- Removes domain from blocklist
+- Unhides mentions that were hidden by the blocklist (not manual or privacy)
+
+## API Reference
+
+### Query Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `page` | number | Page number (0-indexed, default: 0) |
+| `per-page` | number | Items per page (max 10,000, default: 50) |
+| `target` | string | Filter by target URL (with/without trailing slash) |
+| `wm-property` | string | Filter by type: `in-reply-to`, `like-of`, `repost-of`, `mention-of`, `bookmark-of`, `rsvp` |
+
+### Webmention Types
+
+- `in-reply-to` - Replies
+- `like-of` - Likes
+- `repost-of` - Reposts/boosts
+- `mention-of` - General mentions
+- `bookmark-of` - Bookmarks
+- `rsvp` - RSVP responses
+
+## MongoDB Schema
+
+The plugin creates two MongoDB collections:
+
+### `webmentions`
+
+```javascript
+{
+  wmId: 12345,                     // Webmention ID (unique)
+  wmReceived: "2025-02-13T10:00:00.000Z",
+  wmProperty: "in-reply-to",
+  wmTarget: "https://example.com/post",
+  authorName: "Author Name",
+  authorUrl: "https://author.site/",
+  authorPhoto: "https://author.site/photo.jpg",
+  sourceUrl: "https://source.site/post",
+  sourceDomain: "source.site",
+  published: "2025-02-13T09:00:00.000Z",
+  contentHtml: "<p>Reply text...</p>",
+  contentText: "Reply text...",
+  name: "Post title",
+  hidden: false,
+  hiddenAt: null,
+  hiddenReason: null,  // "manual", "blocklist", "privacy"
+  syncedAt: "2025-02-13T10:00:00.000Z",
+  raw: { ... }  // Original JF2 entry
+}
+```
+
+### `webmentionBlocklist`
+
+```javascript
+{
+  domain: "spam.example.com",
+  reason: "spam",  // "spam", "privacy", "manual"
+  blockedAt: "2025-02-13T10:00:00.000Z",
+  mentionsHidden: 5
+}
+```
+
+## How It Works
+
+1. **Background Sync**: Runs every 15 minutes (configurable)
+2. **Incremental Fetching**: Uses `since_id` to only fetch new mentions
+3. **Blocklist Filtering**: Mentions from blocked domains are never stored
+4. **Pagination**: Fetches 100 mentions per page from webmention.io
+5. **Rate Limiting**: 500ms delay between pages to avoid rate limits
+6. **Caching**: Public API responses cached for 60 seconds (configurable)
+
+## HTML Sanitization
+
+All webmention HTML content is sanitized:
+- Strips empty bridgy links
+- Strips empty paragraphs
+- Downgrades heading levels (h1→h3, h2→h4)
+- Normalizes line breaks to paragraph breaks
+
+## Comparison with Other Plugins
+
+| Plugin | Purpose | Storage | API |
+|--------|---------|---------|-----|
+| **@rmdes/indiekit-endpoint-webmention-io** | Full moderation + public API | MongoDB | JF2 JSON |
+| `@rmdes/indiekit-endpoint-webmentions-proxy` | Simple proxy (no moderation) | None | JF2 JSON |
+| `@indiekit/endpoint-webmention` (upstream) | Admin dashboard only | None | HTML only |
+
+Use this plugin if you need:
+- Moderation capabilities
+- Domain blocking
+- Privacy removal (GDPR)
+- Public API with caching
+- Persistent storage
+
+Use `@rmdes/indiekit-endpoint-webmentions-proxy` if you only need a simple public API without moderation.
+
+## License
+
+MIT
+
+## Author
+
+Ricardo Mendes - [rmendes.net](https://rmendes.net)
+
+## Repository
+
+[https://github.com/rmdes/indiekit-endpoint-webmention-io](https://github.com/rmdes/indiekit-endpoint-webmention-io)

package/lib/controllers/blocklist.js

@@ -4,6 +4,7 @@
 
 import { getBlocklist, unblockDomain } from "../storage/blocklist.js";
 import { unhideByDomain } from "../storage/webmentions.js";
+import { ensureISOString } from "../utils.js";
 
 export const blocklistController = {
   /**
@@ -19,7 +20,10 @@ export const blocklistController = {
       if (db) {
         const collection = db.collection("webmentionBlocklist");
         const raw = await getBlocklist(collection);
-        entries = raw;
+        entries = raw.map((entry) => ({
+          ...entry,
+          blockedAt: ensureISOString(entry.blockedAt),
+        }));
       }
 
       response.render("webmentions-blocklist", {

package/lib/controllers/dashboard.js

@@ -13,7 +13,7 @@ import {
 } from "../storage/webmentions.js";
 import { blockDomain } from "../storage/blocklist.js";
 import { getSyncState } from "../sync.js";
-import { getMentionType, getMentionTitle, getAuthorName } from "../utils.js";
+import { getMentionType, getMentionTitle, getAuthorName, ensureISOString } from "../utils.js";
 
 export const dashboardController = {
   /**
@@ -82,7 +82,7 @@ export const dashboardController = {
           locale: application.locale,
           title: getMentionTitle({ name: item.name, "wm-property": item.wmProperty }),
           description: html ? { html } : undefined,
-          published: item.published || item.wmReceived,
+          published: ensureISOString(item.published) || ensureISOString(item.wmReceived),
           url: item.sourceUrl,
           user: {
             avatar: { src: item.authorPhoto },

package/lib/storage/webmentions.js

@@ -2,7 +2,7 @@
  * Webmentions MongoDB storage
  */
 
-import { extractDomain, sanitiseHtml } from "../utils.js";
+import { extractDomain, ensureISOString, sanitiseHtml } from "../utils.js";
 
 /**
  * Ensure indexes exist
@@ -36,7 +36,7 @@ export function jf2ToDocument(item) {
 
   return {
     wmId: item["wm-id"],
-    wmReceived: item["wm-received"] || new Date().toISOString(),
+    wmReceived: ensureISOString(item["wm-received"]) || new Date().toISOString(),
     wmProperty: item["wm-property"],
     wmTarget: item["wm-target"],
     authorName: item.author?.name || null,
@@ -44,7 +44,7 @@ export function jf2ToDocument(item) {
     authorPhoto: item.author?.photo || null,
     sourceUrl: item.url || null,
     sourceDomain: extractDomain(item.author?.url || item.url || ""),
-    published: item.published || null,
+    published: ensureISOString(item.published),
     contentHtml,
     contentText,
     name: item.name || null,
@@ -65,7 +65,7 @@ export function documentToJf2(doc) {
   const jf2 = {
     type: "entry",
     "wm-id": doc.wmId,
-    "wm-received": doc.wmReceived,
+    "wm-received": ensureISOString(doc.wmReceived),
     "wm-property": doc.wmProperty,
     "wm-target": doc.wmTarget,
     author: {
@@ -75,7 +75,7 @@ export function documentToJf2(doc) {
       photo: doc.authorPhoto || "",
     },
     url: doc.sourceUrl || "",
-    published: doc.published || doc.wmReceived,
+    published: ensureISOString(doc.published) || ensureISOString(doc.wmReceived),
   };
 
   if (doc.name) {

package/lib/utils.js

@@ -101,6 +101,21 @@ export const sanitiseHtml = (html) => {
   return html;
 };
 
+/**
+ * Ensure a value is an ISO 8601 date string.
+ * MongoDB may auto-convert ISO strings to Date objects (BSON Date).
+ * The Nunjucks `| date` filter calls `parseISO(string)` which crashes
+ * on Date objects with `dateString.split is not a function`.
+ * @param {*} value - Date object, ISO string, or null
+ * @returns {string|null} ISO string or null
+ */
+export const ensureISOString = (value) => {
+  if (!value) return null;
+  if (value instanceof Date) return value.toISOString();
+  if (typeof value === "string") return value;
+  return null;
+};
+
 /**
  * Extract domain from a URL string
  * @param {string} url - URL

package/locales/de.json

@@ -0,0 +1,75 @@
+{
+  "webmention-io": {
+    "title": "Webmentions",
+    "webmentions": {
+      "none": "Keine Webmentions gefunden. Versuchen Sie eine Synchronisierung."
+    },
+    "mention": {
+      "bookmark-of": "hat %s mit einem Lesezeichen versehen",
+      "in-reply-to": "hat auf %s geantwortet",
+      "like-of": "gefällt %s",
+      "mention-of": "hat %s erwähnt",
+      "repost-of": "hat %s geteilt",
+      "rsvp": "hat auf %s geantwortet"
+    },
+    "sync": {
+      "now": "Jetzt synchronisieren",
+      "full": "Vollständige Neusynchronisierung",
+      "lastSync": "Zuletzt synchronisiert",
+      "never": "Nie synchronisiert",
+      "synced": "Synchronisierung abgeschlossen",
+      "added": "neue Erwähnungen hinzugefügt",
+      "inProgress": "Synchronisierung läuft…"
+    },
+    "filter": {
+      "all": "Alle",
+      "visible": "Sichtbar",
+      "hidden": "Versteckt",
+      "show": "Anzeigen",
+      "type": "Typ",
+      "likes": "Gefällt mir",
+      "replies": "Antworten",
+      "reposts": "Geteilte Beiträge",
+      "mentions": "Erwähnungen"
+    },
+    "actions": {
+      "hide": "Verstecken",
+      "unhide": "Einblenden",
+      "block": "Blockieren",
+      "hidden": "Versteckt",
+      "reasonManual": "manuell",
+      "reasonBlocklist": "Sperrliste",
+      "reasonPrivacy": "Datenschutz"
+    },
+    "blocklist": {
+      "title": "Webmention-Sperrliste",
+      "description": "Blockierte Domains werden nicht in Webmentions angezeigt. Vorhandene Erwähnungen von blockierten Domains werden automatisch ausgeblendet.",
+      "add": "Domain blockieren",
+      "domainLabel": "Domain",
+      "domainPlaceholder": "spam.example.com",
+      "reasonLabel": "Grund",
+      "reasonSpam": "Spam",
+      "reasonManual": "Manuell",
+      "blockButton": "Domain blockieren",
+      "unblock": "Entsperren",
+      "empty": "Keine Domains blockiert.",
+      "privacy": {
+        "title": "Datenschutzentfernung",
+        "description": "Alle Webmentions von einer Domain dauerhaft löschen und zukünftige blockieren. Verwenden Sie dies für GDPR- oder persönliche Entfernungsanfragen.",
+        "domainLabel": "Zu entfernende Domain",
+        "removeButton": "Entfernen & Blockieren"
+      },
+      "blocked": "Domain blockiert",
+      "unblocked": "Domain entsperrt",
+      "removed": "Erwähnungen dauerhaft gelöscht",
+      "unhidden": "Erwähnungen wiederhergestellt",
+      "mentionsHidden": "Erwähnungen ausgeblendet",
+      "blockedAt": "Blockiert"
+    },
+    "counts": {
+      "total": "gesamt",
+      "hidden": "versteckt",
+      "visible": "sichtbar"
+    }
+  }
+}

package/locales/es-419.json

@@ -0,0 +1,75 @@
+{
+  "webmention-io": {
+    "title": "Webmentions",
+    "webmentions": {
+      "none": "No se encontraron webmentions. Intentá ejecutar una sincronización."
+    },
+    "mention": {
+      "bookmark-of": "marcó %s",
+      "in-reply-to": "respondió a %s",
+      "like-of": "le gustó %s",
+      "mention-of": "mencionó %s",
+      "repost-of": "compartió %s",
+      "rsvp": "respondió a %s"
+    },
+    "sync": {
+      "now": "Sincronizar ahora",
+      "full": "Resincronización completa",
+      "lastSync": "Última sincronización",
+      "never": "Nunca sincronizado",
+      "synced": "Sincronización completada",
+      "added": "nuevas menciones agregadas",
+      "inProgress": "Sincronización en curso…"
+    },
+    "filter": {
+      "all": "Todos",
+      "visible": "Visible",
+      "hidden": "Oculto",
+      "show": "Mostrar",
+      "type": "Tipo",
+      "likes": "Me gusta",
+      "replies": "Respuestas",
+      "reposts": "Compartidos",
+      "mentions": "Menciones"
+    },
+    "actions": {
+      "hide": "Ocultar",
+      "unhide": "Mostrar",
+      "block": "Bloquear",
+      "hidden": "Oculto",
+      "reasonManual": "manual",
+      "reasonBlocklist": "lista de bloqueo",
+      "reasonPrivacy": "privacidad"
+    },
+    "blocklist": {
+      "title": "Lista de bloqueo de Webmentions",
+      "description": "Los dominios bloqueados no aparecerán en webmentions. Las menciones existentes de dominios bloqueados se ocultan automáticamente.",
+      "add": "Bloquear un dominio",
+      "domainLabel": "Dominio",
+      "domainPlaceholder": "spam.example.com",
+      "reasonLabel": "Motivo",
+      "reasonSpam": "Spam",
+      "reasonManual": "Manual",
+      "blockButton": "Bloquear dominio",
+      "unblock": "Desbloquear",
+      "empty": "No hay dominios bloqueados.",
+      "privacy": {
+        "title": "Eliminación por privacidad",
+        "description": "Eliminar permanentemente todos los webmentions de un dominio y bloquear futuros. Usar para solicitudes GDPR o de eliminación personal.",
+        "domainLabel": "Dominio a eliminar",
+        "removeButton": "Eliminar y bloquear"
+      },
+      "blocked": "Dominio bloqueado",
+      "unblocked": "Dominio desbloqueado",
+      "removed": "menciones eliminadas permanentemente",
+      "unhidden": "menciones restauradas",
+      "mentionsHidden": "menciones ocultas",
+      "blockedAt": "Bloqueado"
+    },
+    "counts": {
+      "total": "total",
+      "hidden": "oculto",
+      "visible": "visible"
+    }
+  }
+}

package/locales/es.json

@@ -0,0 +1,75 @@
+{
+  "webmention-io": {
+    "title": "Webmentions",
+    "webmentions": {
+      "none": "No se han encontrado webmentions. Intente ejecutar una sincronización."
+    },
+    "mention": {
+      "bookmark-of": "marcó %s",
+      "in-reply-to": "respondió a %s",
+      "like-of": "le gustó %s",
+      "mention-of": "mencionó %s",
+      "repost-of": "reenvió %s",
+      "rsvp": "respondió a %s"
+    },
+    "sync": {
+      "now": "Sincronizar ahora",
+      "full": "Resincronización completa",
+      "lastSync": "Última sincronización",
+      "never": "Nunca sincronizado",
+      "synced": "Sincronización completada",
+      "added": "nuevas menciones añadidas",
+      "inProgress": "Sincronización en curso…"
+    },
+    "filter": {
+      "all": "Todos",
+      "visible": "Visible",
+      "hidden": "Oculto",
+      "show": "Mostrar",
+      "type": "Tipo",
+      "likes": "Me gusta",
+      "replies": "Respuestas",
+      "reposts": "Reenvíos",
+      "mentions": "Menciones"
+    },
+    "actions": {
+      "hide": "Ocultar",
+      "unhide": "Mostrar",
+      "block": "Bloquear",
+      "hidden": "Oculto",
+      "reasonManual": "manual",
+      "reasonBlocklist": "lista de bloqueo",
+      "reasonPrivacy": "privacidad"
+    },
+    "blocklist": {
+      "title": "Lista de bloqueo de Webmentions",
+      "description": "Los dominios bloqueados no aparecerán en webmentions. Las menciones existentes de dominios bloqueados se ocultan automáticamente.",
+      "add": "Bloquear un dominio",
+      "domainLabel": "Dominio",
+      "domainPlaceholder": "spam.example.com",
+      "reasonLabel": "Motivo",
+      "reasonSpam": "Spam",
+      "reasonManual": "Manual",
+      "blockButton": "Bloquear dominio",
+      "unblock": "Desbloquear",
+      "empty": "No hay dominios bloqueados.",
+      "privacy": {
+        "title": "Eliminación por privacidad",
+        "description": "Eliminar permanentemente todos los webmentions de un dominio y bloquear futuros. Usar para solicitudes GDPR o de eliminación personal.",
+        "domainLabel": "Dominio a eliminar",
+        "removeButton": "Eliminar y bloquear"
+      },
+      "blocked": "Dominio bloqueado",
+      "unblocked": "Dominio desbloqueado",
+      "removed": "menciones eliminadas permanentemente",
+      "unhidden": "menciones restauradas",
+      "mentionsHidden": "menciones ocultas",
+      "blockedAt": "Bloqueado"
+    },
+    "counts": {
+      "total": "total",
+      "hidden": "oculto",
+      "visible": "visible"
+    }
+  }
+}

package/locales/fr.json

@@ -0,0 +1,75 @@
+{
+  "webmention-io": {
+    "title": "Webmentions",
+    "webmentions": {
+      "none": "Aucun webmention trouvé. Essayez d'exécuter une synchronisation."
+    },
+    "mention": {
+      "bookmark-of": "a mis %s en signet",
+      "in-reply-to": "a répondu à %s",
+      "like-of": "a aimé %s",
+      "mention-of": "a mentionné %s",
+      "repost-of": "a repartagé %s",
+      "rsvp": "a répondu à %s"
+    },
+    "sync": {
+      "now": "Synchroniser maintenant",
+      "full": "Resynchronisation complète",
+      "lastSync": "Dernière synchronisation",
+      "never": "Jamais synchronisé",
+      "synced": "Synchronisation terminée",
+      "added": "nouvelles mentions ajoutées",
+      "inProgress": "Synchronisation en cours…"
+    },
+    "filter": {
+      "all": "Tous",
+      "visible": "Visible",
+      "hidden": "Masqué",
+      "show": "Afficher",
+      "type": "Type",
+      "likes": "J'aime",
+      "replies": "Réponses",
+      "reposts": "Repartages",
+      "mentions": "Mentions"
+    },
+    "actions": {
+      "hide": "Masquer",
+      "unhide": "Afficher",
+      "block": "Bloquer",
+      "hidden": "Masqué",
+      "reasonManual": "manuel",
+      "reasonBlocklist": "liste de blocage",
+      "reasonPrivacy": "confidentialité"
+    },
+    "blocklist": {
+      "title": "Liste de blocage Webmention",
+      "description": "Les domaines bloqués n'apparaîtront pas dans les webmentions. Les mentions existantes de domaines bloqués sont automatiquement masquées.",
+      "add": "Bloquer un domaine",
+      "domainLabel": "Domaine",
+      "domainPlaceholder": "spam.example.com",
+      "reasonLabel": "Raison",
+      "reasonSpam": "Spam",
+      "reasonManual": "Manuel",
+      "blockButton": "Bloquer le domaine",
+      "unblock": "Débloquer",
+      "empty": "Aucun domaine bloqué.",
+      "privacy": {
+        "title": "Suppression pour confidentialité",
+        "description": "Supprimer définitivement tous les webmentions d'un domaine et bloquer les futurs. À utiliser pour les demandes GDPR ou de suppression personnelle.",
+        "domainLabel": "Domaine à supprimer",
+        "removeButton": "Supprimer et bloquer"
+      },
+      "blocked": "Domaine bloqué",
+      "unblocked": "Domaine débloqué",
+      "removed": "mentions supprimées définitivement",
+      "unhidden": "mentions restaurées",
+      "mentionsHidden": "mentions masquées",
+      "blockedAt": "Bloqué"
+    },
+    "counts": {
+      "total": "total",
+      "hidden": "masqué",
+      "visible": "visible"
+    }
+  }
+}