@rmdes/indiekit-endpoint-conversations 2.1.6 → 2.2.0

package/README.md

@@ -0,0 +1,161 @@
+# @rmdes/indiekit-endpoint-conversations
+
+Conversation aggregation endpoint for [Indiekit](https://getindiekit.com/). Polls Mastodon, Bluesky, and ActivityPub notifications, stores interactions in MongoDB, and serves them as a JF2-compatible API — including threaded owner replies.
+
+## Features
+
+- **Multi-platform polling** — Mastodon, Bluesky, and native ActivityPub (via Fedify)
+- **JF2 API** — serves likes, reposts, and replies in webmention-compatible format
+- **Owner reply threading** — enriches API responses with the site owner's replies from the `posts` collection, with threading metadata
+- **Webmention ingestion** — accepts incoming webmentions from Bridgy or external services
+- **Admin dashboard** — connection status, polling stats, platform health
+- **Syndication URL matching** — resolves canonical post URLs from syndicated copies
+
+## Installation
+
+```bash
+npm install @rmdes/indiekit-endpoint-conversations
+```
+
+```javascript
+// indiekit.config.js
+import ConversationsEndpoint from "@rmdes/indiekit-endpoint-conversations";
+
+export default {
+  plugins: [
+    new ConversationsEndpoint({
+      mountPath: "/conversations",
+    }),
+  ],
+};
+```
+
+## Environment Variables
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `MASTODON_ACCESS_TOKEN` | For Mastodon | Mastodon API access token |
+| `MASTODON_URL` or `MASTODON_INSTANCE` | For Mastodon | Mastodon instance URL |
+| `BLUESKY_IDENTIFIER` or `BLUESKY_HANDLE` | For Bluesky | Bluesky account identifier |
+| `BLUESKY_PASSWORD` | For Bluesky | Bluesky app password |
+| `AUTHOR_NAME` | Optional | Owner display name (falls back to site hostname) |
+| `AUTHOR_AVATAR` | Optional | Owner avatar URL |
+
+ActivityPub polling is auto-detected when `@rmdes/indiekit-endpoint-activitypub` is installed.
+
+## API
+
+### GET /conversations/api/mentions
+
+Returns interactions for a target URL in JF2 feed format. Compatible with the webmention.io API shape used by `@chrisburnell/eleventy-cache-webmentions`.
+
+**Parameters:**
+
+| Param | Type | Description |
+|-------|------|-------------|
+| `target` | string | Target URL to fetch interactions for |
+| `wm-property` | string | Filter by type: `like-of`, `repost-of`, `in-reply-to` |
+| `per-page` | number | Results per page (default: 50, max: 100) |
+| `page` | number | Page number (default: 0) |
+
+**Response:**
+
+```json
+{
+  "type": "feed",
+  "name": "Conversations",
+  "children": [
+    {
+      "type": "entry",
+      "wm-id": "conv-mastodon:12345",
+      "wm-property": "in-reply-to",
+      "wm-target": "https://example.com/posts/hello",
+      "author": {
+        "type": "card",
+        "name": "Jane Doe",
+        "url": "https://mastodon.social/@jane",
+        "photo": "https://..."
+      },
+      "url": "https://mastodon.social/@jane/67890",
+      "published": "2026-03-11T16:19:52.652Z",
+      "platform": "mastodon",
+      "content": {
+        "html": "<p>Great post!</p>",
+        "text": "Great post!"
+      }
+    },
+    {
+      "type": "entry",
+      "wm-id": "owner-reply-abc123",
+      "wm-property": "in-reply-to",
+      "wm-target": "https://example.com/posts/hello",
+      "author": {
+        "type": "card",
+        "name": "Site Owner",
+        "url": "https://example.com",
+        "photo": "https://..."
+      },
+      "url": "https://example.com/replies/2026/03/11/65e12",
+      "published": "2026-03-11T17:00:00.000Z",
+      "content": {
+        "html": "<p>Thanks!</p>",
+        "text": "Thanks!"
+      },
+      "is_owner": true,
+      "parent_url": "https://mastodon.social/@jane/67890"
+    }
+  ]
+}
+```
+
+### Owner Reply Enrichment
+
+When the API returns replies (`wm-property: "in-reply-to"`), it checks the Indiekit `posts` collection for owner posts whose `properties.in-reply-to` matches any reply's `url`. Matching owner posts are appended to the response with two extra fields:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `is_owner` | boolean | Always `true` for owner replies |
+| `parent_url` | string | The URL of the interaction this reply responds to |
+
+The frontend uses `parent_url` to thread the owner's reply under the correct parent interaction. See [`indiekit-eleventy-theme`](https://github.com/rmdes/indiekit-eleventy-theme) for the client-side threading implementation.
+
+### GET /conversations/api/status
+
+Returns connection health and platform status.
+
+### POST /conversations/ingest
+
+Accepts incoming webmentions. Body: `{ source, target }`.
+
+### POST /conversations/poll (authenticated)
+
+Triggers an immediate poll of all configured platforms.
+
+## Architecture
+
+```
+Mastodon API ──┐
+Bluesky API  ──┼──> Scheduler ──> conversation_items (MongoDB)
+ActivityPub  ──┘                         │
+                                         v
+               GET /api/mentions ──> JF2 response
+                                    + owner reply enrichment
+                                      (from posts collection)
+```
+
+### Collections
+
+| Collection | Purpose |
+|------------|---------|
+| `conversation_items` | Stored interactions (likes, reposts, replies) |
+| `conversation_state` | Polling state (last poll timestamps, cursors) |
+
+### Dependencies
+
+- **`@rmdes/indiekit-endpoint-activitypub`** — Optional. When installed, the scheduler also polls native ActivityPub interactions from the `ap_interactions` collection.
+- **`indiekit-eleventy-theme`** — The theme's `webmentions.js` consumes the `/api/mentions` endpoint and threads owner replies using the `is_owner` and `parent_url` fields.
+- **`@rmdes/indiekit-endpoint-comments`** — Handles native comment replies (not platform interactions). Owner replies to native comments go through the comments API, not this plugin.
+
+## License
+
+MIT

package/index.js

@@ -81,6 +81,7 @@ export default class ConversationsEndpoint {
     // Register MongoDB collections
     Indiekit.addCollection("conversation_items");
     Indiekit.addCollection("conversation_state");
+    Indiekit.addCollection("nodeinfo_cache");
 
     Indiekit.addEndpoint(this);
 

package/lib/controllers/conversations.js

@@ -165,6 +165,58 @@ async function apiMentions(request, response) {
 
     const children = items.map(conversationItemToJf2);
 
+    // Enrich with owner replies from the posts collection
+    // Owner replies are Micropub posts with in-reply-to matching an interaction URL
+    const replyUrls = children
+      .filter((c) => c["wm-property"] === "in-reply-to")
+      .map((c) => c.url)
+      .filter(Boolean);
+
+    if (replyUrls.length > 0) {
+      const postsCollection = application.collections?.get("posts");
+      if (postsCollection) {
+        const siteUrl = application.publication?.me || application.url || "";
+        const ownerName =
+          process.env.AUTHOR_NAME ||
+          (siteUrl ? new URL(siteUrl).hostname : "Owner");
+
+        const ownerPosts = await postsCollection
+          .find({
+            "properties.in-reply-to": { $in: replyUrls },
+          })
+          .sort({ "properties.published": -1 })
+          .limit(50)
+          .toArray();
+
+        for (const post of ownerPosts) {
+          const inReplyTo = post.properties?.["in-reply-to"];
+          if (!inReplyTo || typeof inReplyTo !== "string") continue;
+
+          children.push({
+            type: "entry",
+            "wm-id": `owner-reply-${post._id}`,
+            "wm-property": "in-reply-to",
+            "wm-target": target || "",
+            "wm-received": post.properties?.published || "",
+            author: {
+              type: "card",
+              name: ownerName,
+              url: siteUrl,
+              photo: process.env.AUTHOR_AVATAR || "",
+            },
+            url: post.properties?.url || "",
+            published: post.properties?.published || "",
+            content: {
+              text: post.properties?.content?.text || "",
+              html: post.properties?.content?.html || "",
+            },
+            is_owner: true,
+            parent_url: inReplyTo,
+          });
+        }
+      }
+    }
+
     response.set("Cache-Control", "public, max-age=60");
     response.json({
       type: "feed",

package/lib/nodeinfo/resolver.js

@@ -0,0 +1,153 @@
+/**
+ * NodeInfo-based server software resolver
+ * Fetches /.well-known/nodeinfo from a domain, follows the link,
+ * and returns the software name (e.g., "mastodon", "pleroma", "misskey").
+ * Results are cached in-memory and optionally persisted to MongoDB.
+ * @module nodeinfo/resolver
+ */
+
+const NODEINFO_TIMEOUT_MS = 5000;
+const CACHE_TTL_MS = 7 * 24 * 60 * 60 * 1000; // 7 days
+
+// In-memory cache: domain -> { software, resolvedAt }
+const memoryCache = new Map();
+
+/**
+ * Resolve the server software for a given actor URL via NodeInfo.
+ * Returns a lowercase software name like "mastodon", "pleroma", "misskey",
+ * "gotosocial", "fedify", etc. Falls back to "activitypub" if NodeInfo
+ * is unavailable or unrecognizable.
+ *
+ * @param {string} actorUrl - The actor's URL (e.g., "https://mastodon.social/@user")
+ * @param {object} [collection] - Optional MongoDB collection for persistent cache
+ * @returns {Promise<string>} Lowercase software name or "activitypub"
+ */
+export async function resolveServerSoftware(actorUrl, collection) {
+  const domain = extractDomain(actorUrl);
+  if (!domain) return "activitypub";
+
+  // Check in-memory cache first
+  const cached = memoryCache.get(domain);
+  if (cached && Date.now() - cached.resolvedAt < CACHE_TTL_MS) {
+    return cached.software;
+  }
+
+  // Check MongoDB cache
+  if (collection) {
+    try {
+      const doc = await collection.findOne({ _id: domain });
+      if (doc && Date.now() - new Date(doc.resolvedAt).getTime() < CACHE_TTL_MS) {
+        memoryCache.set(domain, {
+          software: doc.software,
+          resolvedAt: new Date(doc.resolvedAt).getTime(),
+        });
+        return doc.software;
+      }
+    } catch { /* proceed to live fetch */ }
+  }
+
+  // Live fetch via NodeInfo protocol
+  const software = await fetchNodeInfo(domain);
+
+  // Cache result (even "activitypub" fallback — avoids repeated failed lookups)
+  const entry = { software, resolvedAt: Date.now() };
+  memoryCache.set(domain, entry);
+
+  if (collection) {
+    try {
+      await collection.findOneAndUpdate(
+        { _id: domain },
+        { $set: { software, resolvedAt: new Date().toISOString() } },
+        { upsert: true },
+      );
+    } catch { /* non-critical */ }
+  }
+
+  return software;
+}
+
+/**
+ * Batch-resolve software for multiple actor URLs.
+ * Deduplicates by domain so each domain is only queried once.
+ *
+ * @param {string[]} actorUrls - Array of actor URLs
+ * @param {object} [collection] - Optional MongoDB collection for persistent cache
+ * @returns {Promise<Map<string, string>>} Map of domain -> software name
+ */
+export async function batchResolve(actorUrls, collection) {
+  const domains = new Set();
+  for (const url of actorUrls) {
+    const domain = extractDomain(url);
+    if (domain) domains.add(domain);
+  }
+
+  const results = new Map();
+  for (const domain of domains) {
+    results.set(
+      domain,
+      await resolveServerSoftware(`https://${domain}/`, collection),
+    );
+  }
+  return results;
+}
+
+/**
+ * Extract domain from a URL
+ * @param {string} url
+ * @returns {string|null}
+ */
+function extractDomain(url) {
+  try {
+    return new URL(url).hostname;
+  } catch {
+    return null;
+  }
+}
+
+/**
+ * Fetch NodeInfo for a domain and return the software name
+ * @param {string} domain
+ * @returns {Promise<string>} Software name or "activitypub"
+ */
+async function fetchNodeInfo(domain) {
+  try {
+    // Step 1: Fetch /.well-known/nodeinfo
+    const wellKnownUrl = `https://${domain}/.well-known/nodeinfo`;
+    const wellKnownResp = await fetch(wellKnownUrl, {
+      headers: { Accept: "application/json" },
+      signal: AbortSignal.timeout(NODEINFO_TIMEOUT_MS),
+    });
+
+    if (!wellKnownResp.ok) return "activitypub";
+
+    const wellKnown = await wellKnownResp.json();
+    const links = wellKnown.links;
+    if (!Array.isArray(links) || links.length === 0) return "activitypub";
+
+    // Prefer NodeInfo 2.x, fall back to any available link
+    const link =
+      links.find((l) => l.rel?.includes("nodeinfo/2.")) ||
+      links[0];
+
+    if (!link?.href) return "activitypub";
+
+    // Step 2: Fetch the actual NodeInfo document
+    const nodeInfoResp = await fetch(link.href, {
+      headers: { Accept: "application/json" },
+      signal: AbortSignal.timeout(NODEINFO_TIMEOUT_MS),
+    });
+
+    if (!nodeInfoResp.ok) return "activitypub";
+
+    const nodeInfo = await nodeInfoResp.json();
+    const softwareName = nodeInfo.software?.name;
+
+    if (typeof softwareName === "string" && softwareName.trim()) {
+      return softwareName.trim().toLowerCase();
+    }
+
+    return "activitypub";
+  } catch {
+    return "activitypub";
+  }
+}

package/lib/notifications/activitypub.js

@@ -20,11 +20,12 @@ const typeMap = {
  * @param {object} options
  * @param {object} options.ap_activities - MongoDB collection
  * @param {object} options.ap_followers - MongoDB collection (for avatar lookup)
+ * @param {object} [options.nodeinfoCache] - MongoDB collection for NodeInfo cache
  * @param {string} [options.since] - ISO 8601 timestamp cursor (process activities after this)
  * @returns {Promise<{items: Array, cursor: string|null}>}
  */
 export async function fetchActivityPubInteractions(options) {
-  const { ap_activities, ap_followers, since } = options;
+  const { ap_activities, ap_followers, nodeinfoCache, since } = options;
 
   const query = {
     direction: "inbound",
@@ -45,6 +46,11 @@ export async function fetchActivityPubInteractions(options) {
     return { items: [], cursor: null };
   }
 
+  // Resolve server software for all actor domains in this batch
+  const { batchResolve } = await import("../nodeinfo/resolver.js");
+  const actorUrls = activities.map((a) => a.actorUrl).filter(Boolean);
+  const domainSoftware = await batchResolve(actorUrls, nodeinfoCache);
+
   const items = [];
 
   for (const activity of activities) {
@@ -53,7 +59,12 @@ export async function fetchActivityPubInteractions(options) {
     const avatar =
       activity.actorAvatar ||
       (await lookupAvatar(ap_followers, activity.actorUrl));
-    items.push(normalizeActivity(activity, avatar));
+
+    // Resolve platform from NodeInfo (e.g., "mastodon", "pleroma", "misskey")
+    const domain = extractDomain(activity.actorUrl);
+    const platform = domain ? (domainSoftware.get(domain) || "activitypub") : "activitypub";
+
+    items.push(normalizeActivity(activity, avatar, platform));
   }
 
   // Cursor is the receivedAt of the last activity processed
@@ -62,13 +73,27 @@ export async function fetchActivityPubInteractions(options) {
   return { items, cursor };
 }
 
+/**
+ * Extract hostname from a URL
+ * @param {string} url
+ * @returns {string|null}
+ */
+function extractDomain(url) {
+  try {
+    return new URL(url).hostname;
+  } catch {
+    return null;
+  }
+}
+
 /**
  * Normalize an ap_activities document into conversations internal format
  * @param {object} activity - Document from ap_activities
  * @param {string} avatar - Avatar URL from ap_followers lookup
+ * @param {string} platform - Resolved server software (e.g., "mastodon", "pleroma")
  * @returns {object} Normalized interaction
  */
-function normalizeActivity(activity, avatar) {
+function normalizeActivity(activity, avatar, platform) {
   const type = typeMap[activity.type] || "mention";
   const isReply = activity.type === "Reply";
 
@@ -81,7 +106,7 @@ function normalizeActivity(activity, avatar) {
   const url = isReply ? activity.objectUrl : activity.actorUrl;
 
   return {
-    platform: "activitypub",
+    platform,
     platform_id: `activitypub:${activity.type}:${activity.actorUrl}:${activity.objectUrl}`,
     type,
     author: {

package/lib/polling/scheduler.js

@@ -97,6 +97,9 @@ export async function runPollCycle(indiekit, options) {
 
   // Backfill missing avatars from ap_notifications (one-time sweep per cycle)
   await backfillMissingAvatars(indiekit, stateCollection);
+
+  // Backfill platform names for items stored as "activitypub" (one-time)
+  await backfillPlatformNames(indiekit, stateCollection);
 }
 
 /**
@@ -233,6 +236,72 @@ async function backfillMissingAvatars(indiekit, stateCollection) {
   }
 }
 
+/**
+ * Backfill platform names for existing items stored with source "activitypub".
+ * Uses NodeInfo to resolve the actual server software (e.g., "mastodon", "pleroma").
+ * One-time operation — marks complete after first successful run.
+ */
+async function backfillPlatformNames(indiekit, stateCollection) {
+  try {
+    const itemsCollection = indiekit.collections.get("conversation_items");
+    if (!itemsCollection) return;
+
+    // Check if backfill already completed
+    const state = await stateCollection.findOne({ _id: "poll_cursors" });
+    if (state?.platform_backfill_complete) return;
+
+    // Find unique author URLs for items with source "activitypub"
+    const actorUrls = await itemsCollection.distinct("author.url", {
+      source: "activitypub",
+    });
+
+    if (actorUrls.length === 0) {
+      await stateCollection.findOneAndUpdate(
+        { _id: "poll_cursors" },
+        { $set: { platform_backfill_complete: true } },
+        { upsert: true },
+      );
+      return;
+    }
+
+    const { batchResolve } = await import("../nodeinfo/resolver.js");
+    const nodeinfoCache = indiekit.collections?.get("nodeinfo_cache") || null;
+    const domainSoftware = await batchResolve(
+      actorUrls.filter(Boolean),
+      nodeinfoCache,
+    );
+
+    let updated = 0;
+
+    for (const [domain, software] of domainSoftware) {
+      if (software === "activitypub") continue; // No change needed
+
+      const result = await itemsCollection.updateMany(
+        {
+          source: "activitypub",
+          "author.url": { $regex: `^https?://${domain.replace(/\./g, "\\.")}/` },
+        },
+        { $set: { source: software } },
+      );
+      if (result.modifiedCount > 0) updated += result.modifiedCount;
+    }
+
+    if (updated > 0) {
+      console.info(
+        `[Conversations] Platform backfill: updated ${updated} items from "activitypub" to resolved software names`,
+      );
+    }
+
+    await stateCollection.findOneAndUpdate(
+      { _id: "poll_cursors" },
+      { $set: { platform_backfill_complete: true } },
+      { upsert: true },
+    );
+  } catch (error) {
+    console.warn("[Conversations] Platform backfill error:", error.message);
+  }
+}
+
 /**
  * Poll Mastodon notifications and store matching interactions
  */
@@ -470,9 +539,13 @@ async function pollActivityPub(indiekit, stateCollection, state) {
       ""
     ).replace(/\/$/, "");
 
+    // NodeInfo cache collection for resolving server software per domain
+    const nodeinfoCache = indiekit.collections?.get("nodeinfo_cache") || null;
+
     const result = await fetchActivityPubInteractions({
       ap_activities,
       ap_followers,
+      nodeinfoCache,
       since: state.activitypub_last_received_at || null,
     });
 
@@ -491,7 +564,7 @@ async function pollActivityPub(indiekit, stateCollection, state) {
 
       await upsertConversationItem(indiekit, {
         canonical_url: interaction.canonical_url,
-        source: "activitypub",
+        source: interaction.platform,
         type: interaction.type,
         author: interaction.author,
         content: interaction.content,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@rmdes/indiekit-endpoint-conversations",
-  "version": "2.1.6",
+  "version": "2.2.0",
   "description": "Conversation aggregation endpoint for Indiekit. Backend enrichment service that polls Mastodon/Bluesky notifications and serves JF2-compatible data for the interactions page.",
   "keywords": [
     "indiekit",