@rmdes/indiekit-endpoint-activitypub 2.15.0 → 2.15.2

package/README.md

@@ -12,6 +12,28 @@ ActivityPub federation endpoint for [Indiekit](https://getindiekit.com), built o
 - Reply delivery — replies are addressed to and delivered directly to the original post's author
 - Shared inbox support with collection sync (FEP-8fcf)
 - Configurable actor type (Person, Service, Organization, Group)
+- Manual follow approval — review and accept/reject follow requests before they take effect
+- Direct messages — private conversations stored separately from the public timeline
+
+**Federation Resilience** *(v2.14.0+)*
+- Async inbox queue — inbound activities are persisted to MongoDB before processing, ensuring no data loss on crashes
+- Server blocking — block entire remote servers by domain, rejecting all inbound activities from blocked instances
+- Key freshness tracking — tracks when remote actor keys were last verified, skipping redundant re-fetches
+- Redis-cached actor lookups — caches actor resolution results to reduce network round-trips
+- Delivery strike tracking on `ap_followers` — counts consecutive delivery failures per follower
+- FEP-fe34 security — verifies `proof.created` timestamps to reject replayed activities
+
+**Outbox Failure Handling** *(v2.15.0+, inspired by [Hollo](https://github.com/fedify-dev/hollo))*
+- **410 Gone** — immediate full cleanup: removes the follower, their timeline items, and their notifications
+- **404 Not Found** — strike system: 3 consecutive failures over 7+ days triggers the same full cleanup
+- Strike auto-reset — when an actor sends us any activity, their delivery failure count resets to zero
+- Prevents orphaned data from accumulating over time while tolerating temporary server outages
+
+**Reply Intelligence** *(v2.15.0+, inspired by [Hollo](https://github.com/fedify-dev/hollo))*
+- Recursive reply chain fetching — when a reply arrives, fetches parent posts up to 5 levels deep for thread context
+- Ancestor posts stored with `isContext: true` flag for thread view without cluttering the main timeline
+- Reply forwarding to followers — when someone replies to our posts, the reply is forwarded to our followers so they see the full conversation
+- Write-time visibility classification — computes `public`/`unlisted`/`private`/`direct` from `to`/`cc` fields at ingest time
 
 **Reader**
 - Timeline view showing posts from followed accounts with tab filtering (notes, articles, replies, boosts, media)
@@ -36,6 +58,8 @@ ActivityPub federation endpoint for [Indiekit](https://getindiekit.com), built o
 **Moderation**
 - Mute actors or keywords
 - Block actors (also removes from followers)
+- Block entire servers by domain
+- Report remote actors to their home instance (Flag activity)
 - All moderation actions available from the reader UI
 
 **Mastodon Migration**
@@ -186,6 +210,7 @@ When remote servers send activities to your inbox:
 - **Accept(Follow)** → Marks our follow as accepted
 - **Reject(Follow)** → Marks our follow as rejected
 - **Block** → Removes actor from our followers
+- **Flag** → Outbound report sent to remote actor's instance
 
 ### Content Negotiation
 
@@ -254,7 +279,7 @@ The plugin creates these collections automatically:
 
 | Collection | Description |
 |---|---|
-| `ap_followers` | Accounts following your actor |
+| `ap_followers` | Accounts following your actor (includes delivery failure strike tracking) |
 | `ap_following` | Accounts you follow |
 | `ap_activities` | Activity log with automatic TTL cleanup |
 | `ap_keys` | RSA and Ed25519 key pairs for HTTP Signatures |
@@ -262,11 +287,19 @@ The plugin creates these collections automatically:
 | `ap_profile` | Actor profile (single document) |
 | `ap_featured` | Pinned/featured posts |
 | `ap_featured_tags` | Featured hashtags |
-| `ap_timeline` | Reader timeline items from followed accounts |
+| `ap_timeline` | Reader timeline items (includes `visibility` and `isContext` fields) |
 | `ap_notifications` | Interaction notifications |
 | `ap_muted` | Muted actors and keywords |
 | `ap_blocked` | Blocked actors |
 | `ap_interactions` | Per-post like/boost tracking |
+| `ap_messages` | Direct messages / private conversations |
+| `ap_followed_tags` | Hashtags you follow for timeline filtering |
+| `ap_explore_tabs` | Saved Mastodon instances for the explore view |
+| `ap_reports` | Outbound reports (Flag activities) sent to remote instances |
+| `ap_pending_follows` | Follow requests awaiting manual approval |
+| `ap_blocked_servers` | Blocked server domains (instance-level blocks) |
+| `ap_key_freshness` | Tracks when remote actor keys were last verified |
+| `ap_inbox_queue` | Persistent async inbox processing queue |
 
 ## Supported Post Types
 
@@ -306,6 +339,23 @@ Mastodon's `update_account_fields` checks `attachment.is_a?(Array)` and silently
 
 **Revisit when:** Fedify adds an option to preserve arrays during JSON-LD serialization, or Mastodon fixes their array check.
 
+### Endpoints `as:Endpoints` Type Stripping
+
+**File:** `lib/federation-bridge.js` (in `sendFedifyResponse()`)
+**Upstream issue:** [fedify#576](https://github.com/fedify-dev/fedify/issues/576) — FIXED in Fedify 2.1.0
+
+Fedify serializes the `endpoints` object with `"type": "as:Endpoints"`, which is not a valid ActivityStreams type. browser.pub rejects this. The bridge strips the `type` field from the `endpoints` object before sending.
+
+**Remove when:** Upgrading to Fedify ≥ 2.1.0.
+
+### PropertyValue Attachment Type (Known Issue)
+
+**Upstream issue:** [fedify#629](https://github.com/fedify-dev/fedify/issues/629) — OPEN
+
+Fedify serializes `PropertyValue` attachments (used by Mastodon for profile metadata fields) with `"type": "PropertyValue"`, a schema.org type that is not a valid AS2 Object or Link. browser.pub rejects `/attachment` as invalid. However, every Mastodon-compatible server emits `PropertyValue` — removing it would break profile field display across the fediverse.
+
+**No workaround applied.** This is a de facto fediverse standard despite not being in the AS2 vocabulary.
+
 ### `.authorize()` Not Chained on Actor Dispatcher
 
 **File:** `lib/federation-setup.js` (line ~254)
@@ -346,6 +396,16 @@ This is not a bug — Fedify requires explicit opt-in for signed fetches. But it
 - **No custom emoji rendering** — Custom emoji shortcodes display as text
 - **In-process queue without Redis** — Activities may be lost on restart
 
+## Acknowledgements
+
+This plugin builds on the excellent [Fedify](https://fedify.dev) framework by [Hong Minhee](https://github.com/dahlia). Fedify provides the core ActivityPub federation layer — HTTP Signatures, content negotiation, message queues, and the vocabulary types that make all of this possible.
+
+Several federation patterns in this plugin were inspired by studying other open-source ActivityPub implementations:
+
+- **[Hollo](https://github.com/fedify-dev/hollo)** (by the Fedify author) — A single-user Fedify-based ActivityPub server that served as the primary reference implementation. The outbox permanent failure handling (410 cleanup and 404 strike system), recursive reply chain fetching, reply forwarding to followers, and write-time visibility classification in v2.15.0 are all adapted from Hollo's patterns for a MongoDB/single-user context.
+
+- **[Wafrn](https://github.com/gabboman/wafrn)** — A federated social network whose ActivityPub implementation informed the operational resilience patterns added in v2.14.0. Server blocking, key freshness tracking, async inbox processing with persistent queues, and the general approach to federation hardening were inspired by studying Wafrn's production codebase.
+
 ## License
 
 MIT

package/assets/reader.css

@@ -351,6 +351,12 @@
   margin-left: 0.2em;
 }
 
+.ap-card__visibility {
+  font-size: var(--font-size-xs);
+  margin-left: 0.3em;
+  opacity: 0.7;
+}
+
 .ap-card__timestamp-link {
   color: inherit;
   text-decoration: none;

package/lib/controllers/post-detail.js

@@ -62,51 +62,87 @@ async function loadParentChain(ctx, documentLoader, timelineCol, parentUrl, maxD
   return parents;
 }
 
-// Load replies collection (best-effort)
-async function loadReplies(object, ctx, documentLoader, timelineCol, maxReplies = 10) {
-  const replies = [];
+// Load local replies from ap_timeline (items where inReplyTo matches this post)
+async function loadLocalReplies(timelineCol, postUrl, postUid, maxReplies = 20) {
+  if (!timelineCol) return [];
 
-  try {
-    const repliesCollection = await object.getReplies({ documentLoader });
-    if (!repliesCollection) return replies;
+  const matchUrls = [postUrl, postUid].filter(Boolean);
+  if (matchUrls.length === 0) return [];
 
-    let items = [];
-    try {
-      items = await repliesCollection.getItems({ documentLoader });
-    } catch {
-      return replies;
-    }
+  const localReplies = await timelineCol
+    .find({ inReplyTo: { $in: matchUrls } })
+    .sort({ published: 1 })
+    .limit(maxReplies)
+    .toArray();
 
-    for (const replyItem of items.slice(0, maxReplies)) {
-      try {
-        const replyUrl = replyItem.id?.href || replyItem.url?.href;
-        if (!replyUrl) continue;
-
-        // Check timeline first
-        let reply = timelineCol
-          ? await timelineCol.findOne({
-              $or: [{ uid: replyUrl }, { url: replyUrl }],
-            })
-          : null;
-
-        if (!reply) {
-          // Extract from the item we already have
-          if (replyItem instanceof Note || replyItem instanceof Article) {
-            reply = await extractObjectData(replyItem);
-          }
+  return localReplies;
+}
+
+// Load replies collection (best-effort) — merges local + remote
+async function loadReplies(object, ctx, documentLoader, timelineCol, maxReplies = 20) {
+  const postUrl = object?.id?.href || object?.url?.href;
+
+  // Start with local replies already in our timeline (from organic inbox delivery
+  // or reply chain fetching). These are fast and free — no network requests.
+  const seenUrls = new Set();
+  const replies = await loadLocalReplies(timelineCol, postUrl, postUrl, maxReplies);
+  for (const r of replies) {
+    if (r.uid) seenUrls.add(r.uid);
+    if (r.url) seenUrls.add(r.url);
+  }
+
+  // Supplement with remote replies collection (may contain items we don't have locally)
+  if (object && replies.length < maxReplies) {
+    try {
+      const repliesCollection = await object.getReplies({ documentLoader });
+      if (repliesCollection) {
+        let items = [];
+        try {
+          items = await repliesCollection.getItems({ documentLoader });
+        } catch {
+          // Remote fetch failed — continue with local replies only
         }
 
-        if (reply) {
-          replies.push(reply);
+        for (const replyItem of items.slice(0, maxReplies - replies.length)) {
+          try {
+            const replyUrl = replyItem.id?.href || replyItem.url?.href;
+            if (!replyUrl || seenUrls.has(replyUrl)) continue;
+            seenUrls.add(replyUrl);
+
+            // Check timeline first
+            let reply = timelineCol
+              ? await timelineCol.findOne({
+                  $or: [{ uid: replyUrl }, { url: replyUrl }],
+                })
+              : null;
+
+            if (!reply) {
+              // Extract from the item we already have
+              if (replyItem instanceof Note || replyItem instanceof Article) {
+                reply = await extractObjectData(replyItem);
+              }
+            }
+
+            if (reply) {
+              replies.push(reply);
+            }
+          } catch {
+            continue; // Skip failed replies
+          }
         }
-      } catch {
-        continue; // Skip failed replies
       }
+    } catch {
+      // getReplies() failed or not available
     }
-  } catch {
-    // getReplies() failed or not available
   }
 
+  // Sort all replies chronologically
+  replies.sort((a, b) => {
+    const dateA = a.published || "";
+    const dateB = b.published || "";
+    return dateA < dateB ? -1 : dateA > dateB ? 1 : 0;
+  });
+
   return replies;
 }
 

package/lib/federation-bridge.js

@@ -89,6 +89,12 @@ async function sendFedifyResponse(res, response, request) {
       if (json.attachment && !Array.isArray(json.attachment)) {
         json.attachment = [json.attachment];
       }
+      // WORKAROUND: Fedify serializes endpoints with "type": "as:Endpoints"
+      // which is not a valid AS2 type. The endpoints object should be a plain
+      // object with just sharedInbox/proxyUrl etc. Strip the invalid type.
+      if (json.endpoints && json.endpoints.type) {
+        delete json.endpoints.type;
+      }
       const patched = JSON.stringify(json);
       res.setHeader("content-length", Buffer.byteLength(patched));
       res.end(patched);

package/lib/storage/timeline.js

@@ -73,6 +73,18 @@ export async function getTimelineItems(collections, options = {}) {
 
   const query = {};
 
+  // Exclude context-only items (ancestors fetched for thread reconstruction)
+  // unless explicitly requested via options.includeContext
+  if (!options.includeContext) {
+    query.isContext = { $ne: true };
+  }
+
+  // Exclude private/direct posts from the main timeline feed —
+  // these belong in messages/notifications, not the public reader
+  if (!options.includePrivate) {
+    query.visibility = { $nin: ["private", "direct"] };
+  }
+
   // Type filter
   if (options.type) {
     query.type = options.type;
@@ -252,7 +264,11 @@ export async function countNewItems(collections, after, options = {}) {
   const { ap_timeline } = collections;
   if (!after || Number.isNaN(new Date(after).getTime())) return 0;
 
-  const query = { published: { $gt: after } };
+  const query = {
+    published: { $gt: after },
+    isContext: { $ne: true },
+    visibility: { $nin: ["private", "direct"] },
+  };
   if (options.type) query.type = options.type;
   if (options.excludeReplies) {
     query.$or = [
@@ -289,5 +305,9 @@ export async function markItemsRead(collections, uids) {
  */
 export async function countUnreadItems(collections) {
   const { ap_timeline } = collections;
-  return await ap_timeline.countDocuments({ read: { $ne: true } });
+  return await ap_timeline.countDocuments({
+    read: { $ne: true },
+    isContext: { $ne: true },
+    visibility: { $nin: ["private", "direct"] },
+  });
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@rmdes/indiekit-endpoint-activitypub",
-  "version": "2.15.0",
+  "version": "2.15.2",
   "description": "ActivityPub federation endpoint for Indiekit via Fedify. Adds full fediverse support: actor, inbox, outbox, followers, following, syndication, and Mastodon migration.",
   "keywords": [
     "indiekit",

package/views/partials/ap-item-card.njk

@@ -65,6 +65,9 @@
         </time>
         {% if item.updated %}<span class="ap-card__edited" title="{{ item.updated | date('PPp') }}">✏️</span>{% endif %}
       </a>
+      {% if item.visibility and item.visibility != "public" %}
+        <span class="ap-card__visibility ap-card__visibility--{{ item.visibility }}" title="{% if item.visibility == 'unlisted' %}{{ __('activitypub.reader.compose.visibilityUnlisted') }}{% elif item.visibility == 'private' %}{{ __('activitypub.reader.compose.visibilityFollowers') }}{% elif item.visibility == 'direct' %}DM{% endif %}">{% if item.visibility == "unlisted" %}🔓{% elif item.visibility == "private" %}🔒{% elif item.visibility == "direct" %}✉️{% endif %}</span>
+      {% endif %}
     {% endif %}
   </header>