fedipod 1.30.0 → 1.36.6

package/README.md

@@ -72,7 +72,7 @@ client-to-server, dokieli for one, can post as you. It sends to the outbox
 address in your actor document, which your WebID profile also names, signed in
 at your pod. The post goes out the next time you open fedipod.net.
 
-**The manage page.** `manage account` in the bar opens it: your profile,
+**The manage page.** `manage`, in the bar's account group, opens it: your profile,
 aliases, the gateway, key rotation, recovering posts, parking, moving to
 another server, retiring, and clearing a backlog. It is the same interface
 the DeviceAgent has, described in [the admin interface](gui.md).

package/gateway.md

@@ -153,6 +153,43 @@ on the setup page). Setup reads the account on the pod, keeps its state and
 key there, attaches at the new gateway, and completes the move when the
 agent first acts. See [the DeviceAgent](device-agent.md).
 
+## Accounts that go quiet
+
+A gateway holds no mail. Every delivery it accepts is written into your pod
+inbox, and your agent reads it from there. A BrowserAgent reads only while
+its page is open, so an account nobody opens grows on its pod without limit
+and comes back to a drain of everything at once. fedipod.net keeps two facts
+about each browser account — when its owner last signed in or posted, and
+how much content has arrived since — and acts on them.
+
+**Paused.** After about 5,000 posts, replies, likes, boosts and edits since
+you were last here, or when you say so on the manage page, the door accepts
+content and discards it. Follows, unfollows, account moves, deletions and
+blocks still reach your pod. Signing in ends an automatic pause by itself; a
+pause you set lasts until you lift it.
+
+**Closed.** After six months without a sign-in, or when you say so on the
+manage page, the address is closed for good: its handle, its actor and its
+door answer 410 Gone, other servers drop the account the next time they look,
+and nobody can take the name. Nothing on your pod is touched. An address that
+had already moved to another gateway keeps answering as moved.
+
+Only accounts opened from a browser are counted. A DeviceAgent behind the
+gateway drains its own inbox as it runs, and is never paused or closed by
+time. Accounts from before this was built are counted from their next
+sign-in. A gateway operator sets the cap and the window with
+`FEDIPOD_PAUSE_ITEMS` and `FEDIPOD_CLOSE_DAYS`.
+
+## Notices
+
+Whoever runs the gateway can write notices to everyone with an account
+there. A bell at the right end of the bar, on the record page and in the
+client, shows how many this browser has not opened yet; the list gives the
+titles and each one opens on its own. The operator writes, changes and
+removes them at `/notices`, signed in as the admin the way the roster is.
+A notice is a title and plain text: a blank line starts a paragraph, and a
+web address becomes a link.
+
 ## What the gateway can see
 
 It reads only public data to decide what concerns you: your published

package/gui.md

@@ -1,11 +1,11 @@
 # The Admin interface
 
-In the browser version at fedipod.net, `manage account` in the bar opens this
+In the browser version at fedipod.net, `manage`, in the bar's account group, opens this
 same page for your account. The rest of this page describes it as the
 DeviceAgent serves it; the controls are the same, minus the manual inbox
 drain and the local log, which a browser does not have.
 
-Open `https://localhost:8030/` while any agent is running — it forwards you to the agent — then choose `manage account` and select the actor you want from the local actors dropdown.
+Open `https://localhost:8030/` while any agent is running — it forwards you to the agent — then choose `manage` in the bar's account group and select the actor you want from the local actors dropdown.
 Picking an actor marked "(stopped)" starts its agent, then opens its page.
 
 The software row names the version the agent is running. When the copy on the machine is further ahead — after an update, or after pulling a checkout — it says so and asks for a restart, because an agent goes on serving the code it started with until it is restarted.

package/lib/connections/bskyfeed.mjs

@@ -166,6 +166,10 @@ export class BskyFeed {
     this.lastSweep = new Date().toISOString();
     const self = this.atproto.read()?.did;
     let added = 0;
+    // One commit boundary for the sweep, as acctfeed and tagfeed: every item
+    // awaits Bluesky before the next, so the debounce never coalesced them and
+    // the timeline index went to the pod once per notification.
+    this.store.hold?.();
     try {
       // Our own CROSS-POSTS must not echo back into the feed — but a post or
       // reply written natively on Bluesky is ours to see here too. The mirrors
@@ -224,6 +228,8 @@ export class BskyFeed {
     } catch (e) {
       this._backOff(e.status || 0, null);
       return;
+    } finally {
+      this.store.release?.();
     }
     const all = this.store.getStatuses();
     const mirrored = all.filter(s => s.kind === 'bsky');

package/lib/core/deliver.mjs

@@ -73,6 +73,10 @@ export class Deliverer {
     this.edPrivate = edPrivate;
     this.proofKeyId = proofKeyId;
     this.log = log;
+    // How many fresh deliveries one attempt may carry. The Node deliverer
+    // opens a socket per recipient; the browser's relay deliverer sends a
+    // list, and sets this to what the relay accepts.
+    this.batchSize = 1;
     if (!passive) this.startQueue();
   }
 
@@ -171,33 +175,51 @@ export class Deliverer {
     }
   }
 
+  // One attempt per target, answered in order: `{ ok: true }` or `{ error }`
+  // with the error deliverNow would have thrown. Here one at a time; the
+  // relay deliverer sends the whole list in one call.
+  async deliverManyNow(targets) {
+    const out = [];
+    for (const t of targets) {
+      try { await this.deliverNow(t.inbox, t.activity); out.push({ ok: true }); }
+      catch (error) { out.push({ error }); }
+    }
+    return out;
+  }
+
   async deliver(inbox, activity) {
     // Proved before anything else, so the copy that goes on the queue is the
     // copy that was signed — a retry days later must not post a bare activity.
     const signed = await this.proofed(activity);
-    // A host we already know is refusing: queue without asking again. This is
-    // the path a FRESH activity takes, so without it a fan-out to a struggling
-    // server opened one socket per follower before any of this applied.
+    if (this._queueIfCooling(inbox, signed)) return;
+    const [result] = await this.deliverManyNow([{ inbox, activity: signed }]);
+    await this._settle(inbox, signed, result);
+  }
+
+  // A host we already know is refusing: queue without asking again. This is
+  // the path a FRESH activity takes, so without it a fan-out to a struggling
+  // server opened one socket per follower before any of this applied.
+  _queueIfCooling(inbox, signed) {
     const host = hostOf(inbox);
     const until = this._cooling?.get(host);
-    if (until && until > Date.now()) {
-      this.log(`${host} is cooling — queueing ${signed.type} rather than asking again`);
-      this._enqueue({ inbox, activity: signed, attempts: 1, nextAt: until });
-      return;
-    }
-    try {
-      await this.deliverNow(inbox, signed);
-      this.log(`delivered ${signed.type} → ${inbox}`);
-    } catch (e) {
-      if (unsalvageable(e)) { await this._unsalvageable(inbox, e); return; }
-      this.log(`delivery failed (${e.message}) — queued`);
-      const wait = e.retryAfterMs || 60_000;
-      if (aboutTheHost(e)) {
-        this._cooling ||= new Map();
-        this._cooling.set(host, Date.now() + wait);
-      }
-      this._enqueue({ inbox, activity: signed, attempts: 1, nextAt: Date.now() + wait });
+    if (!until || until <= Date.now()) return false;
+    this.log(`${host} is cooling — queueing ${signed.type} rather than asking again`);
+    this._enqueue({ inbox, activity: signed, attempts: 1, nextAt: until });
+    return true;
+  }
+
+  // What one attempt's outcome means for the queue.
+  async _settle(inbox, signed, result) {
+    if (result.ok) { this.log(`delivered ${signed.type} → ${inbox}`); return; }
+    const e = result.error;
+    if (unsalvageable(e)) { await this._unsalvageable(inbox, e); return; }
+    this.log(`delivery failed (${e.message}) — queued`);
+    const wait = e.retryAfterMs || 60_000;
+    if (aboutTheHost(e)) {
+      this._cooling ||= new Map();
+      this._cooling.set(hostOf(inbox), Date.now() + wait);
     }
+    this._enqueue({ inbox, activity: signed, attempts: 1, nextAt: Date.now() + wait });
   }
 
   // Not retried, and on a 410 the followers who received there are dropped,
@@ -231,8 +253,16 @@ export class Deliverer {
     // Proved once for the whole fan-out: every recipient gets the same bytes,
     // and one signature is computed rather than one per follower.
     const signed = await this.proofed(activity);
-    // Shared inboxes deduplicate fan-out to the same server.
-    for (const inbox of [...new Set(inboxes)]) await this.deliver(inbox, signed);
+    // Shared inboxes deduplicate fan-out to the same server. Sent in batches
+    // of batchSize, and a host found cooling by one batch is not asked again
+    // by the next.
+    const targets = [...new Set(inboxes)].map((inbox) => ({ inbox, activity: signed }));
+    for (let i = 0; i < targets.length; i += this.batchSize) {
+      const chunk = targets.slice(i, i + this.batchSize).filter((t) => !this._queueIfCooling(t.inbox, signed));
+      if (!chunk.length) continue;
+      const results = await this.deliverManyNow(chunk);
+      for (let k = 0; k < chunk.length; k++) await this._settle(chunk[k].inbox, signed, results[k]);
+    }
   }
 
   // Serialized, for the same reason Intake.drain is: the tick is 60s and a

package/lib/core/intake/index.mjs

@@ -51,6 +51,7 @@ const CHAIN_GAP_MS = 5_000;      // pause between chained backlog sweeps
 // enough that a crash re-does little, large enough that a flood of fast
 // rejections does not become a pod write per item.
 const DELETE_BATCH = 10;
+const ORPHAN_RECEIPTS_PER_SWEEP = 20;   // strays from before receipts were deleted with their items
 // Attempt counts live in pod state, not in memory: a restart used to hand every
 // poison item five fresh tries, and under a crash loop that is unbounded.
 const ATTEMPTS_DOC = 'intake-attempts.json';
@@ -359,6 +360,7 @@ export class Intake {
     // deadletter.json where one would do, and a flood is exactly when the pod
     // should be asked for less rather than more.
     const pending = [];
+    const withReceipt = new Set();   // items the listing showed a receipt beside
     const flush = async () => {
       if (!pending.length) return true;
       // Written down before any of them leaves the mailbox. A failure here
@@ -379,13 +381,17 @@ export class Intake {
         }
         this._clearAttempt(url);
         handled++;
+        // Its receipt goes with it. Left beside a deleted item, a receipt
+        // was one stray document per delivery, forever.
+        if (withReceipt.has(url)) await podInbox.dropReceiptBeside(this.remote, url);
         await new Promise(r => setTimeout(r, DELETE_GAP_MS));
       }
       return true;
     };
 
-    for (const { url, size } of items) {
+    for (const { url, size, receipt: hasReceipt } of items) {
       if (url.endsWith('.keep')) continue;
+      if (hasReceipt) withReceipt.add(url);
       // The listing already carries every child's size, so this costs nothing
       // to ask. An activity is a few kB; anything of this order is not one, and
       // reading it with an unbounded res.text() buffers whatever a stranger
@@ -430,9 +436,11 @@ export class Intake {
         }
         // A gateway that verified this delivery left a receipt beside it. Read
         // it only when a gateway is configured (no config → no fetch, so an
-        // install with no gateway pays nothing); a missing or HMAC-invalid
-        // receipt reads as null, which is exactly today's unverified behavior.
-        const receipt = activity ? await this._readReceipt(url) : null;
+        // install with no gateway pays nothing) and the listing showed one
+        // beside this item (`false` is a definite no; a listing that does not
+        // say leaves it to the read); a missing or HMAC-invalid receipt reads
+        // as null, which is exactly today's unverified behavior.
+        const receipt = activity && hasReceipt !== false ? await this._readReceipt(url) : null;
         if (activity && this.gatewaySecret()) this._bumpGatewayStat(!!receipt?.verified);
         // The owner's own post, taken at the outbox door: not mail to read
         // but a write to make. Never archived or forwarded as if received.
@@ -464,6 +472,11 @@ export class Intake {
       if (pending.length >= DELETE_BATCH && !await flush()) return;
     }
     if (!await this._finishSweep(flush)) return;
+    // Receipts earlier drains left behind, a few per sweep until none remain.
+    for (const stray of podInbox.orphanReceipts(this.remote, this.urls).slice(0, ORPHAN_RECEIPTS_PER_SWEEP)) {
+      if (!await podInbox.dropStrayReceipt(this.remote, stray)) break;
+      await new Promise(r => setTimeout(r, DELETE_GAP_MS));
+    }
     // Made progress and there is more waiting: go straight round rather than
     // sleeping. Gated on progress so a sweep that achieved nothing — a
     // cooldown, an unwritable store, poison at the head — cannot spin.

package/lib/core/lease.mjs

@@ -9,13 +9,13 @@
 import crypto from 'node:crypto';
 
 // Renewal is the one thing that writes even when nothing is happening: at 30s
-// it was 120 PUTs an hour per agent, each taking a write lock on the pod. 90s
-// against a 5-minute TTL leaves three missed renewals of headroom and cuts that
-// load by two thirds. The cost is automatic promotion after a crash waiting up
-// to the TTL — a user acting on a second device does not wait, since takeover()
-// claims the lease outright.
-const TTL_MS = 300_000;
-const RENEW_MS = 90_000;
+// it was 120 PUTs an hour per agent, each taking a write lock on the pod; at
+// 90s, 40. Five minutes against a 15-minute TTL leaves two missed renewals of
+// headroom and makes it 12. The cost is automatic promotion after a crash
+// waiting up to the TTL — a user acting on a second device does not wait,
+// since takeover() claims the lease outright.
+const TTL_MS = 900_000;
+const RENEW_MS = 300_000;
 const JITTER = () => 0.85 + Math.random() * 0.3;
 // Distinct from null: null is "nobody holds it", this is "we could not ask".
 const UNREADABLE = Symbol('lease-unreadable');

package/lib/core/store.mjs

@@ -72,6 +72,11 @@ export function dropFollower(contacts, actor, why) {
   return contacts;
 }
 
+// What a state document looks like on the pod; write() and _put() agree on it.
+const serialise = (obj) => JSON.stringify(obj, null, 2) + '\n';
+// The same record apart from when it was fetched.
+const sameButWhen = (a, b) => JSON.stringify({ ...a, fetchedAt: null }) === JSON.stringify({ ...b, fetchedAt: null });
+
 export class PodStore {
   constructor({ storage = null, log = console.log } = {}) {
     this.lastSkipped = [];        // what the last load could not read, as `name (HTTP n)`
@@ -79,6 +84,7 @@ export class PodStore {
     this.log = log;
     this.cache = new Map();        // name → parsed value
     this.etags = new Map();        // name (or '') → last ETag, for revalidation
+    this.lastText = new Map();     // name → the bytes last loaded from or sent to the pod
     this.timers = new Map();       // name → debounce timer
     this.dirty = new Set();        // written while held; flushed by commit/release
     this._held = 0;
@@ -91,7 +97,7 @@ export class PodStore {
   attach(storage) {
     // A different tree is different state: carrying the old cache and its
     // ETags across would serve one container's documents as another's.
-    if (this.storage && this.storage.base !== storage.base) { this.cache.clear(); this.etags.clear(); }
+    if (this.storage && this.storage.base !== storage.base) { this.cache.clear(); this.etags.clear(); this.lastText.clear(); }
     this.storage = storage;
   }
 
@@ -144,7 +150,7 @@ export class PodStore {
       }
       fetched++;
       this.etags.set(name, r.etag);
-      try { this.cache.set(name, JSON.parse(r.body)); }
+      try { this.cache.set(name, JSON.parse(r.body)); this.lastText.set(name, r.body); }
       catch (e) { this.log(`state load ${name}: unparsable (${e.message})`); }
     }
     // Kept for /status: a document skipped here is a timeline or a contact
@@ -167,6 +173,15 @@ export class PodStore {
   write(name, obj) {
     this.cache.set(name, structuredClone(obj));
     if (!this.storage) return;
+    // The same bytes the pod already holds are not sent again. A like, a
+    // bookmark, a sweep that found nothing new each used to upload the whole
+    // timeline index and the whole people cache, unchanged — the largest
+    // write on the pod, on every small action. A write that had been armed
+    // and is now undone is unarmed with it.
+    if (this.lastText.get(name) === serialise(obj)) {
+      clearTimeout(this.timers.get(name)); this.timers.delete(name); this.dirty.delete(name);
+      return;
+    }
     // Held: record it and leave the writing to the commit boundary the caller
     // already has. See hold().
     if (this._held) { this.dirty.add(name); return; }
@@ -205,13 +220,13 @@ export class PodStore {
   // the returned promise so one failure cannot poison the queue.
   _put(name) {
     const done = this.chain.then(async () => {
-      const body = JSON.stringify(this.cache.get(name), null, 2) + '\n';
+      const body = serialise(this.cache.get(name));
       for (let attempt = 1; attempt <= PUT_RETRIES; attempt++) {
         // A storage that throws is a bug, not a hiccup — but it must not
         // escape into the write queue, where it would look like success.
         const r = await this.storage.write(name, body, 'application/json')
           .catch(e => ({ ok: false, retry: false, why: e.message }));
-        if (r.ok) return true;
+        if (r.ok) { this.lastText.set(name, body); return true; }
         // The storage says whether trying again could possibly help: a pod's
         // 4xx is an answer rather than a hiccup, and a directory's EACCES will
         // still be an EACCES in two seconds.
@@ -236,6 +251,7 @@ export class PodStore {
   // migrates to the local machine — leaving the copy behind would defeat it).
   async remove(name) {
     this.cache.delete(name);
+    this.lastText.delete(name);
     clearTimeout(this.timers.get(name));
     this.timers.delete(name);
     if (!this.storage) return true;
@@ -462,6 +478,7 @@ export class PodStore {
   getActors() { return this.read('actors.json', {}); }
   cacheActor(url, doc) {
     const a = this.getActors();
+    const known = a[url];
     a[url] = {
       name: clamp(plainText(doc.name || doc.preferredUsername || ''), MAX_NAME),
       preferredUsername: clamp(plainText(doc.preferredUsername || ''), MAX_NAME),
@@ -474,9 +491,12 @@ export class PodStore {
       type: doc.type || 'Person',
       followers: doc.followers || null,
       following: doc.following || null,
-      ...(a[url]?.counts ? { counts: a[url].counts } : {}),
+      ...(known?.counts ? { counts: known.counts } : {}),
       fetchedAt: new Date().toISOString(),
     };
+    // Fetched again and found the same: nothing to write. Only the time of
+    // asking differs, and that is not worth the whole cache going to the pod.
+    if (known && sameButWhen(known, a[url])) { a[url] = known; return; }
     this.write('actors.json', prune(a, ACTOR_CACHE_MAX, this.getContacts()));
   }
 

package/lib/gateway/caches.mjs

@@ -0,0 +1,36 @@
+// caches.mjs — what one warm gateway process keeps between requests: the
+// pod-token verifier, and the public keys of the servers that deliver here.
+// Built per request, each costs round trips to somebody else's server for
+// every call; built once, a process pays them once.
+
+// The Solid-OIDC verifier remembers the issuers' key sets it has seen. A
+// fresh one per call re-read the caller's WebID document, the issuer's
+// discovery document and its keys on every signed-in call.
+let verifier = null;
+export async function podTokenVerifier() {
+  if (!verifier) verifier = (await import('@solid/access-token-verifier')).createSolidTokenVerifier();
+  return verifier;
+}
+
+// The keys deliveries are signed with, by key id, in the shape Fedify's
+// verifier asks for. A key is held an hour; a key that could not be fetched
+// is held five minutes, so a server that was down is asked again soon.
+// Fedify re-fetches on its own when a held key no longer verifies, so a
+// rotated key costs one failed check, not an hour of refused mail.
+const KEY_TTL_MS = 60 * 60_000;
+const MISS_TTL_MS = 5 * 60_000;
+const KEY_CACHE_MAX = 500;
+const keys = new Map();   // key id → { key, until }
+export const senderKeys = {
+  async get(keyId) {
+    const hit = keys.get(keyId.href);
+    if (!hit) return undefined;
+    if (hit.until < Date.now()) { keys.delete(keyId.href); return undefined; }
+    return hit.key;
+  },
+  async set(keyId, key) {
+    if (keys.size >= KEY_CACHE_MAX) keys.delete(keys.keys().next().value);
+    keys.set(keyId.href, { key, until: Date.now() + (key ? KEY_TTL_MS : MISS_TTL_MS) });
+  },
+  size: () => keys.size,
+};