fedipod 1.39.0 → 1.40.0

package/README.md

@@ -4,7 +4,9 @@
 
 FediPod gives you a Fediverse account whose data lives on a Solid pod. You
 follow people on Mastodon, Bluesky, and other Fediverse or ATProto servers in
-one timeline. Your posts, followers and settings stay on your pod.
+one timeline. Your posts, followers and settings are kept on your pod. While fedipod.net
+keeps your account running, it works from a copy of them and writes it to your
+pod every fifteen minutes; your signing key never leaves your pod.
 
 The easiest way to run FediPod is to use it in any browser at https://fedipod.net. Nothing to install. Sign-up points you to a pod provider if you need a pod, then attaches a Fediverse identity to the pod you sign in with.
 
@@ -92,14 +94,16 @@ there it takes over; the first drops back to reading.
 
 ## What the browser version does not do
 
-- **Scheduled posts.** Nothing runs between now and the time you picked, so
+- **Scheduled posts, if you stop fedipod.net keeping your account running.**
+  It does by default, and publishes them while FediPod is closed. Turned off
+  on the manage page, nothing runs between now and the time you picked, so
   the composer refuses one rather than dropping it later.
-- **Notifications while the client is closed.** There is no push service.
-  Open the tab and they are there.
+- **Notifications, and other clients, if you stop fedipod.net keeping your
+  account running.** While it does (the default), any Mastodon app, phone or
+  desktop, can use your account with `fedipod.net` as its server, and an app
+  that offers notifications gets them on your phone with everything closed.
 - **Live updates.** The client refreshes by polling.
 - **Hosting a group.** Joining one works.
-- **Other clients.** A phone app or desktop client has nothing on the network
-  to connect to.
 
 For these, see [Other ways to run FediPod](#other-ways-to-run-fedipod).
 
@@ -108,6 +112,10 @@ For these, see [Other ways to run FediPod](#other-ways-to-run-fedipod).
 Everything you publish and everything you read is stored on your pod. Your
 signing key is stored there too, in a container only you can read through
 your pod's login.
+While fedipod.net keeps your account running, its own pod identity can read
+that container too, so it can act for you while FediPod is closed, and it
+works from a copy of your account's data that it writes back to your pod every
+fifteen minutes. Your key stays on your pod only.
 It all lives in the `fedipod` container you chose at sign-up. Your pod's
 public type index records it, as an ActivityStreams actor, which is how
 FediPod and other Solid apps find your account again.
@@ -119,7 +127,7 @@ not change.
 
 ## Other ways to use FediPod
 
-- [fediverse-account]() - an ESM library that supports both Solid-based and regular Fediverse accounts with methods to login, reply, boost, etc.
+- [fediverse-session](lib/session/README.md) - an ESM library that supports both Solid-based and regular Fediverse accounts with methods to login, reply, boost, etc.
 - [FediPod Server](packages/fedipod-server/README.md): a full ActivityPub
   server as a Community Solid Server component, giving every pod on the server
   the option of a Fediverse account.

package/architecture.md

@@ -16,6 +16,14 @@ and forwards the rest to the pod inbox with a receipt. It holds no key. The
 browser version always has one; the DeviceAgent may use one. Any
 lightweight host will do, Netlify included.
 
+A gateway can also keep a browser account running. It then acts as that
+account's agent whenever FediPod is closed, under a pod identity of its own
+that the owner's rules name, reading the signing key from the pod when it
+needs it. The account's state documents live in a working copy at the
+gateway, which FediPod in the browser, the gateway and any Mastodon app all
+work from, and which the gateway writes to the pod every fifteen minutes. See
+[the gateway](gateway.md).
+
 [FediPod Server](packages/fedipod-server/README.md) puts the agent inside a
 Community Solid Server, so anyone with a pod on that server can opt in to a
 Fediverse account fed by the server itself.

package/gateway.md

@@ -6,13 +6,16 @@ each delivery's signature where the headers still exist, drops forgeries and
 junk before they ever touch your pod, and passes the rest on with a receipt
 saying it checked.
 
-Your name, your signing key and your data stay on your own pod. The gateway is
+Your name, your signing key and your data are kept on your own pod. The gateway is
 **keyless** — it never holds the key you sign with, so it cannot post as you,
 read your private things, or be you anywhere. The worst a broken one can do is
 push items into your inbox, and those still face your agent's own checks.
 With the outbox door below, a gateway can also hand your agent a post marked as
 yours, which your agent then signs and sends. It still holds no key, but what you
-trust it with grows by that much.
+trust it with grows by that much. A browser account is also kept running by
+the gateway (below). That gives the gateway your key while it works, so it can
+post as you, as any Fediverse server can, and your account's working data is
+kept at the gateway too and written to your pod every fifteen minutes.
 
 A FediPod install works without any gateway at all. Deliveries go straight to
 your pod inbox, which holds them whether your agent is running or not.
@@ -153,10 +156,61 @@ on the setup page). Setup reads the account on the pod, keeps its state and
 key there, attaches at the new gateway, and completes the move when the
 agent first acts. See [the DeviceAgent](device-agent.md).
 
+## While your app is closed
+
+A browser account runs only while FediPod is open somewhere. While it is
+closed, fedipod.net does two things for it.
+
+**Your mail waits at the gateway.** Deliveries are kept there, not written to
+your pod one by one, and reach your pod in batches of up to a hundred: when you
+open FediPod, or every fifteen minutes otherwise. While FediPod is open, mail
+goes straight to your pod as before. A DeviceAgent's mail always goes straight
+to its pod.
+
+**The gateway keeps your account running.** Unless you turn it off, the
+gateway accepts your follows, tries again what failed to go out, and publishes
+the posts you scheduled, all while FediPod is closed. Other mail waits on your
+pod for FediPod to open, as before. It works under its own pod identity, which your app
+names in the access rules on your FediPod folder. It reads your signing key
+from your pod when it needs it, so it can post as you, and keeps no copy of the
+key. **Stop keeping it running** on the manage page takes it out of the rules.
+If fedipod.net ever changes its own pod identity, your FediPod moves your
+account over the next time you open it, with nothing for you to do.
+Scheduling a post is offered only while this is on, because otherwise nothing
+would be running when the time came.
+
+**Your account's working data is kept at the gateway.** While the gateway
+keeps your account running, your timeline, notifications, followers, settings
+and the rest of your FediPod state are kept at fedipod.net, and FediPod in your
+browser, the gateway and any app you use all work from that one copy, so they
+never disagree. fedipod.net writes what changed to your pod every fifteen
+minutes, and everything at once when you stop it keeping your account running,
+move your address or close it. Your signing key and the passwords of accounts
+you connected elsewhere are never in the copy; they stay on your pod only.
+
+## Using any Mastodon app
+
+While the gateway keeps your account running, any Mastodon app can use it:
+elk.zone, Ivory, Tusky, Phanpy and the rest. In the app, give `fedipod.net` as
+your server. The app sends you to a fedipod.net page that asks for your
+address here and signs you in at your own pod; no password is typed on
+fedipod.net. The app then reads your timeline and notifications from your
+account's copy at the gateway, and posts, boosts, likes and follows as you.
+
+An app checks for new posts every minute or so while it is open; there are no
+live updates. Accounts are made on the fedipod.net front page, with a pod, not
+from an app.
+
+**Notifications on your phone.** An app that offers notifications can sign up
+for them here. A mention, a reply, a direct message, a follow, or a like, boost
+or reaction to your own post then reaches your phone as it arrives, while
+FediPod and the app are closed. While FediPod is open in a browser, it reads
+your mail itself, and those arrivals are not pushed.
+
 ## Accounts that go quiet
 
-A gateway holds no mail. Every delivery it accepts is written into your pod
-inbox, and your agent reads it from there. A BrowserAgent reads only while
+Every delivery the gateway accepts ends up in your pod inbox, and your agent
+reads it from there. A BrowserAgent reads only while
 its page is open, so an account nobody opens grows on its pod without limit
 and comes back to a drain of everything at once. fedipod.net keeps two facts
 about each browser account — when its owner last signed in or posted, and
@@ -194,9 +248,16 @@ web address becomes a link.
 
 It reads only public data to decide what concerns you: your published
 followers and following, and a small public policy document your agent writes
-with a mirror of your blocklist. Nothing private leaves your pod. Publishing
-that mirror does make your blocklist public, which is part of the bargain of
-running behind a door.
+with a mirror of your blocklist. Publishing that mirror does make your
+blocklist public, which is part of the bargain of running behind a door.
+
+Three things go further, all for browser accounts only. Mail that waits at
+the gateway while your app is closed, direct messages included, is stored there
+until it reaches your pod. A gateway keeping your account running can read and
+write everything in your FediPod folder, your signing key included, while it
+works; nothing else on your pod is open to it. And it keeps your account's
+working data — your timeline, notifications and direct messages among it — for
+as long as it keeps your account running.
 
 ## Running a gateway
 

package/lib/client/masto/bridge.mjs

@@ -0,0 +1,61 @@
+// bridge.mjs — a fetch Request handed to code written for Node's request and
+// response: the Mastodon facade and the admin surface. The browser worker
+// answers its own client through it (web/app/sw-src.mjs), and the gateway
+// answers Mastodon apps through it (lib/gateway/masto-gateway.mjs).
+//
+// Returns { req, res, bodyBytes, bodyText, response() }: `response()` is what
+// the handler wrote, as a fetch Response.
+export async function bridge(request, url, { sameOrigin = false } = {}) {
+  // Bytes, not text. A multipart upload is binary — read as text it comes back
+  // through a UTF-8 round trip that replaces every byte that is not valid UTF-8,
+  // which is most of a JPEG, so the boundary search found nothing and every
+  // media and avatar upload answered "422 file required". readBody() does
+  // `data += chunk`, which decodes a Buffer the same way it always did, so the
+  // JSON and form paths are unchanged.
+  const bodyBytes = (request.method === 'GET' || request.method === 'HEAD')
+    ? null : Buffer.from(new Uint8Array(await request.arrayBuffer()));
+  const bodyText = bodyBytes ? new TextDecoder().decode(bodyBytes) : '';
+  const reqHeaders = {}; for (const [k, v] of request.headers) reqHeaders[k.toLowerCase()] = v;
+  // `host` is a forbidden header name, so a fetch Request never carries one and
+  // the loop above cannot produce it — but it is a header every real request
+  // arrives with, and the agent reads it to say where it lives. Without it the
+  // notifications `Link` header named `https://undefined/`, so a client that
+  // paged by following it (which is how a client is meant to page) walked off
+  // the origin and saw nothing past the first screen.
+  reqHeaders.host = url.host;
+  const listeners = {};
+  // The body's events fire on the next microtask. A route that reads the body
+  // may only register for them after an await or two — the facade dispatches
+  // through its area modules first — so a listener that arrives after the
+  // events have fired is given them at once, in the order it asks.
+  let fired = false;
+  const req = { method: request.method, url: url.pathname + url.search, headers: reqHeaders,
+    // Whether the caller is a page on the agent's own origin. The browser
+    // worker has established that before anything reaches the facade, and
+    // MastoApi asks (through its authorities) whether a request is the owner's
+    // own; a request at the gateway never is.
+    sameOrigin,
+    socket: { encrypted: url.protocol === 'https:' },
+    on(ev, cb) {
+      (listeners[ev] ||= []).push(cb);
+      if (fired) { if (ev === 'data' && bodyBytes?.length) cb(bodyBytes); else if (ev === 'end') cb(); }
+      return req;
+    },
+    destroy() {} };
+  queueMicrotask(() => {
+    fired = true;
+    if (bodyBytes?.length) (listeners.data || []).forEach((cb) => cb(bodyBytes));
+    (listeners.end || []).forEach((cb) => cb());
+  });
+  let status = 200; const outHeaders = {}; const chunks = [];
+  const res = {
+    writeHead(s, h) { status = s; if (h) Object.assign(outHeaders, h); return res; },
+    setHeader(k, v) { outHeaders[k] = v; }, getHeader(k) { return outHeaders[k]; },
+    write(c) { chunks.push(c); }, end(c) { if (c) chunks.push(c); },
+  };
+  return {
+    req, res, bodyBytes, bodyText,
+    get status() { return status; },
+    response: () => new Response(chunks.join(''), { status, headers: { 'content-type': 'application/json', ...outHeaders } }),
+  };
+}

package/lib/client/masto/index.mjs

@@ -149,7 +149,10 @@ export class MastoApi {
     // A viewer-mode agent (another agent holds the drain lease) may not act —
     // but a user acting HERE outranks the idle active agent elsewhere, so a
     // write attempt claims the lease and proceeds. Only a failed claim 503s.
-    if (this.agent.viewer && req.method !== 'GET' && req.method !== 'HEAD') {
+    // An agent working from a copy its gateway shares (the browser build,
+    // web/app/copy-mode.mjs) asks every time, since an app at the gateway may
+    // have taken the lease without it hearing yet.
+    if ((this.agent.viewer || this.agent.copy) && req.method !== 'GET' && req.method !== 'HEAD') {
       const took = await this.agent.requestTakeover?.();
       if (!took) return send(503, { error: 'another agent is active for this pod — takeover failed, try again' });
     }

package/lib/client/masto/oauth.mjs

@@ -45,7 +45,7 @@ const parseRedirects = (v) => (Array.isArray(v) ? v : String(v || '').split(/\s+
 // The sign-in page for an account whose owner signs in at their pod: no
 // password of ours exists, and none is asked for. The page carries the
 // client's request in its address, signs the owner in at the pod with the
-// fediverse-account library, and proves that sign-in back to /oauth/authorize.
+// fediverse-session library, and proves that sign-in back to /oauth/authorize.
 function sendPodSigninPage(res, client, webId, mount = '') {
   let asking = '';
   if (client && (client.name || client.redirect)) {

package/lib/client/masto/statuses.mjs

@@ -112,6 +112,9 @@ export async function handle(api, ctx) {
       };
       sched.push(entry);
       api.store.setScheduled(sched);
+      // Whatever keeps the time for this agent hears of it (a browser app
+      // tells its gateway, which publishes it if the app has closed by then).
+      api.agent.onScheduled?.();
       return send(200, api.scheduledJson(entry));
     }
     let note;

package/lib/connections/acctfeed.mjs

@@ -55,18 +55,22 @@ export class AcctFeed {
     });
   }
 
-  start() {
+  // `lastSweptAt` and `onSwept`: see TagFeed.start — a restarted browser
+  // worker sweeps only when the interval is due.
+  start({ lastSweptAt = 0, onSwept = null } = {}) {
     this.stopped = false;
-    this.sweep().catch(e => this.log(`acctfeed: ${e.message}`));
-    const tick = () => {
-      this.timer = setTimeout(() => {
-        this.sweep()
-          .catch(e => this.log(`acctfeed: ${e.message}`))
-          .finally(() => { if (!this.stopped) tick(); });
-      }, Math.round(this.config().intervalMin * 60_000 * (0.85 + Math.random() * 0.3)));
+    const sweep = () => this.sweep()
+      .catch(e => this.log(`acctfeed: ${e.message}`))
+      .finally(() => onSwept?.(Date.now()));
+    const every = () => Math.round(this.config().intervalMin * 60_000 * (0.85 + Math.random() * 0.3));
+    const tick = (ms) => {
+      this.timer = setTimeout(() => { sweep().finally(() => { if (!this.stopped) tick(every()); }); }, ms);
       this.timer.unref?.();
     };
-    tick();
+    const due = lastSweptAt + this.config().intervalMin * 60_000 - Date.now();
+    if (due > 0) { tick(due); return; }
+    sweep();
+    tick(every());
   }
 
   stop() { this.stopped = true; clearTimeout(this.timer); }

package/lib/connections/bskyfeed.mjs

@@ -52,18 +52,22 @@ export class BskyFeed {
     return this.config();
   }
 
-  start() {
+  // `lastSweptAt` and `onSwept`: see TagFeed.start — a restarted browser
+  // worker sweeps only when the interval is due.
+  start({ lastSweptAt = 0, onSwept = null } = {}) {
     this.stopped = false;
-    this.sweep().catch(e => this.log(`bskyfeed: ${e.message}`));
-    const tick = () => {
-      this.timer = setTimeout(() => {
-        this.sweep()
-          .catch(e => this.log(`bskyfeed: ${e.message}`))
-          .finally(() => { if (!this.stopped) tick(); });
-      }, Math.round(this.config().intervalMin * 60_000 * (0.85 + Math.random() * 0.3)));
+    const sweep = () => this.sweep()
+      .catch(e => this.log(`bskyfeed: ${e.message}`))
+      .finally(() => onSwept?.(Date.now()));
+    const every = () => Math.round(this.config().intervalMin * 60_000 * (0.85 + Math.random() * 0.3));
+    const tick = (ms) => {
+      this.timer = setTimeout(() => { sweep().finally(() => { if (!this.stopped) tick(every()); }); }, ms);
       this.timer.unref?.();
     };
-    tick();
+    const due = lastSweptAt + this.config().intervalMin * 60_000 - Date.now();
+    if (due > 0) { tick(due); return; }
+    sweep();
+    tick(every());
   }
 
   stop() { this.stopped = true; clearTimeout(this.timer); }

package/lib/connections/tagfeed.mjs

@@ -72,20 +72,25 @@ export class TagFeed {
     return this.config();
   }
 
-  start() {
+  // `lastSweptAt` and `onSwept` are for a browser worker the browser stops and
+  // restarts: it keeps the time of its last sweep, so a restart sweeps only
+  // when the interval is due rather than on every wake.
+  start({ lastSweptAt = 0, onSwept = null } = {}) {
     this.stopped = false;                     // restartable, the same way Intake.start is
-    this.sweep().catch(e => this.log(`tagfeed: ${e.message}`));
+    const sweep = () => this.sweep()
+      .catch(e => this.log(`tagfeed: ${e.message}`))
+      .finally(() => onSwept?.(Date.now()));
     // Jittered and self-scheduling: every agent polling the same instance on
     // the same 15-minute boundary is a beat nobody asked for.
-    const tick = () => {
-      this.timer = setTimeout(() => {
-        this.sweep()
-          .catch(e => this.log(`tagfeed: ${e.message}`))
-          .finally(() => { if (!this.stopped) tick(); });
-      }, Math.round(this.config().intervalMin * 60_000 * (0.85 + Math.random() * 0.3)));
+    const every = () => Math.round(this.config().intervalMin * 60_000 * (0.85 + Math.random() * 0.3));
+    const tick = (ms) => {
+      this.timer = setTimeout(() => { sweep().finally(() => { if (!this.stopped) tick(every()); }); }, ms);
       this.timer.unref?.();
     };
-    tick();
+    const due = lastSweptAt + this.config().intervalMin * 60_000 - Date.now();
+    if (due > 0) { tick(due); return; }
+    sweep();
+    tick(every());
   }
 
   // The flag is what makes this stick. Clearing the timer only cancels a sweep