switchroom 0.19.26 → 0.19.27

package/telegram-plugin/edit-flood-fuse.ts

@@ -56,11 +56,12 @@
  *   - send gate `editFloorMs` = 1500ms          ⇒ up to 40 edits/min/message
  *   - send gate cosmetic budget 150 / 300s      ⇒ 30 edits/min/message
  *   - worker feed's own floor 2500ms            ⇒ 24 edits/min/message
+ *   - gateway `FEED_HEARTBEAT_TICK_MS` = 6000ms ⇒ 10 repaints/min/turn
  *
- * The per-message ceiling here is 20/60s, i.e. BELOW all three. That is
- * deliberate — Telegram's own per-group limit is ~20 messages/minute and
- * edits count against it, so the *legitimate* cadences are themselves above
- * what Telegram tolerates; that is precisely how `overlord` got banned while
+ * The fuse's cosmetic ceilings sit BELOW all of these. That is deliberate —
+ * Telegram's own per-group limit is ~20 messages/minute and edits count
+ * against it, so the *legitimate* cadences are themselves above what
+ * Telegram tolerates; that is precisely how `overlord` got banned while
  * every in-repo pacer believed it was behaving. The fuse is therefore the
  * BINDING constraint on a hot card, not a never-reached backstop.
  *
@@ -74,12 +75,112 @@
  *     RELEASED late instead of dropped. Dropping it would freeze the card
  *     mid-run AND return `true` to the send gate, which would then record a
  *     never-painted payload as on-screen and no-op-skip every retry.
+ *
+ * ── 2026-07-27: the fuse existed and the ban happened anyway ──────────────
+ * The ceilings above were sized against the in-repo pacers, not against what
+ * Telegram actually tolerates. Agent `overlord` then sustained ~17
+ * `editMessageText`/min on ONE DM chat for hours (326 edits against 6 real
+ * replies in the final 20 minutes) and took a **15908-second** flood ban that
+ * severed every outbound reply. Both original ceilings — 20 edits/60s per
+ * message, 30 edits/60s per chat — sat ABOVE that observed rate, so the fuse
+ * never bound once. Three structural changes follow, and each is a separate,
+ * independently-sufficient reason the same incident cannot recur:
+ *
+ *   1. **Class awareness.** The fuse now reads the send gate's priority class
+ *      off `outbound-class.ts` (AsyncLocalStorage). `cosmetic` traffic — every
+ *      activity/worker/liveness repaint — is governed by its own, much tighter
+ *      ceilings; `useful` / `critical` edits (approval cards, answers) are not
+ *      throttled by them. Previously the fuse was class-blind and had to pick
+ *      one number for both, which is why the number was too high.
+ *   2. **A hard cosmetic rate ceiling.** 4 edits/60s per message (one per 15s,
+ *      matching the worker feed's own `elapsedRefreshMs`) and 6 edits/60s per
+ *      chat across ALL cosmetic surfaces. 6/min is below the ~4-6/min band the
+ *      live chat survived for hours and far below the 15-17/min that earned
+ *      the ban. Coalescing (supersede) means the card still shows CURRENT
+ *      state at every permitted edit — the operator loses refresh frequency,
+ *      never accuracy.
+ *   3. **A shared per-chat budget with a reply reservation.** Telegram meters
+ *      sends and edits against the SAME per-chat allowance, so separate
+ *      edit/send windows could each be "in budget" while their sum was not.
+ *      One `perChatTotalMaxPerWindow` window (default 20/60s) counts every
+ *      chat-targeted call, and `perChatReplyReserve` (default 8) slots of it
+ *      are unreachable by `cosmetic` traffic. A reply therefore cannot be
+ *      starved by repaints even if a future call site is misclassified.
+ *
+ * ── 2026-07-28: the meter was reading ~58% of the traffic (#3855) ─────────
+ * Point 3 above USED to claim `perChatTotalMaxPerWindow` "counts every
+ * admitted call" and "matches Telegram's own per-chat metering". Both were
+ * false, and the gap is the reason a "20/60s" ceiling did not stop a ban.
+ * `apply` opened with `if (!isEdit && !isSend) return runObserved(next)` —
+ * an ALLOWLIST. Anything outside `EDIT_METHODS`/`SEND_METHODS` was entirely
+ * unmetered: it took no slot, and the ceiling therefore governed only the
+ * subset of traffic it happened to recognise.
+ *
+ * Measured on overlord's own gateway log (46791 `tg-post` lines), the
+ * unmetered share was 29% overall and 42% inside the 30 minutes before the
+ * 20:19:56 ban. By method:
+ *
+ *     sendChatAction      10457   ← unmetered, and up to 18/min on ONE DM
+ *     setMessageReaction   2098   ← unmetered
+ *     deleteMessage         370   ← unmetered
+ *     pinChatMessage        366   ← unmetered
+ *     unpinChatMessage      365   ← unmetered
+ *     answerCallbackQuery    31   ← unmetered (carries no chat_id)
+ *
+ * `sendChatAction` alone ran at Telegram's entire documented per-chat/minute
+ * allowance while the fuse counted none of it. So the fix is not a smaller
+ * number, it is an honest meter:
+ *
+ *   4. **DEFAULT-DENY metering.** Every method whose payload carries a
+ *      `chat_id` is charged to `perChatTotalMaxPerWindow`, whether or not this
+ *      file has heard of it. There is no per-chat exemption list to forget to
+ *      update — a new Bot API method, or one this gateway starts calling
+ *      tomorrow (`sendChecklist` is exactly that case: it carries `chat_id`,
+ *      creates a message, and was in neither set), is metered on arrival.
+ *      Methods that are neither edits nor sends are PACED and never dropped,
+ *      except `CHAT_ACTION_METHODS` — a typing indicator expires by itself in
+ *      ~5s and carries no content, so shedding one loses nothing.
+ *      The ceiling NUMBER is unchanged at 20/60s; what changed is that 20
+ *      admitted calls now means 20 on the wire rather than ~34.
+ *   5. **A per-BOT-TOKEN window.** Telegram's flood ban is scoped to the bot
+ *      token, not the chat, and enforces a global rate across all chats. A
+ *      second window keyed by the bot's PUBLIC numeric id (`perTokenWindowMs`
+ *      / `perTokenMaxPerWindow`, default 25 per 1000ms) is now charged for
+ *      EVERY outbound call — including the chat-less ones like
+ *      `answerCallbackQuery`, which the per-chat tier structurally cannot key.
+ *      A call is admitted only if BOTH windows have room.
+ *      LIMITATION, stated plainly: this window is per-PROCESS. Two agents
+ *      sharing one bot token run in separate containers with separate state
+ *      dirs, so neither sees the other's window. It is a real improvement on
+ *      the single process that actually floods and it is NOT full coverage
+ *      under a shared token; a cross-process view would need a token-keyed
+ *      store on a path both processes can write, which does not exist today.
+ *      Only the public bot id (the prefix before the `:`) is ever used as a
+ *      key, and it is never logged — see {@link deriveBotScopeKey}.
+ *
+ * Plus **escalating** backoff: a 429 used to apply ONE 0.5× tightening for a
+ * flat 10 minutes, so a stream that took repeated small 429s re-tightened to
+ * the same level forever and walked into the big ban. Tightening is now
+ * multiplicative PER 429 (`tightenFactor ^ level`, level capped by
+ * `maxTightenLevel`) and decays one level at a time, so sustained pressure
+ * ratchets the cosmetic rate down towards the floor of 1/window.
+ *
+ * Every ceiling is operator-overridable via env — see
+ * {@link editFloodFuseConfigFromEnv}. Previously none of them were.
  */
 
+import { createHash } from 'node:crypto'
+
 import type { Bot } from 'grammy'
 
 import type { Clock } from './send-gate.js'
 import { systemClock } from './send-gate.js'
+import { floodStatePath, makeFloodWaitProbe } from './flood-circuit-breaker.js'
+import {
+  currentOutboundClass,
+  defaultOutboundClass,
+  type OutboundClass,
+} from './outbound-class.js'
 
 /** Methods that mutate an EXISTING message — droppable, coalescible. */
 const EDIT_METHODS: ReadonlySet<string> = new Set([
@@ -112,26 +213,160 @@ const SEND_METHODS: ReadonlySet<string> = new Set([
   'sendRichMessage',
 ])
 
+/**
+ * Chat-targeted methods whose effect EXPIRES on its own and carries no content,
+ * so shedding one under pressure loses nothing a user would notice.
+ *
+ * `sendChatAction` is the entire set and the reason this tier exists: it was
+ * the single largest unmetered method in the incident log (10457 calls, up to
+ * 18/min on ONE DM — Telegram's whole documented per-chat/minute allowance),
+ * and the typing status it sets clears itself after ~5 seconds regardless.
+ * Every OTHER non-edit, non-send method with a `chat_id` (`deleteMessage`,
+ * `pinChatMessage`, `setMessageReaction`, …) has a durable, user-visible
+ * effect and is paced but never dropped.
+ */
+const CHAT_ACTION_METHODS: ReadonlySet<string> = new Set(['sendChatAction'])
+
+/**
+ * The ONLY methods exempt from metering entirely. Deliberately one entry.
+ *
+ * `getUpdates` is the long-poll RECEIVE loop, not outbound traffic: it is how
+ * inbound messages arrive at all. Pacing it would stall message delivery
+ * rather than protect anything, and it targets no chat, so it consumes no
+ * per-chat budget by construction.
+ *
+ * Nothing else is listed. The one-off administrative calls (`setMyCommands`,
+ * `deleteWebhook`, `getFile`, `getChat` — 37 calls TOTAL across a 46791-call
+ * log) are metered like everything else: at 25/second the cost is nil, and
+ * exempting them would mean asserting something about Telegram's accounting
+ * that this repo cannot verify from first-party evidence. Default-deny means
+ * the burden of proof is on the exemption.
+ */
+const UNMETERED_METHODS: ReadonlySet<string> = new Set(['getUpdates'])
+
+/**
+ * Reduce a bot token to a key safe to hold in memory.
+ *
+ * A Telegram token is `<bot_id>:<secret>`. The bot id is PUBLIC (it is the
+ * bot's user id, visible on every message it sends); the secret half is not.
+ * This returns the bot id alone when the token has that shape, so nothing
+ * secret is ever used as a map key, logged, or persisted. A token that does
+ * not match the shape falls back to a truncated SHA-256, which is stable
+ * across processes and irreversible.
+ */
+export function deriveBotScopeKey(token: string | undefined): string {
+  if (token == null || token === '') return 'unknown'
+  const id = token.slice(0, token.indexOf(':'))
+  if (/^\d+$/.test(id)) return id
+  return `h${createHash('sha256').update(token).digest('hex').slice(0, 16)}`
+}
+
 export interface EditFloodFuseConfig {
   /** Master switch. When false, `apply` is a pure passthrough. Default true. */
   enabled?: boolean
   clock?: Clock
-  /** Hard ceiling on edits to ONE message id. Default 20 per 60s. */
+  /**
+   * Hard ceiling on NON-cosmetic (`useful` / `critical`) edits to ONE message
+   * id. Default 20 per 60s. Approval cards and answer finalisations live here;
+   * they are low-cadence by nature, so this stays a backstop, not a pacer.
+   */
   perMessageMaxPerWindow?: number
   perMessageWindowMs?: number
-  /** Hard ceiling on edits to ONE chat across all messages. Default 30 per 60s. */
+  /** Hard ceiling on ALL edits to ONE chat across all messages. Default 30 per 60s. */
   perChatEditMaxPerWindow?: number
   /** Pacing ceiling on non-edit sends to ONE chat. Default 25 per 60s. */
   perChatSendMaxPerWindow?: number
+  /**
+   * ADDITIONAL ceiling applied to COSMETIC edits of ONE message id, on top of
+   * `perMessageMaxPerWindow` (the effective ceiling is the lower of the two).
+   * Default 4 per 60s — one per 15s, matching the worker feed's
+   * `elapsedRefreshMs`. This is the number that bounds a hot progress card.
+   */
+  cosmeticPerMessageMaxPerWindow?: number
+  /**
+   * ADDITIONAL ceiling applied to COSMETIC edits in ONE chat, summed across
+   * every cosmetic surface in it (lower of this and `perChatEditMaxPerWindow`
+   * binds). Default 6 per 60s. Without it, N concurrent cards each inside their
+   * own per-message ceiling still add up to a flood.
+   */
+  cosmeticPerChatMaxPerWindow?: number
+  /**
+   * Shared per-chat allowance counting EVERY admitted call — edits and sends,
+   * every class. Default 20 per 60s, matching Telegram's own per-chat/group
+   * metering (which does not separate the two). Sends and non-cosmetic edits
+   * are paced against it and never dropped.
+   */
+  perChatTotalMaxPerWindow?: number
+  /**
+   * Slots of `perChatTotalMaxPerWindow` that `cosmetic` traffic may NEVER
+   * consume. Default 8. This is the reply-starvation guarantee: whatever the
+   * repaint surfaces do, at least this many calls per window remain available
+   * to an actual answer.
+   */
+  perChatReplyReserve?: number
+  /**
+   * Ceiling on EVERY outbound call made with this bot token, across all chats
+   * and including the chat-less ones. Default 25 per `perTokenWindowMs`
+   * (1000ms), just under the ~30/second Telegram enforces at the token level.
+   * The flood ban is token-scoped, so this is the tier that matches the thing
+   * that actually gets banned. Per-PROCESS — see the docblock's point 5.
+   */
+  perTokenMaxPerWindow?: number
+  perTokenWindowMs?: number
+  /**
+   * Scope key for the token window: the bot's PUBLIC numeric id. Never pass a
+   * raw token — {@link installEditFloodFuse} derives this via
+   * {@link deriveBotScopeKey}. Default `'unknown'` (a single shared window,
+   * which is still correct for the one-bot-per-process case).
+   */
+  botScopeKey?: string
+  /**
+   * Longest a `CHAT_ACTION_METHODS` call may be held before it is shed.
+   * Default 3000ms — a typing indicator that arrives after the status it
+   * describes has already expired is worse than no typing indicator.
+   */
+  chatActionMaxDeferMs?: number
+  /**
+   * Cap on COSMETIC edits that may be released late (over budget) because
+   * nothing newer will repaint them — the bounded form of the R1 rule. Default
+   * 2 per `perChatWindowMs`, so the worst-case sustained cosmetic rate is
+   * `cosmeticPerChatMaxPerWindow + lateReleaseMaxPerWindow`.
+   */
+  lateReleaseMaxPerWindow?: number
   perChatWindowMs?: number
   /** Longest a call may be held before it is dropped (edit) / released (send). Default 30s. */
   maxDeferMs?: number
-  /** Multiplicative decrease applied to every ceiling after a 429. Default 0.5. */
+  /** Multiplicative decrease applied per observed 429. Default 0.5. */
   tightenFactor?: number
-  /** How long a tightened ceiling stays in force. Default 10 minutes. */
+  /** How long ONE level of tightening stays in force before decaying. Default 10 minutes. */
   tightenMs?: number
+  /** Cap on compounding 429 tightening levels. Default 4 (⇒ 0.5^4 = 1/16). */
+  maxTightenLevel?: number
+  /**
+   * Remaining ms of a KNOWN-OPEN Telegram flood window, or 0 when none — the
+   * persisted breaker state (`flood-circuit-breaker.ts`) that every other
+   * outbound path already consults. Wired by {@link editFloodFuseConfigFromEnv}.
+   *
+   * The fuse's tightening lived only in process memory, so before #3856 a
+   * gateway restart during a ban came back FULLY UNTIGHTENED and resumed at
+   * the rate that earned the ban — and a restart is the single most likely
+   * thing to happen during a multi-hour outage. Consulting the persisted
+   * window makes the response durable: while it is open the fuse holds
+   * `maxTightenLevel` regardless of what this process remembers.
+   *
+   * FAILS OPEN (throwing or absent probe ⇒ untightened): a breaker that cannot
+   * read its own state must never gag the bot.
+   */
+  floodWaitRemainingMs?: () => number
+  /** How often the persisted window may be re-read. Default 1000ms. */
+  floodProbeIntervalMs?: number
   /** Observability hook; fired whenever the fuse binds. */
-  onTrip?: (info: { method: string; key: string; action: 'deferred' | 'dropped' | 'superseded' }) => void
+  onTrip?: (info: {
+    method: string
+    key: string
+    action: 'deferred' | 'dropped' | 'superseded'
+    cls: OutboundClass
+  }) => void
 }
 
 export interface EditFloodFuseStats {
@@ -146,8 +381,29 @@ export interface EditFloodFuseStats {
   floodObserved: number
   /** Whether a tightened ceiling is in force right now. */
   tightened: boolean
-  /** Live per-message ceiling (post-AIMD). */
+  /** Compounding 429 tightening level currently in force (0 = untightened). */
+  tightenLevel: number
+  /** Live NON-cosmetic per-message ceiling (post-AIMD). */
   perMessageCeiling: number
+  /** Live COSMETIC per-message ceiling (post-AIMD) — the incident-relevant one. */
+  cosmeticPerMessageCeiling: number
+  /** Live COSMETIC per-chat ceiling (post-AIMD). */
+  cosmeticPerChatCeiling: number
+  /**
+   * Chat-targeted calls charged by the DEFAULT-DENY rule — every one of these
+   * was completely unmetered before #3855. A non-zero value here is the direct
+   * measure of the hole this closed.
+   */
+  meteredByDefault: number
+  /** Calls with no `chat_id`, charged to the token window only. */
+  chatless: number
+  /** Live per-bot-token ceiling (post-AIMD). */
+  perTokenCeiling: number
+  /**
+   * True when a PERSISTED flood window is open right now — i.e. the tightening
+   * in force is durable rather than remembered, and survives a restart (#3856).
+   */
+  persistedFloodOpen: boolean
 }
 
 export const EDIT_FLOOD_FUSE_DEFAULTS = {
@@ -155,12 +411,87 @@ export const EDIT_FLOOD_FUSE_DEFAULTS = {
   perMessageWindowMs: 60_000,
   perChatEditMaxPerWindow: 30,
   perChatSendMaxPerWindow: 25,
+  /**
+   * 4/60s. One cosmetic repaint per 15s per card. Chosen to equal the worker
+   * feed's `elapsedRefreshMs` (15000) — the slowest cadence at which the feed
+   * itself considers a repaint worth making — so the fuse binds the runaway
+   * case without ever throttling the feed's own intended pace.
+   */
+  cosmeticPerMessageMaxPerWindow: 4,
+  /**
+   * 6/60s across every cosmetic surface in a chat. The incident's own timeline
+   * is the evidence: 10-minute buckets of 58/53/41 edits (≈4-6/min) ran for
+   * hours without a ban; 115/78/73 then 152/174 (≈8-17/min) earned one. 6/min
+   * sits at the top of the survived band and less than half the banned rate.
+   */
+  cosmeticPerChatMaxPerWindow: 6,
+  /**
+   * 20/60s. Telegram's documented per-group ceiling, and it meters sends and
+   * edits together — so this is the only window that reflects the real budget.
+   */
+  perChatTotalMaxPerWindow: 20,
+  /** 8 of those 20 slots/min are unreachable by cosmetic traffic. */
+  perChatReplyReserve: 8,
+  /**
+   * 25 per second across the whole bot token. Telegram enforces ~30/s at the
+   * token level and the ban it issues is token-scoped; 25 leaves headroom for
+   * traffic this process cannot see (a peer agent sharing the token, or a
+   * retry issued below the transformer stack).
+   */
+  perTokenMaxPerWindow: 25,
+  perTokenWindowMs: 1_000,
+  /** A typing indicator held longer than this is not worth sending. */
+  chatActionMaxDeferMs: 3_000,
+  /** At most 2 cosmetic frames/min may exceed the ceiling as "lone" releases. */
+  lateReleaseMaxPerWindow: 2,
   perChatWindowMs: 60_000,
   maxDeferMs: 30_000,
   tightenFactor: 0.5,
   tightenMs: 600_000,
+  maxTightenLevel: 4,
+  /** The persisted flood window is re-read at most once a second (#3856). */
+  floodProbeIntervalMs: 1_000,
 } as const
 
+/** Parse a positive integer from env, or undefined when unset/invalid. */
+function envInt(raw: string | undefined): number | undefined {
+  if (raw == null || raw.trim() === '') return undefined
+  const n = Number(raw)
+  return Number.isFinite(n) && n >= 0 ? Math.floor(n) : undefined
+}
+
+/**
+ * Operator config surface. Before the 2026-07-27 incident the fuse had exactly
+ * one knob — `SWITCHROOM_EDIT_FUSE=0`, which turns the whole failsafe OFF —
+ * so an operator whose chat was being flooded had no way to tighten it and no
+ * way to loosen it for a chat that could take more. Every ceiling is now
+ * overridable; unset values keep the defaults above.
+ */
+export function editFloodFuseConfigFromEnv(
+  env: Record<string, string | undefined>,
+): EditFloodFuseConfig {
+  const cfg: EditFloodFuseConfig = { enabled: env.SWITCHROOM_EDIT_FUSE !== '0' }
+  const assign = <K extends keyof EditFloodFuseConfig>(k: K, v: number | undefined): void => {
+    if (v !== undefined) (cfg[k] as number) = v
+  }
+  assign('cosmeticPerMessageMaxPerWindow', envInt(env.SWITCHROOM_FEED_EDIT_MAX_PER_MSG_PER_MIN))
+  assign('cosmeticPerChatMaxPerWindow', envInt(env.SWITCHROOM_FEED_EDIT_MAX_PER_CHAT_PER_MIN))
+  assign('perChatTotalMaxPerWindow', envInt(env.SWITCHROOM_CHAT_TOTAL_MAX_PER_MIN))
+  assign('perChatReplyReserve', envInt(env.SWITCHROOM_CHAT_REPLY_RESERVE))
+  assign('perTokenMaxPerWindow', envInt(env.SWITCHROOM_TOKEN_MAX_PER_SEC))
+  assign('maxDeferMs', envInt(env.SWITCHROOM_EDIT_FUSE_MAX_DEFER_MS))
+  // #3856 — durable ban awareness. The fuse reads the SAME persisted marker
+  // (`flood-wait.json`) that `robustApiCall` and the outbox sweep consult, so a
+  // restart mid-ban comes back tightened instead of at full rate. Wired here
+  // rather than at the call site so no caller can forget it, and because
+  // `gateway.ts` is under a zero-slack line ratchet.
+  const stateDir = env.TELEGRAM_STATE_DIR
+  if (stateDir != null && stateDir !== '') {
+    cfg.floodWaitRemainingMs = makeFloodWaitProbe(floodStatePath(stateDir))
+  }
+  return cfg
+}
+
 /** grammY resolves an edit with `true` when there is nothing to return. */
 const DROPPED_RESULT = true
 
@@ -189,28 +520,110 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
   const perMessageWindowMs = config.perMessageWindowMs ?? D.perMessageWindowMs
   const perChatEditMax = config.perChatEditMaxPerWindow ?? D.perChatEditMaxPerWindow
   const perChatSendMax = config.perChatSendMaxPerWindow ?? D.perChatSendMaxPerWindow
+  // Cosmetic tiers are an ADDITIONAL constraint on the same window keys, so the
+  // effective ceiling is the lower of the two. Folding them in here (rather
+  // than as extra tiers) keeps the admission path at the same number of awaits.
+  const cosmeticPerMessageMax = Math.min(
+    perMessageMax, config.cosmeticPerMessageMaxPerWindow ?? D.cosmeticPerMessageMaxPerWindow)
+  const cosmeticPerChatMax = Math.min(
+    perChatEditMax, config.cosmeticPerChatMaxPerWindow ?? D.cosmeticPerChatMaxPerWindow)
+  const perChatTotalMax = config.perChatTotalMaxPerWindow ?? D.perChatTotalMaxPerWindow
   const perChatWindowMs = config.perChatWindowMs ?? D.perChatWindowMs
+  // Clamped so a misconfigured reserve can never make the cosmetic allowance
+  // negative (deadlocking every card) or eat the whole budget.
+  const perChatReplyReserve = Math.min(
+    Math.max(0, config.perChatReplyReserve ?? D.perChatReplyReserve),
+    Math.max(0, perChatTotalMax - 1),
+  )
+  const perTokenMax = Math.max(1, config.perTokenMaxPerWindow ?? D.perTokenMaxPerWindow)
+  const perTokenWindowMs = Math.max(1, config.perTokenWindowMs ?? D.perTokenWindowMs)
+  // Only ever the PUBLIC bot id (or an irreversible hash) reaches this key.
+  const tokenKey = `g:${config.botScopeKey ?? 'unknown'}`
+  const chatActionMaxDeferMs = config.chatActionMaxDeferMs ?? D.chatActionMaxDeferMs
+  const lateReleaseMax = Math.max(0, config.lateReleaseMaxPerWindow ?? D.lateReleaseMaxPerWindow)
   const maxDeferMs = config.maxDeferMs ?? D.maxDeferMs
   const tightenFactor = config.tightenFactor ?? D.tightenFactor
   const tightenMs = config.tightenMs ?? D.tightenMs
+  const maxTightenLevel = Math.max(0, config.maxTightenLevel ?? D.maxTightenLevel)
+  const floodProbe = config.floodWaitRemainingMs
+  const floodProbeIntervalMs = Math.max(0, config.floodProbeIntervalMs ?? D.floodProbeIntervalMs)
   const onTrip = config.onTrip
 
   const windows = new Map<string, Window>()
-  const counters = { deferred: 0, dropped: 0, superseded: 0, floodObserved: 0 }
-  /** Timestamp until which the AIMD tightening is in force (0 = untightened). */
+  const counters = {
+    deferred: 0, dropped: 0, superseded: 0, floodObserved: 0,
+    /** Chat-targeted calls charged by the DEFAULT-DENY rule that the old
+     *  allowlist would have let through unmetered (#3855 observability). */
+    meteredByDefault: 0,
+    /** Calls with no `chat_id`, charged to the token window only. */
+    chatless: 0,
+  }
+  /**
+   * Compounding 429 backoff. `tightenLevel` is the number of multiplicative
+   * decreases currently in force; `tightenedUntil` is when the NEXT level
+   * decays. A single flat tightening (the pre-2026-07-27 behaviour) let a
+   * stream that kept taking small 429s sit at 0.5× indefinitely and walk into
+   * a long ban; escalating means repeated 429s ratchet the rate down.
+   */
+  let tightenLevel = 0
   let tightenedUntil = 0
 
+  /**
+   * Cached read of the PERSISTED flood window (#3856). Throttled to
+   * `floodProbeIntervalMs` because `levelAt` runs on every admission and the
+   * probe is a file read — an unthrottled read would make the fuse the latency
+   * problem it exists to prevent. Between reads the cached remaining time is
+   * decayed by elapsed wall time, so a window never appears to last longer
+   * than it does.
+   *
+   * FAILS OPEN: any throw is treated as "no window". `makeFloodWaitProbe`
+   * already fails open on an unreadable marker and warns loudly (#3106); this
+   * catch covers the rest.
+   */
+  let probedAt = Number.NEGATIVE_INFINITY
+  let probedRemainingMs = 0
+  function persistedFloodRemainingMs(now: number): number {
+    if (floodProbe === undefined) return 0
+    const since = now - probedAt
+    if (since < floodProbeIntervalMs) return Math.max(0, probedRemainingMs - since)
+    probedAt = now
+    try {
+      const ms = floodProbe()
+      probedRemainingMs = Number.isFinite(ms) && ms > 0 ? ms : 0
+    } catch {
+      probedRemainingMs = 0
+    }
+    return probedRemainingMs
+  }
+
+  /** Decay one level per `tightenMs` of quiet, rather than a single cliff. */
+  function levelAt(now: number): number {
+    while (tightenLevel > 0 && tightenedUntil <= now) {
+      tightenLevel--
+      tightenedUntil = tightenLevel > 0 ? tightenedUntil + tightenMs : 0
+    }
+    // #3856 — a KNOWN-OPEN persisted window pins the fuse at its tightest
+    // regardless of in-memory state. This is what survives a restart: the
+    // process forgets, the marker on disk does not. It does not MUTATE
+    // `tightenLevel`, so when the window closes the fuse returns to whatever
+    // this process actually earned rather than staying pinned.
+    if (persistedFloodRemainingMs(now) > 0) return maxTightenLevel
+    return tightenLevel
+  }
+
   function isTightened(now: number): boolean {
-    return tightenedUntil > now
+    return levelAt(now) > 0
   }
 
   /**
-   * AIMD multiplicative decrease. Applied to every ceiling while tightened;
-   * floored at 1 so the fuse never deadlocks a surface completely.
+   * AIMD multiplicative decrease, compounding per observed 429. Applied to
+   * every ceiling while tightened; floored at 1 so the fuse never deadlocks a
+   * surface completely.
    */
   function ceiling(base: number, now: number): number {
-    if (!isTightened(now)) return base
-    return Math.max(1, Math.floor(base * tightenFactor))
+    const level = levelAt(now)
+    if (level === 0) return base
+    return Math.max(1, Math.floor(base * Math.pow(tightenFactor, level)))
   }
 
   function win(key: string): Window {
@@ -266,23 +679,69 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
     return { chat, msg }
   }
 
-  /** Record a 429 (whatever shape it arrives in) and tighten. */
-  function noteFlood(now: number): void {
+  /**
+   * Record a 429 and tighten IN PROPORTION TO THE PENALTY (#3856).
+   *
+   * Before this, every 429 was worth exactly one level: a 3-second "slow down
+   * a touch" nudge and a **15908-second** ban produced the identical response.
+   * The whole point of AIMD is that the decrease matches the signal, and a
+   * four-hour ban is not one nudge's worth of signal. `retryAfterSec` is the
+   * severity Telegram itself states, so it is what the step is scaled by.
+   *
+   * The tightening is also held for at least the ban's own duration. A flat
+   * 10-minute `tightenMs` expired ~25× over during the 2026-07-27 ban, so the
+   * fuse would have been back at full rate long before the window closed.
+   */
+  function noteFlood(now: number, retryAfterSec: number): void {
     counters.floodObserved++
-    tightenedUntil = now + tightenMs
+    levelAt(now)
+    tightenLevel = Math.min(maxTightenLevel, tightenLevel + tightenStepFor(retryAfterSec))
+    // Every level currently in force is re-armed; the decay clock restarts
+    // from the newest 429, and never expires before the penalty itself does.
+    tightenedUntil = now + Math.max(tightenMs, retryAfterSec * 1000 + tightenMs)
   }
 
-  function looksLikeFlood(err: unknown): boolean {
+  /**
+   * Levels of multiplicative decrease one 429 is worth, by stated penalty:
+   *
+   *   ≤ 5s     1   a routine burst nudge
+   *   ≤ 60s    2   sustained overrate
+   *   ≤ 600s   3   a real ban
+   *   > 600s   ALL the way to `maxTightenLevel` — at this magnitude the rate
+   *               that produced it is categorically wrong, and stepping down
+   *               one level at a time would spend the next window earning the
+   *               next ban. The 2026-07-25 (3713s) and 2026-07-27 (15908s)
+   *               bans are both in this band.
+   */
+  function tightenStepFor(retryAfterSec: number): number {
+    if (!Number.isFinite(retryAfterSec) || retryAfterSec <= 5) return 1
+    if (retryAfterSec <= 60) return 2
+    if (retryAfterSec <= 600) return 3
+    return maxTightenLevel
+  }
+
+  /**
+   * The stated `retry_after` in seconds when `err` is a flood rejection, else
+   * null. Returns 0 for a flood whose magnitude is not stated (the text-match
+   * path), which `tightenStepFor` treats as the mildest case — an unquantified
+   * signal must not be inflated into a maximal response.
+   */
+  function floodRetryAfterSec(err: unknown): number | null {
     const e = err as { error_code?: number; parameters?: { retry_after?: number } } | null
     if (e != null && typeof e === 'object') {
-      if (e.error_code === 429) return true
-      if (e.parameters != null && typeof e.parameters.retry_after === 'number') return true
+      const stated = e.parameters?.retry_after
+      if (typeof stated === 'number') return stated
+      if (e.error_code === 429) return 0
     }
     // Match on the SEMANTIC markers only. A bare "429" substring
     // false-positives on ordinary server text (e.g. "message 429 not found"),
     // and a false positive here halves every ceiling for ten minutes.
     const msg = err instanceof Error ? err.message : String(err ?? '')
-    return /too many requests/i.test(msg) || /retry[ _-]?after/i.test(msg)
+    if (!/too many requests/i.test(msg) && !/retry[ _-]?after/i.test(msg)) return null
+    // grammY and the Bot API both surface the magnitude in the text; use it
+    // when it is there rather than throwing the severity away.
+    const m = /retry[ _-]?after[^0-9]{0,4}(\d+)/i.exec(msg)
+    return m != null ? Number(m[1]) : 0
   }
 
 /**
@@ -311,7 +770,7 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
    * Returns the reserved timestamp, or null when the call must be dropped.
    */
   async function awaitRoom(
-    key: string, windowMs: number, max: number, method: string, mode: WaitMode,
+    key: string, windowMs: number, max: number, method: string, mode: WaitMode, cls: OutboundClass,
     /**
      * Consulted ONLY at the defer deadline for `mode: 'drop'`. Returning false
      * converts the drop into a late release: this call is the last thing that
@@ -319,10 +778,30 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
      * frame", it is freezing a card. Absent ⇒ drop unconditionally (the
      * pre-review behaviour).
      */
-    dropGuard?: () => boolean,
+    dropGuard: (() => boolean) | undefined,
+    /**
+     * Absolute deadline SHARED by every tier of one `apply` call. Each tier
+     * used to start its own `maxDeferMs`, so a call crossing three tiers could
+     * be held 3×`maxDeferMs` — an unbounded-in-practice hold that a caller's
+     * own timeout, not this fuse, would have to end. One deadline per call
+     * makes `maxDeferMs` mean what it says.
+     */
+    deadline: number,
+    /**
+     * Bounded overshoot budget for the R1 late-release path (cosmetic edits
+     * only). R1 says an over-budget edit that nothing newer will repaint must
+     * be RELEASED late rather than dropped, so a card cannot freeze mid-run.
+     * Taken literally that rule has no ceiling: a stream of cosmetic edits to
+     * DISTINCT message ids is a stream of "lone" frames, every one of which
+     * releases over budget — which is how 6 concurrent worker cards can sail
+     * past a 6/min chat ceiling. Charging each late release to a small extra
+     * window keeps R1's guarantee for the rare genuinely-lone frame (a
+     * worker's terminal recap) while capping the leak at `lateReleaseMax` per
+     * window. Absent ⇒ unbounded late release (sends, non-cosmetic edits).
+     */
+    lateReleaseKey?: string,
   ): Promise<number | null> {
     const w = win(key)
-    const deadline = clock.now() + maxDeferMs
     let counted = false
     for (;;) {
       const now = clock.now()
@@ -337,19 +816,32 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
         // worse than a late one).
         if (mode !== 'release' && (dropGuard === undefined || dropGuard())) {
           counters.dropped++
-          onTrip?.({ method, key, action: 'dropped' })
+          onTrip?.({ method, key, action: 'dropped', cls })
           return null
         }
         // A send — or an edit that nothing newer will repaint — is released
         // rather than dropped, and still takes a slot so the window reflects
-        // what actually went out.
+        // what actually went out. Cosmetic late releases are additionally
+        // charged to the bounded overshoot budget; when that is exhausted the
+        // frame is dropped after all, so the ceiling cannot be walked past one
+        // "lone" frame at a time.
+        if (lateReleaseKey !== undefined) {
+          const lw = win(lateReleaseKey)
+          prune(lw, now, perChatWindowMs)
+          if (lw.ts.length >= ceiling(lateReleaseMax, now)) {
+            counters.dropped++
+            onTrip?.({ method, key, action: 'dropped', cls })
+            return null
+          }
+          lw.ts.push(now)
+        }
         w.ts.push(now)
         return now
       }
       if (!counted) {
         counters.deferred++
         counted = true
-        onTrip?.({ method, key, action: 'deferred' })
+        onTrip?.({ method, key, action: 'deferred', cls })
       }
       // Last-write-wins: a newer edit to the same message kills this one. Only
       // valid on the per-MESSAGE tier — on a per-chat key the "newer" edit is
@@ -364,7 +856,7 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
         await Promise.race([clock.sleep(Math.min(wait, deadline - now)), superseded])
         if (killed) {
           counters.superseded++
-          onTrip?.({ method, key, action: 'superseded' })
+          onTrip?.({ method, key, action: 'superseded', cls })
           return null
         }
         w.waiter = null
@@ -381,51 +873,124 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
     next: () => Promise<R>,
   ): Promise<R> {
     if (!enabled) return next()
+    if (UNMETERED_METHODS.has(method)) return runObserved(next)
 
     const isEdit = EDIT_METHODS.has(method)
     const isSend = SEND_METHODS.has(method)
-    if (!isEdit && !isSend) return runObserved(next)
+    const isChatAction = CHAT_ACTION_METHODS.has(method)
 
     const { chat, msg } = payloadKeys(payload)
-    // Inline-message edits carry no chat/message id — nothing to key on, and
-    // they are not part of any card loop. Pass through (still observed).
-    if (chat == null) return runObserved(next)
 
     const now = clock.now()
     evict(now)
+    // ONE deadline for the whole call, shared by every tier it crosses.
+    const deadline = now + maxDeferMs
+
+    // The send gate's priority class, propagated through AsyncLocalStorage.
+    // An untagged edit is COSMETIC by default — see `defaultOutboundClass`.
+    // A chat action is cosmetic too: it is a self-expiring status, not content.
+    const cls =
+      currentOutboundClass() ?? (isChatAction ? 'cosmetic' : defaultOutboundClass(isEdit))
 
-    if (isEdit) {
-      if (msg == null) return runObserved(next)
+    // DEFAULT-DENY (#3855): a call the fuse cannot key to a chat — an inline
+    // edit, `answerCallbackQuery`, `getFile`, `setMyCommands` — is NOT free.
+    // It cannot take a per-chat slot (there is no chat to charge), but it does
+    // consume the bot token's global allowance, which is the thing Telegram
+    // actually bans. Charge it there and pass.
+    if (chat == null) {
+      counters.chatless++
+      await awaitRoom(tokenKey, perTokenWindowMs, perTokenMax, method, 'release', cls, undefined, deadline)
+      return runObserved(next)
+    }
+
+    const totalKey = `t:${chat}`
+    // Cosmetic traffic may only reach `perChatTotalMax - perChatReplyReserve`;
+    // the remaining slots stay available to a real reply no matter how hot the
+    // repaint surfaces are. This reservation is the reply-starvation guarantee.
+    const totalMaxFor = (c: OutboundClass): number =>
+      c === 'cosmetic' ? Math.max(1, perChatTotalMax - perChatReplyReserve) : perChatTotalMax
+
+    /** Final tier for every chat-targeted path: the token-scoped ceiling. */
+    const passToken = async (): Promise<R> => {
+      await awaitRoom(tokenKey, perTokenWindowMs, perTokenMax, method, 'release', cls, undefined, deadline)
+      return runObserved(next)
+    }
+
+    if (isEdit && msg != null) {
       const msgKey = `m:${chat}:${msg}`
       const mw = win(msgKey)
       // Counted BEFORE the first await so a frame that arrives while an older
       // one is waiting is visible to that older one's `dropGuard`.
       mw.inflight++
       try {
-        const msgSlot = await awaitRoom(msgKey, perMessageWindowMs, perMessageMax, method, 'supersede')
+        const msgSlot = await awaitRoom(
+          msgKey, perMessageWindowMs,
+          cls === 'cosmetic' ? cosmeticPerMessageMax : perMessageMax,
+          method, 'supersede', cls, undefined, deadline,
+        )
         if (msgSlot === null) return DROPPED_RESULT as unknown as R
+        // R1: only drop while something newer for THIS message is still in
+        // flight to repaint it. `> 1` = this frame plus at least one newer.
+        const dropGuard = (): boolean => mw.inflight > 1
+        // Cosmetic frames share ONE overshoot budget per chat across both
+        // per-chat tiers, so a frame cannot late-release twice on its way out.
+        const lateKey = cls === 'cosmetic' ? `lr:${chat}` : undefined
+        const reserved: Array<[string, number]> = [[msgKey, msgSlot]]
+        const giveBack = (): void => { for (const [k, at] of reserved) unreserve(k, at) }
+
         const chatKey = `ce:${chat}`
         const chatSlot = await awaitRoom(
-          chatKey, perChatWindowMs, perChatEditMax, method, 'drop',
-          // R1: only drop while something newer for THIS message is still in
-          // flight to repaint it. `> 1` = this frame plus at least one newer.
-          () => mw.inflight > 1,
+          chatKey, perChatWindowMs,
+          cls === 'cosmetic' ? cosmeticPerChatMax : perChatEditMax,
+          method, 'drop', cls, dropGuard, deadline, lateKey,
         )
-        if (chatSlot === null) {
-          // Denied by the second tier — hand the first tier's slot back so a
-          // dropped edit never consumes budget it did not use.
-          unreserve(msgKey, msgSlot)
-          return DROPPED_RESULT as unknown as R
-        }
-        return runObserved(next)
+        if (chatSlot === null) { giveBack(); return DROPPED_RESULT as unknown as R }
+        reserved.push([chatKey, chatSlot])
+
+        // Shared per-chat budget — the one that mirrors Telegram's real
+        // metering. A non-cosmetic edit is RELEASED late rather than dropped:
+        // an approval card or answer finalisation has no newer frame coming.
+        const totalSlot = await awaitRoom(
+          totalKey, perChatWindowMs, totalMaxFor(cls), method,
+          cls === 'cosmetic' ? 'drop' : 'release', cls, dropGuard, deadline, lateKey,
+        )
+        if (totalSlot === null) { giveBack(); return DROPPED_RESULT as unknown as R }
+        return passToken()
       } finally {
         mw.inflight--
       }
     }
 
-    const chatKey = `cs:${chat}`
-    await awaitRoom(chatKey, perChatWindowMs, perChatSendMax, method, 'release')
-    return runObserved(next)
+    if (isSend) {
+      // Sends create user-visible output and are NEVER dropped — only paced.
+      // The shared budget's reserve is what guarantees they still have room
+      // when the repaint surfaces are saturated.
+      await awaitRoom(`cs:${chat}`, perChatWindowMs, perChatSendMax, method, 'release', cls, undefined, deadline)
+      await awaitRoom(totalKey, perChatWindowMs, totalMaxFor(cls), method, 'release', cls, undefined, deadline)
+      return passToken()
+    }
+
+    // DEFAULT-DENY (#3855). Everything else with a `chat_id` — `sendChatAction`,
+    // `setMessageReaction`, `deleteMessage`, `pinChatMessage`, `sendChecklist`,
+    // an inline edit that carries a chat but no message id, and any Bot API
+    // method added after this file was written — is charged to the SAME shared
+    // per-chat window. Previously all of it was free, which is how a "20/60s"
+    // ceiling admitted ~34 calls/60s on the wire.
+    counters.meteredByDefault++
+    if (isChatAction) {
+      // Self-expiring status: shed it rather than deliver it late. Its own
+      // short deadline also keeps a typing ping from occupying a 30s hold.
+      const actionDeadline = Math.min(deadline, now + chatActionMaxDeferMs)
+      const slot = await awaitRoom(
+        totalKey, perChatWindowMs, totalMaxFor(cls), method, 'drop', cls, undefined, actionDeadline,
+      )
+      if (slot === null) return DROPPED_RESULT as unknown as R
+      return passToken()
+    }
+    // Durable, user-visible effect (a deletion, a pin, a reaction): paced,
+    // never dropped — the same contract sends get.
+    await awaitRoom(totalKey, perChatWindowMs, totalMaxFor(cls), method, 'release', cls, undefined, deadline)
+    return passToken()
   }
 
   /** Run the downstream call, tightening the ceilings if it floods. */
@@ -434,13 +999,16 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
       const res = await next()
       // grammY throws on ok:false, but a transformer installed BELOW another
       // one can still observe a raw ApiResponse — handle both shapes.
-      const r = res as unknown as { ok?: boolean; error_code?: number }
+      const r = res as unknown as {
+        ok?: boolean; error_code?: number; parameters?: { retry_after?: number }
+      }
       if (r != null && typeof r === 'object' && r.ok === false && r.error_code === 429) {
-        noteFlood(clock.now())
+        noteFlood(clock.now(), r.parameters?.retry_after ?? 0)
       }
       return res
     } catch (err) {
-      if (looksLikeFlood(err)) noteFlood(clock.now())
+      const retryAfterSec = floodRetryAfterSec(err)
+      if (retryAfterSec !== null) noteFlood(clock.now(), retryAfterSec)
       throw err
     }
   }
@@ -454,7 +1022,14 @@ export function createEditFloodFuse(config: EditFloodFuseConfig = {}) {
       superseded: counters.superseded,
       floodObserved: counters.floodObserved,
       tightened: isTightened(now),
+      tightenLevel: levelAt(now),
       perMessageCeiling: ceiling(perMessageMax, now),
+      cosmeticPerMessageCeiling: ceiling(cosmeticPerMessageMax, now),
+      cosmeticPerChatCeiling: ceiling(cosmeticPerChatMax, now),
+      meteredByDefault: counters.meteredByDefault,
+      chatless: counters.chatless,
+      perTokenCeiling: ceiling(perTokenMax, now),
+      persistedFloodOpen: persistedFloodRemainingMs(now) > 0,
     }
   }
 
@@ -470,7 +1045,13 @@ export type EditFloodFuse = ReturnType<typeof createEditFloodFuse>
 export type FuseInstallable = Bot
 
 export function installEditFloodFuse(bot: FuseInstallable, config: EditFloodFuseConfig = {}): EditFloodFuse {
-  const fuse = createEditFloodFuse(config)
+  // The token window is keyed by the bot's PUBLIC id only. `deriveBotScopeKey`
+  // never returns any part of the secret half, and the key is used solely as an
+  // in-memory Map key — never logged, never persisted.
+  const fuse = createEditFloodFuse({
+    botScopeKey: deriveBotScopeKey((bot as { token?: string }).token),
+    ...config,
+  })
   bot.api.config.use(async (prev, method, payload, signal) =>
     fuse.apply(method, payload, () => prev(method, payload, signal)))
   return fuse