@typetorch/framework 0.3.4 → 0.4.0

package/README.md

@@ -57,16 +57,25 @@ swap. It needs the TypeTorch kernel in the place (`@typetorch/kernel`).
 
 ### Coming from Flamework
 
-Game code only changes imports: `@Service`, `@Controller`, constructor injection, the lifecycle interfaces and
-`createNetwork` are the same.
+Game code only changes imports: `@Service`, `@Controller`, constructor injection, `Dependency<T>()`, the lifecycle
+interfaces and `createNetwork` are the same.
 
 | Flamework | TypeTorch |
 |---|---|
 | `rbxts-transformer-flamework` plugin, `node_modules/@flamework` type root | `@typetorch/transformer` plugin, `node_modules/@typetorch` type root |
 | `import { Modding, Reflect } from "@flamework/core"` | `import { Modding, Reflect } from "@typetorch/framework"` |
+| `import { Dependency } from "@flamework/core"` | `import { Dependency } from "@typetorch/framework"` (from onInit/onStart on, not in constructors) |
+| `import type` + `Dependency<T>()` to break a cycle | the same, or `Lazy<T>` (below) |
+| `ClientEvents.x.predict(...)` | `network.client.x.emit(...)` |
+| `invokeWithTimeout(timeout, ...)` | `invokeWithTimeout(seconds, ...)` (seconds, 0.5 to 120) |
+| `Observers.observeCharacter` / `observeLocalCharacter` (@rbxts/observers) | still fine: put the stop function in the trove, `this.trove.add(observeCharacter(...))` |
 | `import { t } from "@rbxts/t"` (still fine) | also `import { t } from "@typetorch/framework"` |
 | `@metadata flamework:parameters` keys, `"flamework:parameters"` metadata | `@metadata typetorch:parameters`, `"typetorch:parameters"` |
 | `flamework.build`, `include/flamework`, the `@flamework` Rojo mapping | gone; delete them |
+| `Networking.createEvent` / `createFunction` and their call sites | `createFlameworkCompat` (below) keeps every call site; `createNetwork` is the native API |
+
+`typetorch migrate --from flamework` (CLI) does most of this, and the docs' `guides/from-flamework.md` has the whole
+mapping.
 
 ## Usage
 
@@ -99,6 +108,19 @@ export class CoinService extends Module implements OnStart {
 - **Modules:** `@Service()` (server) and `@Controller()` (client) classes extending `Module`. Constructor parameters
   are other modules, injected by type. Lifecycle: `OnInit` (sequential, dependencies first), `OnStart` (spawned),
   `OnStop` (reverse order), `OnTick`, `OnPhysics`, `OnRender`, `OnPlayerAdded` (replays players already in the server).
+- **Modules outside the constructor:** `Dependency<T>()` (or `TypeTorch.module<T>()`) returns the running module of
+  type T from anywhere: a method, a plain class a module built, a command handler. `TypeTorch.tryModule<T>()` returns
+  undefined instead of throwing. They work once every module is constructed (from onInit on); in a constructor, a
+  field initializer or module top-level code they throw `"ShopService isn't constructed yet: ..."`. Only this realm's
+  modules, and per generation: after a swap they return the new generation's instance. An `import type` of the class is
+  enough (no require cycle).
+- **Cycles:** two modules that need each other take one side as `Lazy<T>`, resolved on first use:
+  `constructor(private readonly team: Lazy<TeamService>)` (needs `@typetorch/transformer` 0.2.1+) or a field,
+  `private readonly team = Lazy<TeamService>()` (any transformer). `this.team.get()` works from onInit on (in a
+  constructor it throws), and the start order ignores lazy edges. A cycle error names the cycle and suggests this.
+- **Per-player state:** `this.ctx.playerState("cooldowns.v1", (player) => init)` (or `TypeTorch.playerState`) returns
+  `{ get, set, has, delete }` keyed by UserId. It survives swaps (persist), and a player's entry goes when they really
+  leave, never on a swap. The dev menu shows it under Modules > State > persist, `__playerState`.
 - **Bad deploys roll back (kernel 0.3.2):** a server `onStart` that throws is reported to the kernel, which rolls the
   server back to its last known good artifact when it happens within 30 s of start (so do 3 errors from the new
   code's scripts in that time). On a real shutdown every module's `onStop` runs too (the kernel's BindToClose).
@@ -106,6 +128,18 @@ export class CoinService extends Module implements OnStart {
   behind. `TypeTorch.persist(key, init)` (or `this.ctx.persist`) keeps plain data across swaps.
 - **Network:** `createNetwork<C2S, S2C>()` with nested namespaces. The server checks rate limits, shape limits and the
   generated type guard on every client message. `setNetworkLimits({ "chat.say": { maxString: 200 } })` tunes a leaf.
+  Requests wait 15 s for the answer: `invokeWithTimeout(5, ...args)` sets it per call, and a leaf's `timeout`
+  (`setNetworkLimits({ "vc.spawn": { timeout: 30 } })`, read by the client, so set it in shared code) per leaf; both in
+  seconds, 0.5 to 120. `network.client.x.emit(...args)` runs this client's own `on` handlers for a server -> client
+  leaf right away, with no traffic (Flamework's `predict`); `network.server.x.emit(player, ...args)` does the same on
+  the server, for tests. A request handler may return a Promise: the answer waits for it.
+- **Flamework networking names:** `createFlameworkCompat<C2SEvents, S2CEvents, C2SFunctions, S2CFunctions>()` returns
+  `{ ServerEvents, ClientEvents, ServerFunctions, ClientFunctions, GlobalEvents, GlobalFunctions }` with
+  `@flamework/networking` 1.x's methods (`connect` returning a connection, `fire(player | players)`, `broadcast`,
+  `except`, `predict`, `setCallback`, `invoke`, `invokeWithTimeout`, calling a leaf directly) on this same network:
+  same guards, limits and swap rules, so a Flamework game's call sites compile unchanged. Listeners run on their own
+  thread each, like Flamework's, and belong to the running generation (no trove needed). Not carried over: server ->
+  client requests, middleware, `Networking.Unreliable`. `typetorch build` notes a game still using it.
   Across a swap (0.3.0): what the server sends a player waits until that player's client runs the new generation
   (reliable messages are queued, unreliable ones dropped), and on kernel 0.3.2 a request the server can no longer
   answer fails at once ("The game is updating, try again.") instead of timing out.
@@ -167,6 +201,10 @@ leaves one behind. Each `on*` returns a disconnect function, which a trove can o
 TypeTorch.artifact; // { id, commit, commitHash, branch, channel, builtAt, seq, assetId }
 TypeTorch.generation; // 1, 2, 3... per server (or client)
 TypeTorch.branch; TypeTorch.channel; TypeTorch.serverType; // "public" | "private" | "reserved" | "studio"
+// Kernel 0.3.9: `channel` is the RULES this server runs under ("prod" on every public server; split data stores by it);
+// `branchChannel` is what the branch IS ("prod" for the default branch or one configured prod, else "dev");
+// `rules` is the same as `channel`, by name.
+TypeTorch.branchChannel; TypeTorch.rules;
 TypeTorch.isPinned(); TypeTorch.kernelVersion; TypeTorch.kernelApi; TypeTorch.jobId; TypeTorch.isStudio;
 
 // How this generation started: a branch change starts a new generation.
@@ -182,6 +220,13 @@ this.trove.add(TypeTorch.onUpdatePending((update) => (hint.Visible = !update.can
 
 // State that survives swaps (plain data only)
 const scores = TypeTorch.persist("scores.v1", () => new Map<number, number>());
+// Per player, removed when the player really leaves (same as this.ctx.playerState)
+const cooldowns = TypeTorch.playerState("cooldowns.v1", () => ({ lastUse: 0 }));
+cooldowns.get(player).lastUse = os.clock();
+
+// The running module of a type (same as Dependency<T>()), from onInit/onStart on
+TypeTorch.module<ShopService>().open(player);
+TypeTorch.tryModule<ShopService>()?.open(player); // undefined instead of an error
 
 // Dev and roles (server: the kernel decides; client: only the local player, cosmetic)
 if (TypeTorch.isOwner(player)) showOwnerPanel(player);
@@ -198,6 +243,23 @@ TypeTorch.onLog((entry) => errors.push(entry)); // don't print from inside it
 
 // Hot assets (below): same as hotAsset(...)
 const shop = TypeTorch.asset("ui/shop");
+
+// Live values, server only (kernel 0.3.8): the signed settings record's `game` field (`typetorch settings set game.<key> <json>`).
+const price = TypeTorch.liveConfig("shop.price", { default: 50, parse: (raw) => (typeIs(raw, "number") ? raw : 50) });
+price.get();
+this.trove.add(price.onChanged((value) => this.reprice(value)));
+
+// Cross-server, server only. Kernel 0.3.8: game topics over ONE kernel-held MessagingService topic; listeners hear
+// prod servers and this branch by default (dev branches never reach prod); publish queues and retries, max 1 KiB.
+this.trove.add(TypeTorch.messaging.subscribe<Ban>("1guard", (ban, meta) => applyBan(ban, meta.jobId)));
+TypeTorch.messaging.publish("1guard", { kind: "ban", userId }, { to: "prod" });
+
+// The universe's live servers (the roll call; cached per server, yields up to ~3 s when stale) and this server's public fields
+TypeTorch.setServerInfo({ region: "eu", vc: true }); // JSON, at most 400 bytes, kept across swaps
+const list = TypeTorch.servers(); // [{ jobId, placeVersion, players, maxPlayers, serverType, branch, channel, uptime, here, info }]
+
+// A library call a swap must not cut off halfway (kernel 0.3.8): runs on a kernel thread; write what must survive into persist.
+TypeTorch.runDetached(() => store.StartSessionAsync(`Player_${userId}`)).then((profile) => attach(player, profile));
 ```
 
 - **`startInfo.reason`:** `boot`, `deploy`, `rollback`, `branch`, `pin`, `reload`, `server_rollback`,
@@ -214,6 +276,25 @@ const shop = TypeTorch.asset("ui/shop");
   reason and timings, the next artifact in `onSwapOut`, `onUpdatePending`, server-side `onPlayerDevChanged` and
   `requestReload`. On older kernels `startInfo` still knows boot vs swap, the previous artifact and branch (the
   framework records them), `onBranchChanged` still fires, and the reason of a swap is `unknown`.
+- **`messaging`** (kernel 0.3.8): `subscribe(topic, (data, meta) => ...)` returns a disconnect; `meta` = `{ channel,
+  branch, jobId, serverType, placeVersion, sentAt, self, replayed? }`; `{ from: "all" | "prod" | "branch" }` changes
+  who it hears. `publish(topic, data, { to })` throws on a bad topic, non-JSON data or a message over 1 KiB (with its
+  size), returns false when dropped at once (queue full). Studio loops back; the cloud test publishes nothing; older
+  kernels throw "needs kernel 0.3.8". `servers()` works on every kernel (0.3.8 holds the roll call topic, so a swap
+  doesn't subscribe again).
+- **`liveConfig(key, { default, parse? })`** (kernel 0.3.8, plans/20): reads `game[key]` of the signed settings record the
+  kernel verified (signed by both prod keys, so game code can't forge it); `get()` and `onChanged(fn)` (only on real
+  changes; generation-scoped). The default without a record or key, when `parse` throws (warned per value), and on older
+  kernels (warned once: "needs kernel 0.3.8"; no ConfigService fallback). `TypeTorch.settings()` is the whole copy
+  (`seq`, `at`, `game`, `analytics`, `fleet`, ...): server only, it holds tokens.
+- **`runDetached(fn)`** (kernel 0.3.8, server only): runs `fn` on a thread the kernel owns, so a deploy's hard stop
+  (which kills this generation's threads mid-call) can't cut it off: for ProfileStore / ProfileService loads, saves and
+  releases. Returns a Promise that settles while this generation runs; after a swap the result is dropped, so a job
+  whose result must survive writes it into `persist` itself. The job keeps this generation's closures alive until it
+  ends. Errors reject it and go to the kernel's log with the generation's name (never the health window); at most 256
+  run at once per server (then it throws); one past 60 s is logged and flagged in Server > Status. Older kernels
+  throw "needs kernel 0.3.8; use the DataHost job queue (Player data guide)": the other option, which works on every
+  kernel. `features.runDetached` tells which one this server has.
 - **Edit mode** (UI Labs stories, no kernel): `running` is false, identity has defaults, `persist` keeps a local table,
   events never fire, and the server-only reads throw.
 
@@ -227,6 +308,8 @@ pieces that keep it safe across swaps:
 - **Session handles live in `persist`**, keyed by `UserId`. `onPlayerAdded` replays everyone after a swap, so it
   re-attaches to the open session instead of loading again.
 - **Release only on `Players.PlayerRemoving`**, never in `onStop` or `onSwapOut`.
+- **Library calls can't be cut off by a swap:** run them as DataHost jobs (any kernel) or with
+  `TypeTorch.runDetached` (kernel 0.3.8).
 - **Split store names by channel** (`TypeTorch.channel === "prod" ? "PlayerData" : "PlayerData_dev"`).
 
 Full example with ProfileStore: [Player data guide](https://github.com/typetorch/docs/blob/main/guides/player-data.md).
@@ -287,7 +370,7 @@ self-hosted DuckDB analytics server). Nothing runs until an engine is created: n
 ```ts
 import { AnalyticsEngine } from "@typetorch/framework";
 
-// A server module (onInit): reads the ConfigService key TypeTorchAnalytics.
+// A server module (onInit): the sink comes from the signed settings' `analytics` (kernel 0.3.8).
 const analytics = new AnalyticsEngine();
 analytics.track(player, "quest_done", { quest: "tutorial" }); // custom
 analytics.step(player, "onboarding", 3, "opened_shop"); // funnels
@@ -317,13 +400,13 @@ const variant = analytics.experiment("onboarding", ["short", "long"]); // same a
   `TTZone`, named by a `Name` attribute or the instance name), `screens` (ScreenGuis in PlayerGui, GuiObjects tagged
   `TTScreen`), `recording` (the first-ever session in detail), `fleet` (kernel 0.3.2 heartbeats and deploy reports),
   `identity` (one `{ pid, uid }` row per session to the dev's own server, below). `settings` (server only) replaces the
-  ConfigService key, for tests.
+  signed settings' `analytics`, for tests, Studio and kernels before 0.3.8.
 - **Every row** carries the time (server clock), a random player id (never the UserId), the session, the server
   (JobId, type, place), the artifact (id, seq, branch, channel), the device, new vs returning, the player's state
   (`zone:Lobby|screen:Shop|activity:round`) and experiment variants. The exact rows: `src/analytics/SCHEMA.md`.
 - **Experiments:** `experiment(...)` is deterministic per player and name (they keep their variant in every session),
   may yield briefly the first time (until the player's id loads), and stamps the variant on the player's later events.
-  The settings key turns one off (`active: false`: everyone gets the first variant), sets `weights` or forces a
+  The settings turn one off (`active: false`: everyone gets the first variant), sets `weights` or forces a
   `variant`, live.
 - **First-ever session:** for new players (a share of them, `recordShare`), the client records character and camera
   about 10 times a second, every input (never while a TextBox or the chat has focus; text boxes only say which box was
@@ -333,7 +416,7 @@ const variant = analytics.experiment("onboarding", ["short", "long"]); // same a
   row carries a UserId.
 - **Identities** (option `identity`, default on): once the pid is known, one row `{ pid, uid, t }` (the UserId, nothing
   else) per session goes to the dev's own server only, never into the events: duckdb games in the batch body, Basin
-  games to the fleet API (`TypeTorchFleet` url + `/v1/identity`, or the `identity` / `identityToken` settings). The
+  games to the fleet API (the settings' `fleet` url + `/v1/identity`, or the `identity` / `identityToken` settings). The
   analytics server keeps pid -> UserId in a deletable table (support lookups, Right to Erasure).
 - **Player ids:** DataStore `TypeTorchAnalytics`, key `p/<UserId>` -> `{ pid, first, last }`: one read per join, a
   write on the first join and at leave. Deleting the key (Right to Erasure) leaves that player's rows anonymous.
@@ -341,6 +424,17 @@ const variant = analytics.experiment("onboarding", ["short", "long"]); // same a
   `flushSeconds` (15) or at 500 rows, at most ~10 HttpService requests a minute, retries with backoff, and a last
   flush on shutdown (kernel 0.3.2). Delivery is at least once (a swap mid-request sends that batch again).
   `analytics.stats()` (server) has the counters; `flush()` sends soon.
+- **When uploads fail** (a stale quick-tunnel URL, the dev PC's server or tunnel down, a wrong token, HttpService off;
+  Roblox reports most network problems as `HttpError: NetFail`, `DnsResolve` or `ConnectFail`): the server backs off
+  with jitter (5, 10, 20 ... 300 s, spread +-25% so a fleet of servers doesn't retry in step; a wrong token or URL waits
+  about 5 minutes, and a settings change retries at once), keeps the rows, and logs at most ONE line a minute: the
+  reason and the fix (`analytics upload failed 5 times in a row: NetFail: the connection broke mid-request. ... Retrying
+  in 80 s.`, with "(+N more in the last minute)" for the lines it held back) and one line when uploads work again.
+  `stats()` also has `failures` (in a row), `failed` (since the server started), `lastError`, `lastStatus`
+  (0 = no answer), `lastErrorAt` / `lastOkAt` (`os.time()`), `retryIn`, and, while failing, `reason` and `fix`. The dev
+  menu shows them: **Server > Status** has an *Analytics* line (queued, sent, `FAILING x5: ... retry in 80 s`) and a
+  *Fleet API* line (the kernel's sender: sent, failed, last error and its age), and **Attention** lists the reason and
+  the fix; `typetorch doctor` tests the address and token in the settings record.
 - **The cloud test sends nothing.** `typetorch test --cloud` boots the payload headless in a Luau Execution task where
   HttpService works; there the engine (stub kernel `test = true`, or `workspace:GetAttribute("TypeTorchTest")`) collects
   as usual but never uploads: no event rows, no identity rows, no HTTP request, so a prod deploy never puts a fake
@@ -352,9 +446,17 @@ const variant = analytics.experiment("onboarding", ["short", "long"]); // same a
 - **Friends in the server** (the join's `friends`): one `Players:GetFriendsAsync` per join (at most 10 pages, none
   when the player is alone) intersected with the players there, not a web call per player.
 
-**Settings** (server only, never sent to clients): the ConfigService key `TypeTorchAnalytics`, written with
-`writeSettings()` from `@typetorch/analytics` (Open Cloud, universe:write) or in Creator Hub (Configs); no CLI command
-writes it. Re-read live every few minutes:
+**Settings, framework 0.4.0 / kernel 0.4.0 (plans/21):** the record's `backend` = `{ url, key, analytics?: { flushSeconds,
+recordShare } }` (`typetorch backend setup`) is the whole sink: DuckDB at `<url>/v1/ingest` with the one key (Basin
+identities to `<url>/v1/identity`). The old `analytics` value below is read while `backend` is missing, and stays the
+way to a Basin sink. The engine also tells the kernel each player's pid (`setAnalyticsId`: the kernel's error reports
+count affected players by pid, never by name) and counts its own DataStore and HTTP requests for the budget view (dev
+menu **Server > Budget**: bars per Roblox limit for the player count, who spends it, the DataStore budget left, memory).
+
+**Settings** (server only, never sent to clients): the signed settings record's `analytics` field (kernel 0.3.8,
+plans/20), written with `typetorch settings set analytics -` (JSON on stdin; `@typetorch/analytics`' `writeSettings()`
+and `bun run local` call it). Every new copy applies live within seconds (the CLI pings servers). Before kernel 0.3.8
+there are no settings unless the game passes `new AnalyticsEngine({ settings })`:
 
 ```json
 { "backend": "basin", "events": "https://<stream-id>.ingest.cloudflare.com",
@@ -392,10 +494,20 @@ bun run build   # rbxtsc --type package -> out/
   package.json.
   `scripts/test-layout.luau` checks the dev menu's pane layout core (`devtools/layout.ts`: splits and their limits,
   closing, pop out and dock, singleton pages, ratios, focus, and cleaning a remembered layout).
+  `scripts/test-modules.luau` runs the real `startServer`/`startClient` of `out/` (with RuntimeLib, Promise, trove and
+  t from node_modules) over several generations that share a persist store: `Dependency<T>()`, `TypeTorch.module` /
+  `tryModule` and their errors, `Lazy<T>` (parameters and fields, cycles, start order), network `emit` and request
+  timeouts and `playerState` (swaps, leaves).
+  `scripts/test-flamework-compat.luau` (same harness) drives `createFlameworkCompat` through each generation's
+  dispatcher: connect (one thread per call, guards, Disconnect, troves), fire / except / broadcast / the call
+  shorthand, setCallback with values and Promises, predict, a swap, and the client side (invoke, invokeWithTimeout,
+  `createClient({ defaultTimeout })`).
   `scripts/test-analytics.luau` checks the analytics engine's pure parts (experiment assignment, settings, the queue
-  and HTTP budget, tt-rec-1, the sink request bodies); `test-analytics-server.luau` and `test-analytics-client.luau`
-  run the compiled server and client cores against mocked services (sessions, intake, retries, a swap with a request
-  in flight, shutdown; the recorder, screens, batching).
+  and HTTP budget, tt-rec-1, the sink request bodies, the failure help); `test-analytics-server.luau` and
+  `test-analytics-client.luau` run the compiled server and client cores against mocked services (sessions, intake,
+  retries, a NetFail streak and its one log line a minute, a swap with a request in flight, shutdown; the recorder,
+  screens, batching). `scripts/test-net-health.luau` checks the dev menu's Fleet API / Analytics status lines and
+  Attention issues.
 - **Publishing:** `npm publish` runs `prepublishOnly` (clean + build). The package ships only `out/` (no
   `.tsbuildinfo`), `README.md` and `LICENSE`; check with `bun pm pack --dry-run`.
 

package/out/analytics/SCHEMA.md

@@ -128,8 +128,10 @@ start, i.e. the row's `t`). Time is the running sum of `dt`.
 
 ## Sink settings
 
-ConfigService key `TypeTorchAnalytics` (server-only), a JSON object (or a JSON string of one). Re-read on
-`UpdateAvailable` and every 3 minutes, so dials change live. `new AnalyticsEngine({ settings })` replaces the key.
+The signed settings record's `analytics` field (kernel 0.3.8, plans/20; server-only; before 0.3.8 the ConfigService key
+`TypeTorchAnalytics`), a JSON object, written with `typetorch settings set analytics -` (JSON on stdin; the analytics
+repo's `bun run local` does it). Every new signed copy applies at once, so dials change live (the CLI pings servers).
+`new AnalyticsEngine({ settings })` replaces it (tests, Studio, kernels before 0.3.8).
 
 ```json
 {
@@ -154,8 +156,8 @@ ConfigService key `TypeTorchAnalytics` (server-only), a JSON object (or a JSON s
 | `recordShare` | 1 | 0..1, share of new players recorded (stable per pid) |
 | `techEvery` | 60 | 15..3600 seconds between tech samples |
 | `experiments` | none | Per experiment: `active: false` (everyone gets the first variant, not stamped), `weights` (per variant, in the game's order), `variant` (force one) |
-| `identity` | `TypeTorchFleet` url + `/v1/identity` | basin: where identity rows go (below); none and no readable `TypeTorchFleet`: not sent. duckdb: unused |
-| `identityToken` | the `TypeTorchFleet` token | basin: the token for `identity`. Never logged |
+| `identity` | the settings' `fleet` url + `/v1/identity` | basin: where identity rows go (below); none and no `fleet` in the settings: not sent. duckdb: unused |
+| `identityToken` | the settings' `fleet` token | basin: the token for `identity`. Never logged |
 
 Unknown fields are ignored. Settings errors are warned once (never with the token).
 
@@ -174,8 +176,13 @@ Unknown fields are ignored. Settings errors are warned once (never with the toke
 - **Batches:** every `flushSeconds`, or at 500 queued events or 20 recordings. At most 500 event rows (fleet rows
   first) and 50 recording rows per request, about 900 KB each. At most ~10 requests a minute per server (burst 3).
 - **Answers:** 2xx sent; 413 the batch is halved and resent (a single row is dropped); 401/403/404 rows are kept and
-  sending waits 300 s (or until the settings change); 408/429/5xx/no answer retry with backoff 5, 10, 20... 300 s;
-  other 4xx drop the batch (counted as rejected).
+  sending waits about 300 s (225-375 s, jittered; or until the settings change); 408/429/5xx/no answer retry with
+  backoff 5, 10, 20... 300 s (each +-25%); other 4xx drop the batch (counted as rejected).
+- **Failure log and status:** at most one log line a minute (the reason and the fix from `hints.ts`, the streak, the next
+  try; later lines say how many were held back) and one line when uploads work again. `stats()`: `failures` (in a
+  row), `failed` (total), `lastError`, `lastStatus` (0: no answer, an `HttpError` such as NetFail / DnsResolve),
+  `lastErrorAt`, `lastOkAt`, `retryIn`, and while failing `reason` and `fix`. The dev menu reads them through
+  `analytics/status.ts`.
 - **Delivery is at least once:** a hot swap during a request sends its rows again. Readers may drop exact duplicate
   rows.
 - **The cloud test sends nothing:** inside `typetorch test --cloud` (the stub kernel's `test = true`, or the workspace

package/out/analytics/engine.luau

@@ -10,7 +10,7 @@ local ServerAnalytics = TS.import(script, script.Parent, "server").ServerAnalyti
 	 * client. Nothing runs until one is created. Every engine created in a generation shares one core, which stops with
 	 * the generation; its unsent rows survive a hot swap in `persist`.
 	 *
-	 * The server sends rows to the sink in the settings (ConfigService key `TypeTorchAnalytics`, or `options.settings`).
+	 * The server sends rows to the sink in the settings (the signed settings' `analytics`, kernel 0.3.8; or `options.settings`).
 	 * The client sends everything through the server, never to the internet. See the README "Analytics" section and
 	 * src/analytics/SCHEMA.md.
 	 

package/out/analytics/hints.d.ts

@@ -0,0 +1,36 @@
+/**
+ * What a failed upload to the dev's analytics / fleet server means, and what to do about it. Shared by the analytics
+ * engine's log line (at most one a minute) and the dev menu (Status > Analytics / Fleet API, the "Attention" list), so
+ * both say the same thing.
+ *
+ * `HttpService.RequestAsync` throws a message with Roblox's HttpError name in it ("HttpError: NetFail"); a reply that
+ * isn't 2xx has a status. The names that matter to a dev running the server on their own PC behind a Cloudflare quick
+ * tunnel (`bun run local`):
+ *
+ *   NetFail        the connection broke mid-request (curl send/receive error, nothing usable came back): the server or
+ *                  tunnel restarted, dropped it, or is overloaded. The usual one when the dev PC's tunnel flaps.
+ *   DnsResolve     the host name doesn't resolve: the quick tunnel's URL died with its cloudflared (a new run gets a new
+ *                  URL) and the settings still hold the old one.
+ *   ConnectFail    nothing accepted the connection.
+ *   TimedOut       it accepted but never answered.
+ *   (HTTP 530/1033 a tunnel with nothing behind it, 502/504 a tunnel whose server is down, 401/403 a wrong token, 404 a
+ *                  wrong path.)
+ *
+ * Pure: no imports and no services, so it also runs offline under Lune (scripts/test-net-health.luau). Nothing here
+ * ever sees or returns a token.
+ */
+export interface FailureHelp {
+    /** What happened, in a few words ("the connection broke mid-request"). */
+    reason: string;
+    /** What to do about it, one or two sentences. */
+    fix: string;
+}
+/** The Roblox HttpError name in an error message ("HttpError: NetFail" -> "NetFail"), or undefined. */
+export declare function httpErrorName(message: string | undefined): string | undefined;
+/**
+ * Help for a failed request. `status` is the HTTP status (0: no answer at all) and `message` the error text (from the
+ * thrown HttpError, or "HTTP 530" style text for a status; either may be missing).
+ */
+export declare function describeFailure(status: number, message?: string): FailureHelp;
+/** Help for a kernel Fleet sender error text: "HTTP 530", or an HttpError message (`status().fleet.lastError`). */
+export declare function describeFleetError(lastError: string | undefined): FailureHelp | undefined;

package/out/analytics/hints.luau

@@ -0,0 +1,190 @@
+-- Compiled with roblox-ts v3.0.0
+--[[
+	*
+	 * What a failed upload to the dev's analytics / fleet server means, and what to do about it. Shared by the analytics
+	 * engine's log line (at most one a minute) and the dev menu (Status > Analytics / Fleet API, the "Attention" list), so
+	 * both say the same thing.
+	 *
+	 * `HttpService.RequestAsync` throws a message with Roblox's HttpError name in it ("HttpError: NetFail"); a reply that
+	 * isn't 2xx has a status. The names that matter to a dev running the server on their own PC behind a Cloudflare quick
+	 * tunnel (`bun run local`):
+	 *
+	 *   NetFail        the connection broke mid-request (curl send/receive error, nothing usable came back): the server or
+	 *                  tunnel restarted, dropped it, or is overloaded. The usual one when the dev PC's tunnel flaps.
+	 *   DnsResolve     the host name doesn't resolve: the quick tunnel's URL died with its cloudflared (a new run gets a new
+	 *                  URL) and the settings still hold the old one.
+	 *   ConnectFail    nothing accepted the connection.
+	 *   TimedOut       it accepted but never answered.
+	 *   (HTTP 530/1033 a tunnel with nothing behind it, 502/504 a tunnel whose server is down, 401/403 a wrong token, 404 a
+	 *                  wrong path.)
+	 *
+	 * Pure: no imports and no services, so it also runs offline under Lune (scripts/test-net-health.luau). Nothing here
+	 * ever sees or returns a token.
+	 
+]]
+--* The Roblox HttpError name in an error message ("HttpError: NetFail" -> "NetFail"), or undefined. 
+local function httpErrorName(message)
+	if message == nil then
+		return nil
+	end
+	local name = string.match(message, "HttpError:%s*(%w+)")
+	return if type(name) == "string" then name else nil
+end
+local RUN_LOCAL = "run bun run local on the dev PC (it opens a new tunnel and updates these settings)"
+--[[
+	*
+	 * Help for a failed request. `status` is the HTTP status (0: no answer at all) and `message` the error text (from the
+	 * thrown HttpError, or "HTTP 530" style text for a status; either may be missing).
+	 
+]]
+local describeStatus
+local function describeFailure(status, message)
+	if status > 0 then
+		return describeStatus(status)
+	end
+	local _condition = message
+	if _condition == nil then
+		_condition = ""
+	end
+	local text = _condition
+	local name = httpErrorName(text)
+	local lower = string.lower(text)
+	if (string.find(lower, "not enabled", 1, true)) ~= nil or (string.find(lower, "httpenabled", 1, true)) ~= nil then
+		return {
+			reason = "HttpService is off for this place",
+			fix = "turn on Game Settings > Security > Allow HTTP Requests (HttpService.HttpEnabled).",
+		}
+	end
+	if (string.find(lower, "exceeded limit", 1, true)) ~= nil or (string.find(lower, "too many requests", 1, true)) ~= nil then
+		return {
+			reason = "Roblox's HTTP limit for this server is used up",
+			fix = "a server may send 500 requests a minute in all; analytics sends about 10, so something else is using the rest.",
+		}
+	end
+	if name == "NetFail" then
+		return {
+			reason = "NetFail: the connection broke mid-request",
+			fix = `the dev PC's quick tunnel or server restarted, dropped it or is overloaded (a quick tunnel is for testing; use a real domain in production). Run typetorch doctor; if the address is stale, {RUN_LOCAL}.`,
+		}
+	end
+	if name == "DnsResolve" then
+		return {
+			reason = "DnsResolve: the host name doesn't resolve",
+			fix = `for a trycloudflare.com address the quick tunnel is gone and the URL changed: {RUN_LOCAL}. Otherwise check the URL for typos (typetorch doctor).`,
+		}
+	end
+	if name == "ConnectFail" then
+		return {
+			reason = "ConnectFail: nothing accepted the connection",
+			fix = `the server or its tunnel is down, or the address or port is wrong. Run typetorch doctor; {RUN_LOCAL} restarts both.`,
+		}
+	end
+	if name == "TimedOut" then
+		return {
+			reason = "TimedOut: the server didn't answer in time",
+			fix = "it is overloaded, asleep (the dev PC suspended?) or its tunnel is stuck. Run typetorch doctor.",
+		}
+	end
+	if name == "SslConnectFail" or name == "SslVerificationFail" then
+		return {
+			reason = `{name}: the TLS certificate was refused`,
+			fix = "use an address with a valid certificate (the Cloudflare tunnel, or Caddy on a VPS).",
+		}
+	end
+	if name == "InvalidUrl" then
+		return {
+			reason = "InvalidUrl: the address isn't a valid URL",
+			fix = "set it again with typetorch backend setup (typetorch doctor tests it).",
+		}
+	end
+	if name == "TooManyRedirects" or name == "InvalidRedirect" then
+		return {
+			reason = `{name}: the address redirects`,
+			fix = "use the final address directly in the settings.",
+		}
+	end
+	if name ~= nil then
+		return {
+			reason = `HttpError {name}`,
+			fix = "run typetorch doctor to test the address and token.",
+		}
+	end
+	return {
+		reason = if text == "" then "no answer" else string.sub(text, 1, 120),
+		fix = "run typetorch doctor to test the address and token.",
+	}
+end
+function describeStatus(status)
+	if status == 530 then
+		return {
+			reason = "HTTP 530: the tunnel has nothing behind it",
+			fix = `the dev PC's tunnel or server stopped: {RUN_LOCAL}.`,
+		}
+	end
+	if status == 502 or status == 504 then
+		return {
+			reason = `HTTP {status}: the server behind the tunnel doesn't answer`,
+			fix = `start the analytics server again: {RUN_LOCAL}.`,
+		}
+	end
+	if status == 401 or status == 403 then
+		return {
+			reason = `HTTP {status}: the server refused the token`,
+			fix = "put the server's API key (TYPETORCH_API_KEY; TT_ANALYTICS_INGEST_TOKENS before the rename) in the settings (typetorch backend setup); typetorch doctor tests it.",
+		}
+	end
+	if status == 404 then
+		return {
+			reason = "HTTP 404: no such address on the server",
+			fix = `the URL is wrong (analytics ends in /v1/ingest) or the tunnel points at nothing. Run typetorch doctor; {RUN_LOCAL}.`,
+		}
+	end
+	if status == 408 then
+		return {
+			reason = "HTTP 408: the server timed out",
+			fix = "it is overloaded; the engine retries with a growing pause.",
+		}
+	end
+	if status == 413 then
+		return {
+			reason = "HTTP 413: the batch was too large",
+			fix = "the engine splits it and sends again.",
+		}
+	end
+	if status == 429 then
+		return {
+			reason = "HTTP 429: rate limited",
+			fix = "too many requests for the server or tunnel; the engine retries with a growing pause.",
+		}
+	end
+	if status >= 500 then
+		return {
+			reason = `HTTP {status}: the server is failing`,
+			fix = "check the analytics server's log.",
+		}
+	end
+	return {
+		reason = `HTTP {status}`,
+		fix = "run typetorch doctor to test the address and token.",
+	}
+end
+--* Help for a kernel Fleet sender error text: "HTTP 530", or an HttpError message (`status().fleet.lastError`). 
+local function describeFleetError(lastError)
+	if lastError == nil then
+		return nil
+	end
+	local code = string.match(lastError, "^HTTP (%d+)$")
+	if type(code) == "string" then
+		local _condition = tonumber(code)
+		if _condition == nil then
+			_condition = 0
+		end
+		return describeFailure(_condition, lastError)
+	end
+	return describeFailure(0, lastError)
+end
+return {
+	httpErrorName = httpErrorName,
+	describeFailure = describeFailure,
+	describeFleetError = describeFleetError,
+}

package/out/analytics/queue.d.ts

@@ -44,6 +44,11 @@ export interface BucketState {
 export declare function takeToken(state: BucketState, now: number, burst: number, perMinute: number): boolean;
 /** Seconds to wait after `failures` failed requests in a row: 5, 10, 20... at most 300, spread by `jitter` (0..1). */
 export declare function backoffSeconds(failures: number, jitter: number): number;
+/**
+ * Seconds to wait after a settings problem (401/403/404: wrong token or URL): long, because retrying doesn't help until
+ * the settings change (that retries at once), spread by `jitter` (0..1) so a whole fleet of servers doesn't retry in step.
+ */
+export declare function configWaitSeconds(jitter: number): number;
 /** What to do with a batch after an HTTP status (0 = no response: network error, timeout, HttpService off). */
 export type Outcome = "sent" | "retry" | "split" | "drop" | "config";
 /**

package/out/analytics/queue.luau

@@ -192,6 +192,15 @@ local function backoffSeconds(failures, jitter)
 	local base = math.min(300, 5 * 2 ^ math.min(failures - 1, 10))
 	return base * (0.75 + 0.5 * math.clamp(jitter, 0, 1))
 end
+--[[
+	*
+	 * Seconds to wait after a settings problem (401/403/404: wrong token or URL): long, because retrying doesn't help until
+	 * the settings change (that retries at once), spread by `jitter` (0..1) so a whole fleet of servers doesn't retry in step.
+	 
+]]
+local function configWaitSeconds(jitter)
+	return 300 * (0.75 + 0.5 * math.clamp(jitter, 0, 1))
+end
 --* What to do with a batch after an HTTP status (0 = no response: network error, timeout, HttpService off). 
 --[[
 	*
@@ -219,6 +228,7 @@ return {
 	newQueueState = newQueueState,
 	takeToken = takeToken,
 	backoffSeconds = backoffSeconds,
+	configWaitSeconds = configWaitSeconds,
 	classifyStatus = classifyStatus,
 	RowQueue = RowQueue,
 }

package/out/analytics/schema.d.ts

@@ -5,10 +5,6 @@
  */
 /** Row format version (column `v`). */
 export declare const SCHEMA_VERSION = 1;
-/** The ConfigService key (server-only) that holds the sink settings. */
-export declare const SETTINGS_KEY = "TypeTorchAnalytics";
-/** The kernel's fleet API settings `{ url, token }` (server-only ConfigService key); identities can go there. */
-export declare const FLEET_SETTINGS_KEY = "TypeTorchFleet";
 /** The first-session recording codec (column `codec`). */
 export declare const RECORDING_CODEC = "tt-rec-1";
 /** Most bytes of the `props` JSON text. Larger props become `{"_trunc":<bytes>}`. */
@@ -91,7 +87,7 @@ export interface ExperimentOverride {
     /** Everyone gets this variant (must be one of the game's variants). */
     variant?: string;
 }
-/** The sink settings: ConfigService key `TypeTorchAnalytics` (server-only), or `new AnalyticsEngine({ settings })`. */
+/** The sink settings: the signed settings' `analytics` (kernel 0.3.8, server-only), or `new AnalyticsEngine({ settings })`. */
 export interface AnalyticsSettings {
     backend: "basin" | "duckdb";
     /** basin: the events stream's ingest URL; duckdb: the analytics server's ingest URL. */
@@ -109,12 +105,12 @@ export interface AnalyticsSettings {
     /** Live experiment overrides by experiment name. */
     experiments?: Record<string, ExperimentOverride>;
     /**
-     * basin: where identity rows go (the fleet API's `POST /v1/identity` URL). Default: the `TypeTorchFleet` key's url
+     * basin: where identity rows go (the fleet API's `POST /v1/identity` URL). Default: the settings' `fleet` url
      * + /v1/identity when the server can read it; without either, identities aren't sent. duckdb: unused (they go in
      * the batch body).
      */
     identity?: string;
-    /** basin: the token for `identity` (default: the `TypeTorchFleet` token). Never logged. */
+    /** basin: the token for `identity` (default: the settings' `fleet` token). Never logged. */
     identityToken?: string;
 }
 /** AnalyticsSettings with every default filled in. */
@@ -147,7 +143,7 @@ export interface AnalyticsPurchase {
 }
 /** What the automatic collectors do. Everything is on by default. */
 export interface AnalyticsOptions {
-    /** Server only: the sink settings. When set, the ConfigService key isn't read (tests, Studio). */
+    /** Server only: the sink settings. When set, the signed settings aren't read (tests, Studio, kernels before 0.3.8). */
     settings?: AnalyticsSettings;
     /** Server: joins, leaves, device, join source, first-ever vs returning. Default true. */
     sessions?: boolean;
@@ -197,9 +193,25 @@ export interface AnalyticsStats {
     dropped: number;
     /** Rows the backend refused (HTTP 4xx). */
     rejected: number;
-    /** Failed requests in a row. */
+    /** Failed requests in a row (0 after a success). */
     failures: number;
+    /** Failed requests since the server started, every attempt (across swaps). */
+    failed: number;
+    /**
+     * The last failure's message ("both: HttpError: NetFail", "both: HTTP 530 ..."). Kept after a later success:
+     * compare `lastErrorAt` with `lastOkAt`.
+     */
     lastError?: string;
+    /** HTTP status of the last failure (0: no answer at all, an HttpError). */
+    lastStatus?: number;
+    /** os.time() of the last failed request and of the last accepted one. */
+    lastErrorAt?: number;
+    lastOkAt?: number;
+    /** Seconds until the next attempt while backing off after failures. */
+    retryIn?: number;
+    /** While failing: what the last failure means and what to do (hints.ts); the same words the log line and the dev menu use. */
+    reason?: string;
+    fix?: string;
     /** Settings problems (never contains the token). */
     settingsErrors: string[];
 }