@typetorch/framework 0.3.5 → 0.4.0

package/README.md

@@ -201,6 +201,10 @@ leaves one behind. Each `on*` returns a disconnect function, which a trove can o
 TypeTorch.artifact; // { id, commit, commitHash, branch, channel, builtAt, seq, assetId }
 TypeTorch.generation; // 1, 2, 3... per server (or client)
 TypeTorch.branch; TypeTorch.channel; TypeTorch.serverType; // "public" | "private" | "reserved" | "studio"
+// Kernel 0.3.9: `channel` is the RULES this server runs under ("prod" on every public server; split data stores by it);
+// `branchChannel` is what the branch IS ("prod" for the default branch or one configured prod, else "dev");
+// `rules` is the same as `channel`, by name.
+TypeTorch.branchChannel; TypeTorch.rules;
 TypeTorch.isPinned(); TypeTorch.kernelVersion; TypeTorch.kernelApi; TypeTorch.jobId; TypeTorch.isStudio;
 
 // How this generation started: a branch change starts a new generation.
@@ -420,6 +424,17 @@ const variant = analytics.experiment("onboarding", ["short", "long"]); // same a
   `flushSeconds` (15) or at 500 rows, at most ~10 HttpService requests a minute, retries with backoff, and a last
   flush on shutdown (kernel 0.3.2). Delivery is at least once (a swap mid-request sends that batch again).
   `analytics.stats()` (server) has the counters; `flush()` sends soon.
+- **When uploads fail** (a stale quick-tunnel URL, the dev PC's server or tunnel down, a wrong token, HttpService off;
+  Roblox reports most network problems as `HttpError: NetFail`, `DnsResolve` or `ConnectFail`): the server backs off
+  with jitter (5, 10, 20 ... 300 s, spread +-25% so a fleet of servers doesn't retry in step; a wrong token or URL waits
+  about 5 minutes, and a settings change retries at once), keeps the rows, and logs at most ONE line a minute: the
+  reason and the fix (`analytics upload failed 5 times in a row: NetFail: the connection broke mid-request. ... Retrying
+  in 80 s.`, with "(+N more in the last minute)" for the lines it held back) and one line when uploads work again.
+  `stats()` also has `failures` (in a row), `failed` (since the server started), `lastError`, `lastStatus`
+  (0 = no answer), `lastErrorAt` / `lastOkAt` (`os.time()`), `retryIn`, and, while failing, `reason` and `fix`. The dev
+  menu shows them: **Server > Status** has an *Analytics* line (queued, sent, `FAILING x5: ... retry in 80 s`) and a
+  *Fleet API* line (the kernel's sender: sent, failed, last error and its age), and **Attention** lists the reason and
+  the fix; `typetorch doctor` tests the address and token in the settings record.
 - **The cloud test sends nothing.** `typetorch test --cloud` boots the payload headless in a Luau Execution task where
   HttpService works; there the engine (stub kernel `test = true`, or `workspace:GetAttribute("TypeTorchTest")`) collects
   as usual but never uploads: no event rows, no identity rows, no HTTP request, so a prod deploy never puts a fake
@@ -431,6 +446,13 @@ const variant = analytics.experiment("onboarding", ["short", "long"]); // same a
 - **Friends in the server** (the join's `friends`): one `Players:GetFriendsAsync` per join (at most 10 pages, none
   when the player is alone) intersected with the players there, not a web call per player.
 
+**Settings, framework 0.4.0 / kernel 0.4.0 (plans/21):** the record's `backend` = `{ url, key, analytics?: { flushSeconds,
+recordShare } }` (`typetorch backend setup`) is the whole sink: DuckDB at `<url>/v1/ingest` with the one key (Basin
+identities to `<url>/v1/identity`). The old `analytics` value below is read while `backend` is missing, and stays the
+way to a Basin sink. The engine also tells the kernel each player's pid (`setAnalyticsId`: the kernel's error reports
+count affected players by pid, never by name) and counts its own DataStore and HTTP requests for the budget view (dev
+menu **Server > Budget**: bars per Roblox limit for the player count, who spends it, the DataStore budget left, memory).
+
 **Settings** (server only, never sent to clients): the signed settings record's `analytics` field (kernel 0.3.8,
 plans/20), written with `typetorch settings set analytics -` (JSON on stdin; `@typetorch/analytics`' `writeSettings()`
 and `bun run local` call it). Every new copy applies live within seconds (the CLI pings servers). Before kernel 0.3.8
@@ -481,9 +503,11 @@ bun run build   # rbxtsc --type package -> out/
   shorthand, setCallback with values and Promises, predict, a swap, and the client side (invoke, invokeWithTimeout,
   `createClient({ defaultTimeout })`).
   `scripts/test-analytics.luau` checks the analytics engine's pure parts (experiment assignment, settings, the queue
-  and HTTP budget, tt-rec-1, the sink request bodies); `test-analytics-server.luau` and `test-analytics-client.luau`
-  run the compiled server and client cores against mocked services (sessions, intake, retries, a swap with a request
-  in flight, shutdown; the recorder, screens, batching).
+  and HTTP budget, tt-rec-1, the sink request bodies, the failure help); `test-analytics-server.luau` and
+  `test-analytics-client.luau` run the compiled server and client cores against mocked services (sessions, intake,
+  retries, a NetFail streak and its one log line a minute, a swap with a request in flight, shutdown; the recorder,
+  screens, batching). `scripts/test-net-health.luau` checks the dev menu's Fleet API / Analytics status lines and
+  Attention issues.
 - **Publishing:** `npm publish` runs `prepublishOnly` (clean + build). The package ships only `out/` (no
   `.tsbuildinfo`), `README.md` and `LICENSE`; check with `bun pm pack --dry-run`.
 

package/out/analytics/SCHEMA.md

@@ -176,8 +176,13 @@ Unknown fields are ignored. Settings errors are warned once (never with the toke
 - **Batches:** every `flushSeconds`, or at 500 queued events or 20 recordings. At most 500 event rows (fleet rows
   first) and 50 recording rows per request, about 900 KB each. At most ~10 requests a minute per server (burst 3).
 - **Answers:** 2xx sent; 413 the batch is halved and resent (a single row is dropped); 401/403/404 rows are kept and
-  sending waits 300 s (or until the settings change); 408/429/5xx/no answer retry with backoff 5, 10, 20... 300 s;
-  other 4xx drop the batch (counted as rejected).
+  sending waits about 300 s (225-375 s, jittered; or until the settings change); 408/429/5xx/no answer retry with
+  backoff 5, 10, 20... 300 s (each +-25%); other 4xx drop the batch (counted as rejected).
+- **Failure log and status:** at most one log line a minute (the reason and the fix from `hints.ts`, the streak, the next
+  try; later lines say how many were held back) and one line when uploads work again. `stats()`: `failures` (in a
+  row), `failed` (total), `lastError`, `lastStatus` (0: no answer, an `HttpError` such as NetFail / DnsResolve),
+  `lastErrorAt`, `lastOkAt`, `retryIn`, and while failing `reason` and `fix`. The dev menu reads them through
+  `analytics/status.ts`.
 - **Delivery is at least once:** a hot swap during a request sends its rows again. Readers may drop exact duplicate
   rows.
 - **The cloud test sends nothing:** inside `typetorch test --cloud` (the stub kernel's `test = true`, or the workspace

package/out/analytics/hints.d.ts

@@ -0,0 +1,36 @@
+/**
+ * What a failed upload to the dev's analytics / fleet server means, and what to do about it. Shared by the analytics
+ * engine's log line (at most one a minute) and the dev menu (Status > Analytics / Fleet API, the "Attention" list), so
+ * both say the same thing.
+ *
+ * `HttpService.RequestAsync` throws a message with Roblox's HttpError name in it ("HttpError: NetFail"); a reply that
+ * isn't 2xx has a status. The names that matter to a dev running the server on their own PC behind a Cloudflare quick
+ * tunnel (`bun run local`):
+ *
+ *   NetFail        the connection broke mid-request (curl send/receive error, nothing usable came back): the server or
+ *                  tunnel restarted, dropped it, or is overloaded. The usual one when the dev PC's tunnel flaps.
+ *   DnsResolve     the host name doesn't resolve: the quick tunnel's URL died with its cloudflared (a new run gets a new
+ *                  URL) and the settings still hold the old one.
+ *   ConnectFail    nothing accepted the connection.
+ *   TimedOut       it accepted but never answered.
+ *   (HTTP 530/1033 a tunnel with nothing behind it, 502/504 a tunnel whose server is down, 401/403 a wrong token, 404 a
+ *                  wrong path.)
+ *
+ * Pure: no imports and no services, so it also runs offline under Lune (scripts/test-net-health.luau). Nothing here
+ * ever sees or returns a token.
+ */
+export interface FailureHelp {
+    /** What happened, in a few words ("the connection broke mid-request"). */
+    reason: string;
+    /** What to do about it, one or two sentences. */
+    fix: string;
+}
+/** The Roblox HttpError name in an error message ("HttpError: NetFail" -> "NetFail"), or undefined. */
+export declare function httpErrorName(message: string | undefined): string | undefined;
+/**
+ * Help for a failed request. `status` is the HTTP status (0: no answer at all) and `message` the error text (from the
+ * thrown HttpError, or "HTTP 530" style text for a status; either may be missing).
+ */
+export declare function describeFailure(status: number, message?: string): FailureHelp;
+/** Help for a kernel Fleet sender error text: "HTTP 530", or an HttpError message (`status().fleet.lastError`). */
+export declare function describeFleetError(lastError: string | undefined): FailureHelp | undefined;

package/out/analytics/hints.luau

@@ -0,0 +1,190 @@
+-- Compiled with roblox-ts v3.0.0
+--[[
+	*
+	 * What a failed upload to the dev's analytics / fleet server means, and what to do about it. Shared by the analytics
+	 * engine's log line (at most one a minute) and the dev menu (Status > Analytics / Fleet API, the "Attention" list), so
+	 * both say the same thing.
+	 *
+	 * `HttpService.RequestAsync` throws a message with Roblox's HttpError name in it ("HttpError: NetFail"); a reply that
+	 * isn't 2xx has a status. The names that matter to a dev running the server on their own PC behind a Cloudflare quick
+	 * tunnel (`bun run local`):
+	 *
+	 *   NetFail        the connection broke mid-request (curl send/receive error, nothing usable came back): the server or
+	 *                  tunnel restarted, dropped it, or is overloaded. The usual one when the dev PC's tunnel flaps.
+	 *   DnsResolve     the host name doesn't resolve: the quick tunnel's URL died with its cloudflared (a new run gets a new
+	 *                  URL) and the settings still hold the old one.
+	 *   ConnectFail    nothing accepted the connection.
+	 *   TimedOut       it accepted but never answered.
+	 *   (HTTP 530/1033 a tunnel with nothing behind it, 502/504 a tunnel whose server is down, 401/403 a wrong token, 404 a
+	 *                  wrong path.)
+	 *
+	 * Pure: no imports and no services, so it also runs offline under Lune (scripts/test-net-health.luau). Nothing here
+	 * ever sees or returns a token.
+	 
+]]
+--* The Roblox HttpError name in an error message ("HttpError: NetFail" -> "NetFail"), or undefined. 
+local function httpErrorName(message)
+	if message == nil then
+		return nil
+	end
+	local name = string.match(message, "HttpError:%s*(%w+)")
+	return if type(name) == "string" then name else nil
+end
+local RUN_LOCAL = "run bun run local on the dev PC (it opens a new tunnel and updates these settings)"
+--[[
+	*
+	 * Help for a failed request. `status` is the HTTP status (0: no answer at all) and `message` the error text (from the
+	 * thrown HttpError, or "HTTP 530" style text for a status; either may be missing).
+	 
+]]
+local describeStatus
+local function describeFailure(status, message)
+	if status > 0 then
+		return describeStatus(status)
+	end
+	local _condition = message
+	if _condition == nil then
+		_condition = ""
+	end
+	local text = _condition
+	local name = httpErrorName(text)
+	local lower = string.lower(text)
+	if (string.find(lower, "not enabled", 1, true)) ~= nil or (string.find(lower, "httpenabled", 1, true)) ~= nil then
+		return {
+			reason = "HttpService is off for this place",
+			fix = "turn on Game Settings > Security > Allow HTTP Requests (HttpService.HttpEnabled).",
+		}
+	end
+	if (string.find(lower, "exceeded limit", 1, true)) ~= nil or (string.find(lower, "too many requests", 1, true)) ~= nil then
+		return {
+			reason = "Roblox's HTTP limit for this server is used up",
+			fix = "a server may send 500 requests a minute in all; analytics sends about 10, so something else is using the rest.",
+		}
+	end
+	if name == "NetFail" then
+		return {
+			reason = "NetFail: the connection broke mid-request",
+			fix = `the dev PC's quick tunnel or server restarted, dropped it or is overloaded (a quick tunnel is for testing; use a real domain in production). Run typetorch doctor; if the address is stale, {RUN_LOCAL}.`,
+		}
+	end
+	if name == "DnsResolve" then
+		return {
+			reason = "DnsResolve: the host name doesn't resolve",
+			fix = `for a trycloudflare.com address the quick tunnel is gone and the URL changed: {RUN_LOCAL}. Otherwise check the URL for typos (typetorch doctor).`,
+		}
+	end
+	if name == "ConnectFail" then
+		return {
+			reason = "ConnectFail: nothing accepted the connection",
+			fix = `the server or its tunnel is down, or the address or port is wrong. Run typetorch doctor; {RUN_LOCAL} restarts both.`,
+		}
+	end
+	if name == "TimedOut" then
+		return {
+			reason = "TimedOut: the server didn't answer in time",
+			fix = "it is overloaded, asleep (the dev PC suspended?) or its tunnel is stuck. Run typetorch doctor.",
+		}
+	end
+	if name == "SslConnectFail" or name == "SslVerificationFail" then
+		return {
+			reason = `{name}: the TLS certificate was refused`,
+			fix = "use an address with a valid certificate (the Cloudflare tunnel, or Caddy on a VPS).",
+		}
+	end
+	if name == "InvalidUrl" then
+		return {
+			reason = "InvalidUrl: the address isn't a valid URL",
+			fix = "set it again with typetorch backend setup (typetorch doctor tests it).",
+		}
+	end
+	if name == "TooManyRedirects" or name == "InvalidRedirect" then
+		return {
+			reason = `{name}: the address redirects`,
+			fix = "use the final address directly in the settings.",
+		}
+	end
+	if name ~= nil then
+		return {
+			reason = `HttpError {name}`,
+			fix = "run typetorch doctor to test the address and token.",
+		}
+	end
+	return {
+		reason = if text == "" then "no answer" else string.sub(text, 1, 120),
+		fix = "run typetorch doctor to test the address and token.",
+	}
+end
+function describeStatus(status)
+	if status == 530 then
+		return {
+			reason = "HTTP 530: the tunnel has nothing behind it",
+			fix = `the dev PC's tunnel or server stopped: {RUN_LOCAL}.`,
+		}
+	end
+	if status == 502 or status == 504 then
+		return {
+			reason = `HTTP {status}: the server behind the tunnel doesn't answer`,
+			fix = `start the analytics server again: {RUN_LOCAL}.`,
+		}
+	end
+	if status == 401 or status == 403 then
+		return {
+			reason = `HTTP {status}: the server refused the token`,
+			fix = "put the server's API key (TYPETORCH_API_KEY; TT_ANALYTICS_INGEST_TOKENS before the rename) in the settings (typetorch backend setup); typetorch doctor tests it.",
+		}
+	end
+	if status == 404 then
+		return {
+			reason = "HTTP 404: no such address on the server",
+			fix = `the URL is wrong (analytics ends in /v1/ingest) or the tunnel points at nothing. Run typetorch doctor; {RUN_LOCAL}.`,
+		}
+	end
+	if status == 408 then
+		return {
+			reason = "HTTP 408: the server timed out",
+			fix = "it is overloaded; the engine retries with a growing pause.",
+		}
+	end
+	if status == 413 then
+		return {
+			reason = "HTTP 413: the batch was too large",
+			fix = "the engine splits it and sends again.",
+		}
+	end
+	if status == 429 then
+		return {
+			reason = "HTTP 429: rate limited",
+			fix = "too many requests for the server or tunnel; the engine retries with a growing pause.",
+		}
+	end
+	if status >= 500 then
+		return {
+			reason = `HTTP {status}: the server is failing`,
+			fix = "check the analytics server's log.",
+		}
+	end
+	return {
+		reason = `HTTP {status}`,
+		fix = "run typetorch doctor to test the address and token.",
+	}
+end
+--* Help for a kernel Fleet sender error text: "HTTP 530", or an HttpError message (`status().fleet.lastError`). 
+local function describeFleetError(lastError)
+	if lastError == nil then
+		return nil
+	end
+	local code = string.match(lastError, "^HTTP (%d+)$")
+	if type(code) == "string" then
+		local _condition = tonumber(code)
+		if _condition == nil then
+			_condition = 0
+		end
+		return describeFailure(_condition, lastError)
+	end
+	return describeFailure(0, lastError)
+end
+return {
+	httpErrorName = httpErrorName,
+	describeFailure = describeFailure,
+	describeFleetError = describeFleetError,
+}

package/out/analytics/queue.d.ts

@@ -44,6 +44,11 @@ export interface BucketState {
 export declare function takeToken(state: BucketState, now: number, burst: number, perMinute: number): boolean;
 /** Seconds to wait after `failures` failed requests in a row: 5, 10, 20... at most 300, spread by `jitter` (0..1). */
 export declare function backoffSeconds(failures: number, jitter: number): number;
+/**
+ * Seconds to wait after a settings problem (401/403/404: wrong token or URL): long, because retrying doesn't help until
+ * the settings change (that retries at once), spread by `jitter` (0..1) so a whole fleet of servers doesn't retry in step.
+ */
+export declare function configWaitSeconds(jitter: number): number;
 /** What to do with a batch after an HTTP status (0 = no response: network error, timeout, HttpService off). */
 export type Outcome = "sent" | "retry" | "split" | "drop" | "config";
 /**

package/out/analytics/queue.luau

@@ -192,6 +192,15 @@ local function backoffSeconds(failures, jitter)
 	local base = math.min(300, 5 * 2 ^ math.min(failures - 1, 10))
 	return base * (0.75 + 0.5 * math.clamp(jitter, 0, 1))
 end
+--[[
+	*
+	 * Seconds to wait after a settings problem (401/403/404: wrong token or URL): long, because retrying doesn't help until
+	 * the settings change (that retries at once), spread by `jitter` (0..1) so a whole fleet of servers doesn't retry in step.
+	 
+]]
+local function configWaitSeconds(jitter)
+	return 300 * (0.75 + 0.5 * math.clamp(jitter, 0, 1))
+end
 --* What to do with a batch after an HTTP status (0 = no response: network error, timeout, HttpService off). 
 --[[
 	*
@@ -219,6 +228,7 @@ return {
 	newQueueState = newQueueState,
 	takeToken = takeToken,
 	backoffSeconds = backoffSeconds,
+	configWaitSeconds = configWaitSeconds,
 	classifyStatus = classifyStatus,
 	RowQueue = RowQueue,
 }

package/out/analytics/schema.d.ts

@@ -193,9 +193,25 @@ export interface AnalyticsStats {
     dropped: number;
     /** Rows the backend refused (HTTP 4xx). */
     rejected: number;
-    /** Failed requests in a row. */
+    /** Failed requests in a row (0 after a success). */
     failures: number;
+    /** Failed requests since the server started, every attempt (across swaps). */
+    failed: number;
+    /**
+     * The last failure's message ("both: HttpError: NetFail", "both: HTTP 530 ..."). Kept after a later success:
+     * compare `lastErrorAt` with `lastOkAt`.
+     */
     lastError?: string;
+    /** HTTP status of the last failure (0: no answer at all, an HttpError). */
+    lastStatus?: number;
+    /** os.time() of the last failed request and of the last accepted one. */
+    lastErrorAt?: number;
+    lastOkAt?: number;
+    /** Seconds until the next attempt while backing off after failures. */
+    retryIn?: number;
+    /** While failing: what the last failure means and what to do (hints.ts); the same words the log line and the dev menu use. */
+    reason?: string;
+    fix?: string;
     /** Settings problems (never contains the token). */
     settingsErrors: string[];
 }

package/out/analytics/server.d.ts

@@ -69,6 +69,10 @@ export declare class ServerAnalytics {
     private backoffUntil;
     private batchRows;
     private lastWarnAt;
+    /** Lines held back by the one-a-minute limit since the last one that was printed. */
+    private suppressed;
+    /** A failure line was printed since the last recovery line (so a recovery is only announced after an announced failure). */
+    private outageLogged;
     private flushing;
     private closing;
     private readonly identity;
@@ -114,6 +118,13 @@ export declare class ServerAnalytics {
      * changes nothing.
      */
     private resolve;
+    /**
+     * Kernel 0.4.0 (plans/21 C): the kernel's error reports count affected players by pid (never a name or UserId), so
+     * the kernel hears each player's pid. Older kernels: nothing.
+     */
+    private tellKernelPid;
+    /** Kernel 0.4.0: counts this engine's own requests for the budget view (older kernels: nothing). */
+    private countBudget;
     private joinSource;
     /**
      * How many of the player's friends are in this server: one `GetFriendsAsync` per join (at most FRIEND_PAGES_MAX
@@ -139,7 +150,10 @@ export declare class ServerAnalytics {
     private startZones;
     private startFleet;
     private onDeployReport;
+    /** At most one log line a minute; the ones held back are counted in the next line. Returns whether it was printed. */
     private warn;
+    /** A request was accepted: ends a failure streak, and says so once if the streak was announced. */
+    private noteSuccess;
     /** POST one request. Returns the HTTP status (0: no response). Never logs the token or the body. */
     private post;
     /** Sends one batch per table. `final`: shutdown, ignore the budget. */