@serve.zone/interfaces 32.45.0 → 32.47.0

package/readme.md

@@ -1553,6 +1553,62 @@ session, that the assignment is this node's and carries the batch's attempt, and
 joined length against the policy. `bindReportRuntimeAssignmentLogsResponse(response, request)`
 binds the acknowledgement to the batch.
 
+**Reading kept logs.** An operator reads what Cloudly keeps through two service requests, which
+take `{ identity, serviceId }` and are authorized like every other read of that service. Cloudly
+keeps an attempt's logs from the moment it issues the attempt, so an attempt that failed before
+writing a line is listed with nothing held, and an attempt whose slot was replaced or released stays
+readable until its logs expire.
+
+- `requests.service.getServiceRuntimeLogAttempts { identity, serviceId, cursor, limit }` answers an
+  `IRuntimeAssignmentLogAttemptPage { attempts, nextCursor }`: the service's attempts, newest issued
+  first and ties in `assignmentId` order. An `IRuntimeAssignmentLogAttemptView` states `serviceId`,
+  `replicaId`, `nodeId`, `assignmentId`, `attemptId`, `issuedAt`, `lastAcceptedAt` (null until a
+  batch was accepted), `expiresAt` (when the attempt's logs are dropped whole: the latest of
+  `issuedAt`, `lastAcceptedAt` and the time the attempt's slot released it, plus the config's
+  `logs.retentionMs`; null while the slot still holds the attempt, which schedules no expiry however
+  quiet it is), `captures` and `capturesEnded` (captures with a batch, and
+  those whose final batch arrived), `keptBytes` (UTF-8 bytes of `text` held) and `truncatedBatches`.
+  `limit` runs from 1 to `runtimeAssignmentLogReadContract.maximumAttemptPageSize` (256).
+- `requests.service.getRuntimeAssignmentLogs { identity, serviceId, assignmentId, cursor, limit }`
+  answers an `IRuntimeAssignmentLogPage { attempt, records, nextCursor }`: one attempt's records in
+  delivery order, captures in the order the first batch of each was accepted and each capture by
+  sequence. `limit` counts records, from 1 to `maximumPageRecords` (256). A page also ends before
+  its records would hold more than `maximumPageEntries` (4 096, four full batches) entries or its
+  canonical JSON would cross `maximumPageCanonicalBytes` (2 MiB), so it may hold fewer than `limit`
+  while `nextCursor` names more. One batch always fits: its record is smaller than the report that
+  carried it.
+
+A `TRuntimeAssignmentLogRecord` is one of three, so nothing is ever dropped silently:
+
+- `batch { captureId, sequence, acceptedAt, final, entries }`: a batch as the node sent it, its
+  lines and loss entries unchanged;
+- `gap { captureId, fromSequence, toSequence }`: sequence numbers whose batches Cloudly did not
+  receive or no longer holds, its own loss: numbers that never arrived and that the node did not
+  account for, and accepted batches whose payload Cloudly has lost. Numbers the node skipped are not
+  a gap: the batch after them opens with the `not-kept` loss that counts them;
+- `truncated { captureId, throughSequence, reasons, batches, bytes, entries }`: every batch Cloudly
+  held of the capture up to `throughSequence` and has since dropped, oldest first, with exact counts.
+  `reasons` (`TRuntimeAssignmentLogTruncationReason`, duplicate-free, in the canonical order
+  `batch-cap`, `byte-cap`, `retention`) are `batch-cap`, Cloudly's bound on how many batches it
+  keeps of one attempt, `byte-cap`, its bound on the text it keeps of one attempt, and `retention`,
+  the config's `logs.retentionMs`. A node delivers about once a second, so a workload that writes
+  now and then sends many small batches, which the batch bound holds where the text bound would
+  not. Gaps inside the range are no longer stated.
+
+`snapshotRuntimeAssignmentLogPage` enforces that order: a capture's records are contiguous and never
+return, a truncation only opens its capture, a gap starts at the number after the record before it,
+a gap never directly follows another gap (the controller states adjacent losses, received or not,
+as one gap, across pages too), a batch continues at the next number or, past numbers the node
+skipped, opens with a `not-kept` loss, and nothing follows a final batch. A page may resume inside a
+capture, so a capture's first record on a page is not judged against what came before it. A `nextCursor` is
+an opaque identifier, null on the last page, and never follows an empty page.
+`snapshotRuntimeAssignmentLogAttemptView` and `snapshotRuntimeAssignmentLogAttemptPage` read the
+attempt side. `bindRuntimeAssignmentLogAttemptPage(page, serviceId, limit)` and
+`bindRuntimeAssignmentLogPage(page, serviceId, assignmentId, limit)` match an answer to its read. A
+refused read carries `IRuntimeAssignmentLogReadErrorData { code, retryable: false }`, where
+`TRuntimeAssignmentLogReadRefusal` is `attempt-not-found` (no logs of that attempt for that
+service), `cursor-invalid` or `limit-invalid`.
+
 **Stats.** `IReq_Controller_Pallet_ReadRuntimeAssignmentStats` (`readRuntimeAssignmentStats`)
 reads one sample of one attempt: `{ session, assignment }` names the exact current revision, and
 the answer is `{ status: 'sampled', assignment, sample }` or `{ status: 'not-running', assignment,
@@ -1906,7 +1962,8 @@ evidence has not arrived). `refused` carries a `TServiceRuntimeRefusal`:
 `volume-options-unsupported`, `volume-shared-unsupported`, `volume-placement-unsupported`,
 `volume-initialization-unknown`, `volume-ownership-unknown`, `objectstorage-endpoint-missing`,
 `objectstorage-binding-invalid`, `objectstorage-environment-conflict`,
-`objectstorage-endpoint-unselected`, `database-endpoint-missing` or `database-endpoint-unselected`.
+`objectstorage-endpoint-unselected`, `database-endpoint-missing`, `database-endpoint-unselected`,
+`cluster-relay-missing` or `cluster-ingress-environment-conflict`.
 The first four volume
 refusals name what a local storage claim cannot express: a volume driver other than a node-local
 directory, driver or mount options, one volume mounted by more than one replica or service, and a
@@ -1916,7 +1973,12 @@ refusals, `objectstorage-endpoint-missing`, `objectstorage-binding-invalid` and
 `objectstorage-environment-conflict` name why a binding's environment cannot be composed, see
 [Object-Storage Binding Environment](#object-storage-binding-environment), and
 `objectstorage-endpoint-unselected`, like `database-endpoint-unselected`, names an endpoint the
-workload's network does not select (below). `ports-too-many` is a service declaring more target ports than the
+workload's network does not select (below). The two `cluster-` refusals concern only a cluster's
+designated ingress service, whose environment `composeClusterIngressEnvironment` composes, see
+[Cluster Ingress](#cluster-ingress): `cluster-relay-missing` is a cluster that states no valid
+relay, which an administrator enables with `enableClusterRelay`, and
+`cluster-ingress-environment-conflict` an ingress service whose own environment already sets one of
+`clusterIngressEnvironmentNames`. `ports-too-many` is a service declaring more target ports than the
 `runtimeConfigContract.maximumTargetPorts` (64) a config carries, where `port-protocol-unsupported`
 is about one port's protocol; `node-authority-unsupported` is a node whose session resolved fewer
 workload authorities than the spec needs, refused at the admit step rather than handed to a node
@@ -2003,6 +2065,9 @@ through every slot on a cluster's nodes; `limit` runs from 1 to
 `serviceRuntimeContract.maximumAssignmentPageSize` (256), and Cloudly refuses any other.
 `requests.service.pushServiceRuntimeChanged { serviceId }` tells admin UIs that a service's spec,
 status or assignments changed; it carries no state, so a late push never overwrites a newer read.
+A slot's replaced and released attempts leave these views; their kept workload logs stay readable
+through `getServiceRuntimeLogAttempts` and `getRuntimeAssignmentLogs` (see
+[Runtime Workload Logs, Stats, Exec And Volume Archives](#runtime-workload-logs-stats-exec-and-volume-archives)).
 
 ### Node Retirement
 
@@ -3245,6 +3310,24 @@ already applied and ignores a hint that is not newer, so a lost or repeated hint
 `SERVEZONE_CLUSTER_RELAY_ORIGIN` (the exact `IClusterRelay.origin`) and `SERVEZONE_CLUSTER_NODE_ID`
 (the node the assignment runs on). Cloudly writes them; the ingress reads them.
 
+`composeClusterIngressEnvironment(serviceEnvironment, { relayOrigin, nodeId })` is how Cloudly
+writes the two non-secret names for one replica of the designated ingress service: the service's
+own environment with `SERVEZONE_CLUSTER_RELAY_ORIGIN` and `SERVEZONE_CLUSTER_NODE_ID` added
+(`IClusterIngressLaunchFacts`; `relayOrigin` is the cluster's `IClusterRelay.origin` or `null`,
+`nodeId` the node the replica's assignment runs on). The bearer is not added: it reaches the
+workload through its secret manifest. The answer is a `TClusterIngressEnvironmentComposition`:
+
+| `status` | `reason` | When |
+| --- | --- | --- |
+| `composed` | — | `environment` is the service's own with both names added. |
+| `refused` | `cluster-relay-missing` | `relayOrigin` is `null` or `validateClusterRelay` refuses it. Answered first. |
+| `refused` | `cluster-ingress-environment-conflict` | The service's own environment already sets one of the three names; the message names every such key and no value. |
+
+`refused` carries the contract's `message`, and both reasons are `TClusterIngressEnvironmentRefusal`,
+members of `TServiceRuntimeRefusal`. A `nodeId` that is not a node id is the caller's error and is
+thrown. Both values are part of what the replica runs, so a moved relay or another node is another
+environment. The inputs are not changed.
+
 A cluster born on Pallet has no cutover scope, census or target set, so its ingress is designated by
 `IClusterIngressDesignation { id, generation, previous, clusterId, runtime, ingressServiceId,
 preparedSpecRevision, credential, actorId, acceptedAt, digest }` instead of the fleet-cutover

package/ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@serve.zone/interfaces',
-  version: '32.45.0',
+  version: '32.47.0',
   description: 'Shared TypeScript interfaces and TypedRequest contracts for the serve.zone ecosystem.'
 }

package/ts/data/clusteringress.ts

@@ -3,6 +3,8 @@ import { isUtcCalendarDay } from '../private/utccalendarday.js';
 import { validateProtocolOfferOn } from '../protocol.js';
 import { hasExactKeys, readExactDataRecord } from '../private/dataproperties.js';
 import { isClusterNodeId } from '../private/nodeidentifier.js';
+import { validateClusterRelay } from './cluster.js';
+import type { TServiceRuntimeRefusal } from './serviceruntime.js';
 
 /**
  * The cluster ingress contracts.
@@ -440,6 +442,74 @@ export const clusterIngressEnvironmentNames = Object.freeze({
   nodeId: 'SERVEZONE_CLUSTER_NODE_ID',
 } as const);
 
+/**
+ * The service runtime refusals the ingress environment reports: `cluster-relay-missing`, a cluster
+ * that states no valid relay, and `cluster-ingress-environment-conflict`, a service environment
+ * that already sets one of `clusterIngressEnvironmentNames`.
+ */
+export type TClusterIngressEnvironmentRefusal = Extract<
+  TServiceRuntimeRefusal,
+  'cluster-relay-missing' | 'cluster-ingress-environment-conflict'
+>;
+
+/** What the designated ingress service runs with, or the one reason it cannot be composed. */
+export type TClusterIngressEnvironmentComposition =
+  | { status: 'composed'; environment: Record<string, string> }
+  | { status: 'refused'; reason: TClusterIngressEnvironmentRefusal; message: string };
+
+/** The facts Cloudly states about one replica of the designated ingress service. */
+export interface IClusterIngressLaunchFacts {
+  /** The cluster's `IClusterRelay.origin`, or `null` when the cluster states no relay. */
+  relayOrigin: string | null;
+  /** The node the replica's assignment runs on. */
+  nodeId: string;
+}
+
+/**
+ * The environment one replica of a cluster's designated ingress service runs with: the service's
+ * own values with `relayOrigin` and `nodeId` added under their `clusterIngressEnvironmentNames`.
+ *
+ * The bearer is not added here: it is a secret and reaches the workload through its secret manifest.
+ * Its name is still Cloudly's, so a service environment that sets any of the three names is
+ * refused `cluster-ingress-environment-conflict`, naming every such key and none of their values,
+ * rather than having one value silently replace another. A relay origin that is `null` or that
+ * `validateClusterRelay` refuses is `cluster-relay-missing`, answered before a conflict. A node id
+ * that is not a node id is the caller's error, not a refusal: the assignment named it, and it is
+ * thrown. The inputs are not changed. Both values are part of what the replica runs, so a relay or
+ * node that changes is a new environment.
+ */
+export const composeClusterIngressEnvironment = (
+  serviceEnvironmentArg: Readonly<Record<string, string>>,
+  launchArg: IClusterIngressLaunchFacts,
+): TClusterIngressEnvironmentComposition => {
+  const { relayOrigin, nodeId } = launchArg;
+  if (!isClusterNodeId(nodeId)) {
+    throw new Error('cluster ingress environment requires a node id');
+  }
+  if (relayOrigin === null) {
+    return { status: 'refused', reason: 'cluster-relay-missing',
+      message: 'cluster ingress has no relay origin: the cluster states no relay' };
+  }
+  const relayErrors = validateClusterRelay({ origin: relayOrigin });
+  if (relayErrors.length > 0) {
+    return { status: 'refused', reason: 'cluster-relay-missing',
+      message: `cluster ingress has no relay origin: ${relayErrors.join('; ')}` };
+  }
+  const collisions = Object.values(clusterIngressEnvironmentNames)
+    .filter((nameArg) => Object.hasOwn(serviceEnvironmentArg, nameArg))
+    .sort();
+  if (collisions.length > 0) {
+    return { status: 'refused', reason: 'cluster-ingress-environment-conflict',
+      message: `cluster ingress environment collides with ${collisions.join(', ')} already set on the `
+        + 'service' };
+  }
+  return { status: 'composed', environment: {
+    ...serviceEnvironmentArg,
+    [clusterIngressEnvironmentNames.relayOrigin]: relayOrigin,
+    [clusterIngressEnvironmentNames.nodeId]: nodeId,
+  } };
+};
+
 /**
  * Every reason Cloudly refuses a cluster ingress registration, in the order it judges them.
  *

package/ts/data/runtimeassignmentlogs.ts

@@ -1,11 +1,13 @@
 import type { IRuntimeAssignment } from './runtimeassignment.js';
-import { snapshotRuntimeAssignment, validateRuntimeAssignment } from './runtimeassignment.js';
+import { isRuntimeReplicaId, snapshotRuntimeAssignment, validateRuntimeAssignment } from './runtimeassignment.js';
 import type { IRuntimeConfigLogPolicy } from './runtimeconfig.js';
 import type { IRuntimeSessionBinding } from './runtimesession.js';
 import { snapshotRuntimeSessionBinding } from './runtimesession.js';
 import {
-  captureNetwork, networkCounter, networkEqual, networkId, networkList, networkRecord, requireNetwork,
+  captureNetwork, networkCounter, networkEqual, networkId, networkList, networkNodeId, networkRecord, networkSorted,
+  requireNetwork,
 } from '../private/networkcontract.js';
+import { isOneOf, variantOf } from '../private/producercontract.js';
 import { isRuntimeLogPolicy, runtimePolicyLimits } from '../private/runtimepolicy.js';
 
 /** The container output a log line was written to. */
@@ -333,3 +335,306 @@ export const bindReportRuntimeAssignmentLogsResponse = (
     && response.sequence === batch.sequence);
   return response;
 });
+
+// Reading kept logs
+
+/**
+ * Why a controller no longer holds batches it accepted: `batch-cap`, its bound on how many batches
+ * it keeps of one attempt; `byte-cap`, its bound on the text it keeps of one attempt; both drop
+ * the oldest batches first; `retention`, the config's `logs.retentionMs` since it accepted them.
+ * A node delivers about once a second, so a workload that writes now and then sends many small
+ * batches, which the batch bound holds where the text bound would not.
+ */
+export type TRuntimeAssignmentLogTruncationReason = 'batch-cap' | 'byte-cap' | 'retention';
+
+/** One batch the controller accepted and still holds, as the node sent it. */
+export interface IRuntimeAssignmentLogBatchRecord {
+  kind: 'batch';
+  captureId: string;
+  sequence: number;
+  /** When the controller accepted the batch, in milliseconds since the Unix epoch. */
+  acceptedAt: number;
+  final: boolean;
+  entries: TRuntimeAssignmentLogEntry[];
+}
+
+/**
+ * Sequence numbers of one capture whose batches the controller did not receive or no longer holds
+ * (its own loss): numbers that never arrived and that the node did not account for, and batches it
+ * accepted whose payload it has lost. A gap the node left by discarding is not one: the batch
+ * after it opens with the `not-kept` loss that counts it.
+ */
+export interface IRuntimeAssignmentLogGapRecord {
+  kind: 'gap';
+  captureId: string;
+  fromSequence: number;
+  toSequence: number;
+}
+
+/**
+ * Batches of one capture the controller accepted and has since dropped: every batch it held of the
+ * capture up to and including `throughSequence`, oldest first, so a truncation is always the start
+ * of what remains of a capture. Gaps inside that range are no longer stated. The counts are exact.
+ */
+export interface IRuntimeAssignmentLogTruncationRecord {
+  kind: 'truncated';
+  captureId: string;
+  throughSequence: number;
+  /** In the canonical order `batch-cap`, `byte-cap`, `retention`; duplicate-free, at least one. */
+  reasons: TRuntimeAssignmentLogTruncationReason[];
+  /** The accepted batches dropped, at least one and at most `throughSequence`. */
+  batches: number;
+  /** The UTF-8 bytes of `text` the dropped batches held. */
+  bytes: number;
+  /** The entries, lines and losses alike, the dropped batches held. */
+  entries: number;
+}
+
+export type TRuntimeAssignmentLogRecord =
+  | IRuntimeAssignmentLogBatchRecord | IRuntimeAssignmentLogGapRecord | IRuntimeAssignmentLogTruncationRecord;
+
+/**
+ * One attempt whose logs the controller keeps, as an operator reads it. The controller keeps an
+ * attempt's logs from the moment it issues the attempt, so an attempt that failed before writing a
+ * line is listed with nothing held, and an attempt its slot released stays readable until
+ * `expiresAt`.
+ */
+export interface IRuntimeAssignmentLogAttemptView {
+  serviceId: string;
+  replicaId: string;
+  nodeId: string;
+  assignmentId: string;
+  attemptId: string;
+  /** When the controller issued the attempt, in milliseconds since the Unix epoch. */
+  issuedAt: number;
+  /** When it last accepted a batch of the attempt; null until it accepted one. */
+  lastAcceptedAt: number | null;
+  /**
+   * When the controller drops the attempt's logs whole: the latest of `issuedAt`, `lastAcceptedAt`
+   * and the time the attempt's slot released it, plus the attempt's `logs.retentionMs`. Null while
+   * the slot still holds the attempt: no expiry is scheduled, however quiet the attempt is.
+   */
+  expiresAt: number | null;
+  /** The captures of the attempt the controller accepted a batch of. */
+  captures: number;
+  /** Those of them whose final batch it accepted. */
+  capturesEnded: number;
+  /** The UTF-8 bytes of `text` the controller holds of the attempt. */
+  keptBytes: number;
+  /** The accepted batches of the attempt it has since dropped, stated in truncation records. */
+  truncatedBatches: number;
+}
+
+/** One page of a service's attempts, newest issued first, ties in `assignmentId` order. */
+export interface IRuntimeAssignmentLogAttemptPage {
+  attempts: IRuntimeAssignmentLogAttemptView[];
+  /** Where the next page starts; null when this page is the last. Opaque to the reader. */
+  nextCursor: string | null;
+}
+
+/**
+ * One page of one attempt's kept logs, in delivery order: captures in the order the controller
+ * first accepted a batch of each, and within a capture by sequence. A capture's truncation record
+ * comes first, a gap record directly before the batch that follows it.
+ */
+export interface IRuntimeAssignmentLogPage {
+  attempt: IRuntimeAssignmentLogAttemptView;
+  records: TRuntimeAssignmentLogRecord[];
+  /** Where the next page starts; null when this page is the last. Opaque to the reader. */
+  nextCursor: string | null;
+}
+
+/**
+ * Why a controller refuses a log read: `attempt-not-found` (it keeps no logs of that attempt for
+ * that service, never did or no longer does), `cursor-invalid` (a cursor it did not hand out for
+ * this read) and `limit-invalid` (a `limit` outside its bound).
+ */
+export type TRuntimeAssignmentLogReadRefusal = 'attempt-not-found' | 'cursor-invalid' | 'limit-invalid';
+
+/** The named refusal of a log read, sent as `TypedResponseError.errorData`. */
+export interface IRuntimeAssignmentLogReadErrorData {
+  code: TRuntimeAssignmentLogReadRefusal;
+  retryable: false;
+}
+
+export const runtimeAssignmentLogReadContract = Object.freeze({
+  error: 'Runtime assignment log read contract rejected.',
+  /** The largest `limit` `getServiceRuntimeLogAttempts` accepts; a controller refuses a larger one. */
+  maximumAttemptPageSize: 256,
+  /** The largest `limit` `getRuntimeAssignmentLogs` accepts, counted in records. */
+  maximumPageRecords: 256,
+  /**
+   * The most entries the records of one log page hold together: four full batches. It keeps a page
+   * within the node count every captured value is held to; one batch always fits.
+   */
+  maximumPageEntries: 4 * runtimeAssignmentLogContract.maximumEntriesPerBatch,
+  /**
+   * The canonical JSON of a whole log page is at most two mebibytes, and the controller closes a
+   * page before the next record would cross it. One batch always fits: its record is smaller than
+   * the report that carried it, which `maximumReportCanonicalBytes` bounded.
+   */
+  maximumPageCanonicalBytes: 2 * runtimeAssignmentLogContract.maximumReportCanonicalBytes,
+} as const);
+
+const readFail = (): never => { throw new Error(runtimeAssignmentLogReadContract.error); };
+const readSafe = <T>(operationArg: () => T): T => { try { return operationArg(); } catch { return readFail(); } };
+/** The canonical order of `reasons`. */
+const truncationReasons = ['batch-cap', 'byte-cap', 'retention'] as const;
+const cursorOf = (valueArg: unknown): string | null => valueArg === null ? null : networkId(valueArg);
+
+const batchRecordOf = (valueArg: unknown): IRuntimeAssignmentLogBatchRecord => {
+  const value = networkRecord(valueArg, ['kind', 'captureId', 'sequence', 'acceptedAt', 'final', 'entries']);
+  const entries = networkList(value.entries, runtimeAssignmentLogContract.maximumEntriesPerBatch, entryOf);
+  requireNetwork(value.kind === 'batch' && typeof value.final === 'boolean' && (entries.length > 0 || value.final));
+  return {
+    kind: 'batch', captureId: networkId(value.captureId), sequence: networkCounter(value.sequence),
+    acceptedAt: networkCounter(value.acceptedAt), final: value.final, entries,
+  };
+};
+
+const gapRecordOf = (valueArg: unknown): IRuntimeAssignmentLogGapRecord => {
+  const value = networkRecord(valueArg, ['kind', 'captureId', 'fromSequence', 'toSequence']);
+  const fromSequence = networkCounter(value.fromSequence), toSequence = networkCounter(value.toSequence);
+  requireNetwork(value.kind === 'gap' && fromSequence <= toSequence);
+  return { kind: 'gap', captureId: networkId(value.captureId), fromSequence, toSequence };
+};
+
+const truncationRecordOf = (valueArg: unknown): IRuntimeAssignmentLogTruncationRecord => {
+  const value = networkRecord(
+    valueArg, ['kind', 'captureId', 'throughSequence', 'reasons', 'batches', 'bytes', 'entries'],
+  );
+  requireNetwork(value.kind === 'truncated');
+  const throughSequence = networkCounter(value.throughSequence);
+  const reasons = networkSorted(networkList(value.reasons, truncationReasons.length, (reasonArg) => {
+    requireNetwork(isOneOf(truncationReasons, reasonArg));
+    return reasonArg;
+  }), (reasonArg) => String(truncationReasons.indexOf(reasonArg)));
+  requireNetwork(reasons.length > 0);
+  return {
+    kind: 'truncated', captureId: networkId(value.captureId), throughSequence, reasons,
+    batches: networkCounter(value.batches, throughSequence), bytes: countOf(value.bytes), entries: countOf(value.entries),
+  };
+};
+
+const recordOf = (valueArg: unknown): TRuntimeAssignmentLogRecord => {
+  const kind = variantOf(valueArg, 'kind');
+  if (kind === 'batch') return batchRecordOf(valueArg);
+  if (kind === 'gap') return gapRecordOf(valueArg);
+  return truncationRecordOf(valueArg);
+};
+
+const opensWithNotKept = (recordArg: IRuntimeAssignmentLogBatchRecord): boolean => {
+  const first = recordArg.entries[0];
+  return first !== undefined && first.kind === 'lost' && first.reason === 'not-kept';
+};
+
+/**
+ * Delivery order within a page. Each capture's records are contiguous and never return once
+ * another capture's began. A truncation only opens its capture. Every other record continues where
+ * the one before it ended: a gap at the next number, never right after another gap, and a batch at
+ * the next number or, past numbers the node skipped, opening with the `not-kept` loss that counts
+ * them. Nothing follows a final batch.
+ */
+const requireDeliveryOrder = (recordsArg: TRuntimeAssignmentLogRecord[]): void => {
+  const begun = new Set<string>();
+  let previous: TRuntimeAssignmentLogRecord | null = null;
+  for (const record of recordsArg) {
+    const continues: boolean = previous !== null && previous.captureId === record.captureId;
+    if (!continues) {
+      requireNetwork(!begun.has(record.captureId));
+      begun.add(record.captureId);
+    }
+    if (previous === null || !continues) {
+      previous = record;
+      continue;
+    }
+    requireNetwork(record.kind !== 'truncated' && !(previous.kind === 'batch' && previous.final));
+    const next = (previous.kind === 'batch' ? previous.sequence
+      : previous.kind === 'gap' ? previous.toSequence : previous.throughSequence) + 1;
+    if (record.kind === 'gap') {
+      requireNetwork(previous.kind !== 'gap' && record.fromSequence === next);
+    } else {
+      requireNetwork(record.sequence === next || (record.sequence > next && opensWithNotKept(record)));
+    }
+    previous = record;
+  }
+};
+
+const attemptOf = (valueArg: unknown): IRuntimeAssignmentLogAttemptView => {
+  const value = networkRecord(valueArg, [
+    'serviceId', 'replicaId', 'nodeId', 'assignmentId', 'attemptId', 'issuedAt', 'lastAcceptedAt', 'expiresAt',
+    'captures', 'capturesEnded', 'keptBytes', 'truncatedBatches',
+  ]);
+  requireNetwork(isRuntimeReplicaId(value.replicaId));
+  const issuedAt = networkCounter(value.issuedAt);
+  const lastAcceptedAt = value.lastAcceptedAt === null ? null : networkCounter(value.lastAcceptedAt);
+  const expiresAt = value.expiresAt === null ? null : networkCounter(value.expiresAt);
+  const captures = countOf(value.captures), capturesEnded = countOf(value.capturesEnded);
+  const keptBytes = countOf(value.keptBytes), truncatedBatches = countOf(value.truncatedBatches);
+  requireNetwork((expiresAt === null || expiresAt > Math.max(issuedAt, lastAcceptedAt ?? issuedAt))
+    && capturesEnded <= captures);
+  // Nothing accepted means nothing captured, held or dropped.
+  requireNetwork(lastAcceptedAt !== null || (captures === 0 && keptBytes === 0 && truncatedBatches === 0));
+  return {
+    serviceId: networkId(value.serviceId), replicaId: value.replicaId, nodeId: networkNodeId(value.nodeId),
+    assignmentId: networkId(value.assignmentId), attemptId: networkId(value.attemptId), issuedAt, lastAcceptedAt,
+    expiresAt, captures, capturesEnded, keptBytes, truncatedBatches,
+  };
+};
+
+const pageLimitOf = (limitArg: unknown, maximumArg: number): number => networkCounter(limitArg, maximumArg);
+
+/** Detached exact-schema snapshot of one attempt view. */
+export const snapshotRuntimeAssignmentLogAttemptView = (valueArg: unknown): IRuntimeAssignmentLogAttemptView =>
+  readSafe(() => attemptOf(captureNetwork(valueArg)));
+
+/** Detached exact-schema snapshot of one attempt page, in its one order. */
+export const snapshotRuntimeAssignmentLogAttemptPage = (valueArg: unknown): IRuntimeAssignmentLogAttemptPage =>
+  readSafe(() => {
+    const value = networkRecord(captureNetwork(valueArg), ['attempts', 'nextCursor']);
+    const attempts = networkList(value.attempts, runtimeAssignmentLogReadContract.maximumAttemptPageSize, attemptOf);
+    attempts.forEach((attemptArg, indexArg) => {
+      const before = attempts[indexArg - 1];
+      requireNetwork(before === undefined || before.issuedAt > attemptArg.issuedAt
+        || (before.issuedAt === attemptArg.issuedAt && before.assignmentId < attemptArg.assignmentId));
+    });
+    const nextCursor = cursorOf(value.nextCursor);
+    requireNetwork(nextCursor === null || attempts.length > 0);
+    return { attempts, nextCursor };
+  });
+
+/** Detached exact-schema snapshot of one log page, bounded and in delivery order. */
+export const snapshotRuntimeAssignmentLogPage = (valueArg: unknown): IRuntimeAssignmentLogPage => readSafe(() => {
+  const value = networkRecord(
+    captureNetwork(valueArg, runtimeAssignmentLogReadContract.maximumPageCanonicalBytes),
+    ['attempt', 'records', 'nextCursor'],
+  );
+  const records = networkList(value.records, runtimeAssignmentLogReadContract.maximumPageRecords, recordOf);
+  requireNetwork(records.reduce((totalArg, recordArg) => totalArg + (recordArg.kind === 'batch' ? recordArg.entries.length : 0), 0)
+    <= runtimeAssignmentLogReadContract.maximumPageEntries);
+  requireDeliveryOrder(records);
+  const nextCursor = cursorOf(value.nextCursor);
+  requireNetwork(nextCursor === null || records.length > 0);
+  return { attempt: attemptOf(value.attempt), records, nextCursor };
+});
+
+/** Matches an attempt page to the read that asked for it: that service's attempts, within its limit. */
+export const bindRuntimeAssignmentLogAttemptPage = (
+  valueArg: unknown, serviceIdArg: string, limitArg: number,
+): IRuntimeAssignmentLogAttemptPage => readSafe(() => {
+  const page = snapshotRuntimeAssignmentLogAttemptPage(valueArg);
+  const limit = pageLimitOf(limitArg, runtimeAssignmentLogReadContract.maximumAttemptPageSize);
+  requireNetwork(page.attempts.length <= limit && page.attempts.every((attemptArg) => attemptArg.serviceId === serviceIdArg));
+  return page;
+});
+
+/** Matches a log page to the read that asked for it: that attempt of that service, within its limit. */
+export const bindRuntimeAssignmentLogPage = (
+  valueArg: unknown, serviceIdArg: string, assignmentIdArg: string, limitArg: number,
+): IRuntimeAssignmentLogPage => readSafe(() => {
+  const page = snapshotRuntimeAssignmentLogPage(valueArg);
+  const limit = pageLimitOf(limitArg, runtimeAssignmentLogReadContract.maximumPageRecords);
+  requireNetwork(page.records.length <= limit && page.attempt.serviceId === serviceIdArg
+    && page.attempt.assignmentId === assignmentIdArg);
+  return page;
+});

package/ts/data/serviceruntime.ts

@@ -92,6 +92,7 @@ const serviceRuntimeRefusals = [
   'objectstorage-endpoint-missing', 'objectstorage-binding-invalid', 'objectstorage-environment-conflict',
   'objectstorage-endpoint-unselected', 'database-endpoint-missing', 'database-endpoint-unselected',
   'published-port-reserved', 'published-port-symmetric-without-egress', 'readiness-network-unsupported',
+  'cluster-relay-missing', 'cluster-ingress-environment-conflict',
 ] as const;
 
 /**
@@ -165,6 +166,15 @@ export type TServiceRuntimeWaitingReason = typeof serviceRuntimeWaitingReasons[n
  * `unsupported-readiness`). `isServiceRuntimeReadinessReachable` is this rule; a producer answers
  * what it returns. The spec snapshot still reads such a spec, so a stored one stays readable and
  * is answered with this refusal rather than refused as data.
+ *
+ * The two `cluster-` refusals name why the environment of a cluster's designated ingress service
+ * cannot be composed (`composeClusterIngressEnvironment`); a service no designation names is never
+ * judged by them. `cluster-relay-missing` is a cluster that states no valid relay
+ * (`IClusterRelay`): the ingress registers through its relay's exact origin, so without one it has
+ * nothing to start with, and an administrator enables the relay (`enableClusterRelay`).
+ * `cluster-ingress-environment-conflict` is an ingress service whose own environment already sets
+ * one of `clusterIngressEnvironmentNames`: Cloudly owns those names, and a service value would
+ * otherwise silently replace Cloudly's or be replaced by it.
  */
 export type TServiceRuntimeRefusal = typeof serviceRuntimeRefusals[number];
 

package/ts/private/networkcontract.ts

@@ -46,14 +46,23 @@ export const networkSafe = <T>(operation: () => T): T => {
   }
 };
 
-/** Bound decoded transport input before canonicalization; never execute getters/toJSON. */
-export const captureNetwork = (value: unknown): unknown => networkSafe(() => {
+/** The canonical JSON bound every runtime network contract captures with, unless it states its own. */
+export const networkMaximumCanonicalBytes = 1_048_576;
+
+/**
+ * Bound decoded transport input before canonicalization; never execute getters/toJSON.
+ * `maximumBytes` bounds the whole value's canonical JSON; a single string stays within
+ * `networkMaximumCanonicalBytes` whatever the whole may hold.
+ */
+export const captureNetwork = (
+  value: unknown, maximumBytes = networkMaximumCanonicalBytes,
+): unknown => networkSafe(() => {
   const encoder = new TextEncoder(), ancestors = new Set<object>();
   let nodes = 0, bytes = 0;
   const account = (text: string) => {
-    requireNetwork(text.length <= 1_048_576);
+    requireNetwork(text.length <= networkMaximumCanonicalBytes);
     bytes += encoder.encode(text).length;
-    requireNetwork(bytes <= 1_048_576);
+    requireNetwork(bytes <= maximumBytes);
   };
   const copy = (item: unknown, depth: number): unknown => {
     requireNetwork(++nodes <= 65_536 && depth <= 24);
@@ -90,7 +99,7 @@ export const captureNetwork = (value: unknown): unknown => networkSafe(() => {
     } finally { ancestors.delete(item); }
   };
   const result = copy(value, 0);
-  requireNetwork(encoder.encode(networkCanonical(result)).length <= 1_048_576);
+  requireNetwork(encoder.encode(networkCanonical(result)).length <= maximumBytes);
   return result;
 });
 

package/ts/requests/service.ts

@@ -19,6 +19,10 @@ import type {
 import type { TSha256Digest } from '../data/immutableimage.js';
 import type { ILegacyServiceRuntimeAttestation } from '../data/deploymentlegacyimport.js';
 import type { IProtocolOffer } from '../protocol.js';
+import type {
+  IRuntimeAssignmentLogAttemptPage,
+  IRuntimeAssignmentLogPage,
+} from '../data/runtimeassignmentlogs.js';
 import type {
   IServiceRuntimeAssignmentView,
   IServiceRuntimeSpec,
@@ -629,6 +633,52 @@ extends plugins.typedrequestInterfaces.implementsTR<
   };
 }
 
+/**
+ * The attempts of one service whose workload logs Cloudly keeps, newest issued first, a page at a
+ * time: a slot's replaced and released attempts stay readable until their logs expire, which no
+ * single response bounds. `limit` runs from 1 to
+ * `runtimeAssignmentLogReadContract.maximumAttemptPageSize`; Cloudly refuses any other with
+ * `IRuntimeAssignmentLogReadErrorData`. `cursor` is null for the first page, then the page's own
+ * `nextCursor`. `bindRuntimeAssignmentLogAttemptPage` checks an answer against its read.
+ */
+export interface IRequest_Any_Cloudly_GetServiceRuntimeLogAttempts
+extends plugins.typedrequestInterfaces.implementsTR<
+  plugins.typedrequestInterfaces.ITypedRequest,
+  IRequest_Any_Cloudly_GetServiceRuntimeLogAttempts
+> {
+  method: 'getServiceRuntimeLogAttempts';
+  request: {
+    identity: IIdentity;
+    serviceId: string;
+    cursor: string | null;
+    limit: number;
+  };
+  response: IRuntimeAssignmentLogAttemptPage;
+}
+
+/**
+ * One attempt's kept workload logs, a page at a time in delivery order, with the gaps Cloudly found
+ * and the truncations it made stated in place. `limit` counts records and runs from 1 to
+ * `runtimeAssignmentLogReadContract.maximumPageRecords`; a page also ends before it would cross
+ * `maximumPageEntries` or `maximumPageCanonicalBytes`, so it may hold fewer records than `limit`
+ * while `nextCursor` still names more. `bindRuntimeAssignmentLogPage` checks an answer against its read.
+ */
+export interface IRequest_Any_Cloudly_GetRuntimeAssignmentLogs
+extends plugins.typedrequestInterfaces.implementsTR<
+  plugins.typedrequestInterfaces.ITypedRequest,
+  IRequest_Any_Cloudly_GetRuntimeAssignmentLogs
+> {
+  method: 'getRuntimeAssignmentLogs';
+  request: {
+    identity: IIdentity;
+    serviceId: string;
+    assignmentId: string;
+    cursor: string | null;
+    limit: number;
+  };
+  response: IRuntimeAssignmentLogPage;
+}
+
 /**
  * Cloudly -> admin UIs: a service's runtime spec, status or assignments changed. It carries no
  * state, so a late or repeated push can never overwrite a newer read; the UI reads the views again.