@serve.zone/appstore 33.7.0 → 33.8.0

package/changelog.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## 2026-10-14 - 33.8.0
+
+### Features
+
+- publish the Cloudly 1.29.1 template for Cloudly 33.20.1 (cloudly)
+  - Add Cloudly app version 1.29.1, pinned to the Cloudly 33.20.1 multi-platform image index `code.foss.global/serve.zone/cloudly@sha256:fa7aaa4f68006421aa674ff8afef320919904091469909bbea5ab87bfe500296` (tag `v33.20.1`), record it in `catalog.lock.json`, qualify it in `test/test.cloudlytemplates.node.ts` and make it Cloudly's latest version.
+  - Keep the 1.29.0 runtime contract: 1.29.1 differs from 1.29.0 only in its image and changelog, so it keeps `backupExclusions` for `runtime-logs/` and `requiresFeatures: ['storage.object-storage.backup-exclusions']` and is the upgrade target of 1.29.0, 1.28.6, 1.28.5 and 1.28.3; a host without the feature lists it and refuses it, and its newest Cloudly target stays 1.28.4. Cloudly's `ts_migration`, Dockerfile and `qenv.yml` are unchanged from `v33.20.0` to `v33.20.1`: no data step, no new collection, data version 0.30.0, Onebox minimum 33.8.0.
+  - Keep breaking, manual-review, migration-required and backup-before-upgrade: hosts read the flags from the target version alone, every upgrade from 1.28.6 or earlier takes the workload log ledger rows of Cloudly 33.20.0 that 33.19.2 and earlier refuse, and every upgrade from 1.26.0 or earlier runs the forward-only 0.28.0 to 0.30.0 steps, so a return to an earlier template needs the backup taken before the upgrade.
+  - The 1.29.1 changelog carries the changes of 33.20.1 (a Cloudly restart no longer blanks cluster route tables: the first table after a start is the full one, a reporter is one node instance under one credential generation (`nodeInstanceNonce`), a revocation withdraws exactly the revoked routes, unreadable route sources keep the relay's table for at most 180 s; registration refusals Cloudly causes itself are momentary per @serve.zone/interfaces 32.57.0, with the new verdict reasons `node-retiring` and `registration-replay-mismatch`; a credential check fails closed when its store cannot be read), the backup-exclusions declaration and the upgrade path from 1.29.0, 1.28.6, 1.28.5 and 1.28.3, then the 1.29.0 changelog for Cloudly 33.20.0 and every earlier template.
+
 ## 2026-10-14 - 33.7.0
 
 ### Features

package/dist_catalog/catalog.digest

@@ -1 +1 @@
-sha256:747ea478ea4c79aacc22e70a7c9a2568101fb78f8f50373dfbd5fee57fb63c16
+sha256:d4aa2fa25d500b12293a9423cd614034a6fdc12f9bfebaa420320a8e790251b4

package/dist_catalog/catalog.envelope.json

@@ -5,9 +5,9 @@
       "major": 1,
       "minor": 0
     },
-    "revision": 33000007000000,
-    "publishedAt": "2026-10-14T01:06:51.000Z",
-    "sourceRevision": "29eab2aff549d4f31431c45f5308c068d48c3fa9",
+    "revision": 33000008000000,
+    "publishedAt": "2026-10-14T08:37:04.000Z",
+    "sourceRevision": "541e7a5e9bf996466d124f85301049499989c6c2",
     "apps": [
       {
         "info": {
@@ -60,7 +60,7 @@
             "Docs": "https://serve.zone"
           }
         },
-        "latestVersion": "1.29.0",
+        "latestVersion": "1.29.1",
         "versions": [
           {
             "version": "1.0.0",
@@ -8109,6 +8109,195 @@
               }
             },
             "configDigest": "sha256:1544df3a1e4e7d3413118e225e234f81061ed9fbc1d824b5827c4a14a31fd3c9"
+          },
+          {
+            "version": "1.29.1",
+            "config": {
+              "image": "code.foss.global/serve.zone/cloudly@sha256:fa7aaa4f68006421aa674ff8afef320919904091469909bbea5ab87bfe500296",
+              "targetPorts": [
+                {
+                  "name": "web",
+                  "port": 3000,
+                  "protocol": "http",
+                  "default": true
+                }
+              ],
+              "envVars": [
+                {
+                  "key": "SERVEZONE_ENVIRONMENT",
+                  "value": "production",
+                  "description": "Cloudly runtime environment.",
+                  "required": true
+                },
+                {
+                  "key": "SERVEZONE_PUBLIC_ORIGIN",
+                  "value": "https://${SERVICE_DOMAIN}",
+                  "description": "Canonical public Cloudly origin including protocol and optional public port.",
+                  "required": true
+                },
+                {
+                  "key": "SERVEZONE_LISTENPORT",
+                  "value": "3000",
+                  "description": "Internal Cloudly HTTP port inside the container.",
+                  "required": true
+                },
+                {
+                  "key": "SERVEZONE_SSLMODE",
+                  "value": "external",
+                  "description": "Use external TLS termination through Onebox or dcrouter.",
+                  "required": true
+                },
+                {
+                  "key": "SERVEZONE_SECRET_KEYRING_PATH",
+                  "value": "/run/secrets/cloudly-secrets-v2-keyring.json",
+                  "description": "Path to the Cloudly Secrets v2 keyring mounted by Onebox; the keyring value is never placed in the public environment.",
+                  "required": true
+                },
+                {
+                  "key": "SERVEZONE_SECRET_KEYRING_FILE",
+                  "description": "Existing Cloudly Secrets v2 keyring. Onebox stores it encrypted and mounts it only as a root-readable file; retain the existing keyring when upgrading.",
+                  "required": true,
+                  "secret": true,
+                  "delivery": {
+                    "type": "file",
+                    "targetPath": "/run/secrets/cloudly-secrets-v2-keyring.json",
+                    "uid": 0,
+                    "gid": 0,
+                    "mode": 256
+                  }
+                },
+                {
+                  "key": "SERVEZONE_WORKLOADINIT_APPROVAL_PATH",
+                  "value": "/run/secrets/cloudly-workloadinit-approval.json",
+                  "description": "Private WorkloadInit release-approval artifact mounted by Onebox for Coreflow secret-runtime registration.",
+                  "required": true
+                },
+                {
+                  "key": "SERVEZONE_WORKLOADINIT_APPROVAL_FILE",
+                  "description": "Operator-approved WorkloadInit release authority. Onebox stores it encrypted and mounts it only as a root-readable file.",
+                  "required": true,
+                  "secret": true,
+                  "delivery": {
+                    "type": "file",
+                    "targetPath": "/run/secrets/cloudly-workloadinit-approval.json",
+                    "uid": 0,
+                    "gid": 0,
+                    "mode": 256
+                  }
+                },
+                {
+                  "key": "MONGODB_URL",
+                  "value": "${MONGODB_URI}",
+                  "description": "Authenticated MongoDB wire-protocol URL provisioned by Onebox Corestore.",
+                  "required": true
+                },
+                {
+                  "key": "MONGODB_NAME",
+                  "value": "${MONGODB_DATABASE}",
+                  "description": "MongoDB wire-protocol database name provisioned by Onebox Corestore.",
+                  "required": true
+                },
+                {
+                  "key": "MONGODB_USER",
+                  "value": "${MONGODB_USERNAME}",
+                  "description": "MongoDB wire-protocol username provisioned by Onebox Corestore.",
+                  "required": true
+                },
+                {
+                  "key": "MONGODB_PASS",
+                  "value": "${MONGODB_PASSWORD}",
+                  "description": "MongoDB wire-protocol password provisioned by Onebox Corestore.",
+                  "required": true
+                },
+                {
+                  "key": "S3_ENDPOINT",
+                  "value": "onebox-corestore",
+                  "description": "S3 endpoint host provisioned by Onebox Corestore.",
+                  "required": true
+                },
+                {
+                  "key": "S3_PORT",
+                  "value": "9000",
+                  "description": "S3 endpoint port provisioned by Onebox Corestore.",
+                  "required": true
+                },
+                {
+                  "key": "S3_USESSL",
+                  "value": "false",
+                  "description": "Use plain HTTP for internal Corestore S3 traffic on the Onebox network.",
+                  "required": true
+                },
+                {
+                  "key": "S3_BUCKET",
+                  "value": "${S3_BUCKET}",
+                  "description": "Cloudly's own S3 bucket, provisioned by Onebox Corestore: registry storage and kept workload logs.",
+                  "required": true
+                },
+                {
+                  "key": "S3_ACCESSKEY",
+                  "value": "${S3_ACCESS_KEY}",
+                  "description": "S3 access key provisioned by Onebox Corestore.",
+                  "required": true
+                },
+                {
+                  "key": "S3_SECRETKEY",
+                  "value": "${S3_SECRET_KEY}",
+                  "description": "S3 secret key provisioned by Onebox Corestore.",
+                  "required": true
+                }
+              ],
+              "platformRequirements": {
+                "mongodb": true,
+                "s3": true
+              },
+              "backupExclusions": [
+                {
+                  "storageRequest": "legacy-s3",
+                  "keyPrefixes": [
+                    "runtime-logs/"
+                  ],
+                  "reason": "Workload logs; a restore brings back state, not past logs."
+                }
+              ],
+              "platformOidc": {
+                "redirectPath": "/auth/oidc/callback",
+                "roles": [
+                  {
+                    "id": "admin",
+                    "label": "Administrator",
+                    "description": "Full Cloudly administration for this hosted instance."
+                  },
+                  {
+                    "id": "user",
+                    "label": "User",
+                    "description": "Standard authenticated Cloudly access for this hosted instance."
+                  }
+                ],
+                "environmentVariables": {
+                  "issuerUrl": "SERVEZONE_PLATFORM_OIDC_ISSUER",
+                  "clientId": "SERVEZONE_PLATFORM_OIDC_CLIENT_ID",
+                  "clientSecret": "SERVEZONE_PLATFORM_OIDC_CLIENT_SECRET",
+                  "redirectUri": "SERVEZONE_PLATFORM_OIDC_REDIRECT_URI",
+                  "audience": "SERVEZONE_PLATFORM_OIDC_AUDIENCE"
+                },
+                "clientAuthenticationMethod": "client_secret_basic"
+              },
+              "minOneboxVersion": "33.8.0",
+              "changelog": "Move Cloudly to 33.20.1, a fixes-only patch over 33.20.0: a Cloudly restart no longer blanks a cluster's ingress routes, a revocation withdraws exactly the routes it revokes, and a Pallet registration Cloudly cannot serve for a reason of its own is refused as momentary. After each cold start of Cloudly 33.20.0 the cluster relay applied two route tables of 0 routes before the full one, and every site of the cluster answered 502 for about 2.5 minutes. Two causes, both fixed. First, Cloudly published an empty revocation as its first table of a cluster, because it held no table of it yet; now the first pull or refresh compiles the plan and issues the full table, and a relay's read that comes before it waits up to 6 s for it and is otherwise answered { table: null }, so the relay keeps the table it holds and the full one follows as a push. Second, every node registers again after a restart, which opens a new session, and a route endpoint required an observation reported under the current session, so every route waited until each node reported anew. A reporter is now one node instance under one credential generation (nodeInstanceNonce, Pallet's per-process nonce, stored on every session record): a node instance that only registered again keeps its observations, its cutover uplink endpoint and its cluster ingress registration authority until it reports anew, while a new Pallet process, a new credential generation or a deleted node supersedes them at once. A route's identity names the node instance and credential generation instead of the session id and epoch, so such a registration changes no route. Route sources that cannot be read right after a start keep the relay's table for at most 180 s, logged once per cluster; then one line names the clusters still waiting and their routes are withdrawn as before. A revocation is now the held table less the revoked routes and port routes, issued and pushed at once: before, any revoked route identity made Cloudly publish an empty table for the whole cluster before it fetched certificates for the next one, so one node registering again or one changed route blanked every site of the cluster. A revoked ingress designation still withdraws everything, and route sources that cannot be read still withdraw every route. A Pallet registration that fails for a reason of Cloudly's own (a database connection lost, a write conflict, a race with a credential being applied, a controller not ready yet or any unnamed failure) is answered controller-unavailable and a spent budget controller-deadline, the @serve.zone/interfaces 32.57.0 runtime session refusal contract, so the node waits and registers again instead of ending its session; before, only a spent budget or a stopping controller carried the refusal payload, and the node read every other failure as a verdict. Only the named verdicts carry no payload: an inactive credential, a deleted, retiring (node-retiring) or foreign node, an unverifiable carrier, a replay that states another registration (registration-replay-mismatch), a moved controller identity, an invalid body. A credential check whose store cannot be read now fails instead of reading as a credential that does not verify, so a registration whose carrier Cloudly could not check is momentary too. A request is answered at its budget even when its work ignores the abort signal, and each registration refusal is logged once per node and reason, value-free, until the node's next accepted registration. Cloudly 33.20.1 adds no data step and keeps the data version 0.30.0, so an upgrade from 1.29.0 runs no data step; it adds no collection, changes neither the Dockerfile nor the qenv.yml requirements, and the runtime contract is unchanged from 1.29.0 and requires Onebox 33.8.0. Like 1.29.0, this template declares Cloudly's workload logs non-backup data: backupExclusions names the key prefix runtime-logs/ of Cloudly's own bucket (storage request legacy-s3), and requiresFeatures names storage.object-storage.backup-exclusions; a host that does not fulfil the feature keeps listing this version and refuses it as an install or upgrade target. From 1.29.0 the upgrade moves Cloudly from 33.20.0 to 33.20.1. From 1.28.6 or earlier it also takes the changes of Cloudly 33.20.0, including the first start that creates the collection cloudly_runtime_log_inline_payloads and the log ledger rows Cloudly 33.19.2 and earlier refuse, so a return to 1.28.6 or an earlier template needs the backup taken before the upgrade; from 1.28.5 or 1.28.3, which run the Cloudly 33.19.1 image, it also takes the changes of Cloudly 33.19.2, and from 1.28.4 or earlier the template also gains the declaration. The rest of this changelog is that of 1.29.0, the Cloudly 33.20.0 template, and describes the changes of Cloudly 33.20.0 and the releases before it that an upgrade from 1.28.6 or earlier also takes: Move Cloudly to 33.20.0, which serves non-public hostnames to dcrouter alone, writes workload logs as segments and lets a service whose retained rollout plan cannot run be redeployed. A node's cluster ingress serves non-public hostnames, restricted to dcrouter: before, Cloudly compiled only PUBLIC domains into the cluster route table, so a STANDARD hostname got a hub route and a private DNS answer while nothing on the node served it. A domain whose sourceProfileRef is not PUBLIC now compiles, on a moved and on a Pallet-born cluster, into a route whose sourceAllowList is the new setting dcrouterForwardSourceAddresses, the addresses dcrouter's forwarded connections arrive from at a node (set in the dashboard's dcrouter panel or with updateSettings; @serve.zone/interfaces 32.55.1 refuses IPv4-mapped IPv6 prefixes there, write the IPv4 prefix). A change of the setting revokes and reissues the route and refreshes every cluster ingress at once, and the ingress-to-workload grant covers every domain's target port. A non-public domain is left out by name while public routes are still served: while dcrouterForwardSourceAddresses is unset, while the cluster's ingress has not registered route-source-allowlist, and while the live cluster relay speaks @serve.zone/interfaces below 32.55.0. To serve a non-public hostname, the cluster's Coreflow and CoreTraffic must run releases on @serve.zone/interfaces 32.55.0 or later, and CoreTraffic must register the capability. Public port mappings stay public-only. Workload log payloads are written as segments, one object per capture every few minutes instead of one per batch: one object per batch grew Cloudly's bucket by about 7,000 objects an hour. A committed batch keeps its payload inline in the database, in the new collection cloudly_runtime_log_inline_payloads, so it reads as soon as it is committed; the drain then writes a capture's inline payloads as one segment, runtime-logs/<serviceId>/<assignmentId>/<captureId>/<first>-<last>.jsonl, and only after the write moves the ledger rows onto it and removes the inline payloads. A segment is due 5 minutes after its capture's oldest inline payload was committed, at 1 MiB, when the capture's final batch is inline, and early, largest capture first, once all inline payloads pass 16 MiB; at 32 MiB the drain commits nothing more until segments are written, so the queue's bounds refuse new reports with the retryable log-intake-saturated. A stop writes no segment early, and the next start writes the inline payloads when due. Rows committed before this version name their own object as before and age out under the 30-day retention. All payload objects stay below runtime-logs/, the prefix this template declares non-backup. The ledger collection cloudly_runtime_log_batches gains no index, so a start builds none over the existing ledger. A spec-owned service whose retained rollout plan the current runtime cannot execute, such as a Swarm-era promotion written before @serve.zone/interfaces 23.0.0 made the sealed containerInvocation required, can be redeployed: its runtime producer refuses the plan image-plan-missing instead of failing its build step on every pass, and while no deployment operation is in flight the reserve preflight answers GO with the auditable RECOVERY_REDEPLOY_OVER_FAILED_RUNTIME warning, which names the plan and the rules it breaks, instead of NO-GO with CURRENT_RUNTIME_DIGEST_MISSING and RUNTIME_ATTESTATION_MISMATCH. An attested runtime, a mutable current reference, a valid plan over an unattested runtime and an operation in flight still block as before. Cloudly 33.20.0 adds no data step and keeps the data version 0.30.0, so an upgrade from 1.28.6 runs no data step; it changes neither the Dockerfile nor the qenv.yml requirements, and the runtime contract is unchanged from 1.28.6 and requires Onebox 33.8.0. Its first start creates the collection cloudly_runtime_log_inline_payloads. On the platform Corestore 32.2.4 (NoSQLDB 10.6.1) a collection creation waits at most 5 s for the open transactions on the Cloudly database, and a transaction an earlier Cloudly left open by exiting before its stop completed (it logs that its shutdown deadline passed) holds that database until its session expires 30 minutes later, so that first start fails with EFENCE_BUSY until then: after an unclean stop of the previous Cloudly, start this version no earlier than 30 minutes after that exit. Every workload log batch this version commits names where its payload is, and Cloudly 33.19.2 and earlier refuse such a ledger row whenever they read it, so a return to 1.28.6 or an earlier template after this version committed a log batch requires the backup taken before the upgrade. Like 1.28.5 and 1.28.6, this template declares Cloudly's workload logs non-backup data: backupExclusions names the key prefix runtime-logs/ of Cloudly's own bucket, the object storage platformRequirements.s3 provisions (storage request legacy-s3), and requiresFeatures names storage.object-storage.backup-exclusions. A host that fulfils that feature leaves the objects under runtime-logs/ out of Cloudly's backups, and a restore brings back Cloudly's state, not its past workload logs: it neither restores nor removes objects under that prefix, and Cloudly reports a log batch whose payload it no longer has as a gap. A host that does not fulfil the feature keeps listing this version and refuses it as an install or upgrade target. From 1.28.6 the upgrade moves Cloudly from 33.19.2 to 33.20.0. From 1.28.5, which runs the Cloudly 33.19.1 image of 1.28.3, it also takes the changes of Cloudly 33.19.2; from 1.28.4 or earlier the template also gains the declaration. The rest of this changelog is that of 1.28.4, the Cloudly 33.19.2 template, and describes the changes of Cloudly 33.19.2 and the releases before it that an upgrade from 1.28.5 or earlier also takes: Move Cloudly to 33.19.2, a fixes-only patch over 33.19.1. A greenfield deployment, a legacy import's successor included, proves its namespace blank on the nodes of the Pallet clusters (phase switching or pallet) and nowhere else: a reservation whose service had no rollout plan yet asked every registered node, so a fleet that still keeps a Coreflow-owned Swarm cluster without a relay could never reserve one, and a legacy import's reserve preflight answered NO-GO with CORESTORE_INVENTORY_UNAVAILABLE and NAMESPACE_NOT_PROVEN_BLANK. The deployment preflight and reserveServiceDeployment read the same nodes; a Pallet node whose inventory cannot be read still blocks, a fleet with no Pallet cluster node refuses the reservation CORESTORE_INVENTORY_UNAVAILABLE, and an existing service's preflight still reads its plan's clusters. A greenfield rollout without a Pallet runtime spec is refused GREENFIELD_PALLET_RUNTIME_SPEC_REQUIRED, which is not retryable, instead of falling back to Coreflow placement onto Swarm nodes the namespace proof never covered: state the spec with setServiceRuntimeSpec, then promote again. A rollout with a Pallet spec plans onto the spec's nodes as before. A node's network compilation proves each workload lease once and each protected authority revision and allocation eligibility proof once per transaction (before, every lease was read up to three times per compilation and each read re-proved the same revision and proof, about 24 proof reads per endpoint), so a compilation with 100 endpoints reads 539 records instead of 2,839 (7.8 s instead of 19.8 s on Cloudly's test fixture), and every compilation signs and refuses exactly as before. Cloudly 33.19.2 adds no data step and no collection and keeps the data version 0.30.0, so an upgrade from 1.28.3 runs no data step; it changes neither the Dockerfile nor the qenv.yml requirements, and the runtime contract is unchanged from 1.28.3 and requires Onebox 33.8.0. From 1.28.2 or earlier, the changes of Cloudly 33.18.3, 33.18.4, 33.18.5, 33.19.0 and 33.19.1 apply too: Runtime log intake no longer holds a node's request while its payload uploads: an admitted reportRuntimeAssignmentLogs costs one append transaction that queues the batch with its payload in the new collection cloudly_runtime_log_queue, and the node is answered once it commits. A background drain writes the payloads, up to 32 at once, and commits the ledger rows and removes the queue rows in coalesced transactions that fence no controller record; every other node request under a request budget counts as critical work, and the drain waits for it at most 1 s per step. A resend of a batch the ledger or the queue already holds is answered replay with nothing written, and while the queue is at its bounds (1024 batches, 64 MiB, 8 concurrent intakes) a report is refused at once with the retryable log-intake-saturated. Object writes list nothing (@lossless.org/client 1.23.3), workload logs are kept for at least 30 days whatever shorter logs.retentionMs a spec states, and one runtime log intake line, logged on its own timer at most once a minute, reports intake latency, queue depth, the drain and refusals. A node that applied its network head is no longer handed a successor that only drops the head's withdrawals, and a successor of the projection a node's receipt names lists only the tombstones its own step retires, so a node with more than 256 retired leases or handoffs stays within the contract's bound. Route verification reads the certificate a route presents on a connection of its own, without pooling or TLS session resumption, so it records the route's subject alternative names; a connection that exposes no certificate fails with TLS_PEER_CERTIFICATE_UNAVAILABLE and a certificate without names with TLS_SUBJECT_ALT_NAMES_MISSING. readServiceDeploymentStatus (@serve.zone/interfaces 32.54.0) reads a deployment operation's status without changing it, and a failed service deletion answers SERVICE_DELETION_FAILED and logs its cause. The first start after 1.28.2 or earlier creates the collection cloudly_runtime_log_queue. On the platform Corestore 32.2.4 (NoSQLDB 10.6.1) a collection creation waits at most 5 s for the open transactions on the Cloudly database, and a transaction an earlier Cloudly left open by exiting before its stop completed (it logs that its shutdown deadline passed) holds that database until its session expires 30 minutes later, so that first start fails with EFENCE_BUSY until then: after an unclean stop of the previous Cloudly, start this version no earlier than 30 minutes after that exit. Cloudly 33.19.1 adds no data step and keeps the data version 0.30.0, so an upgrade from 1.28.2, 1.28.1 or 1.28.0 runs no data step; it changes neither the Dockerfile nor the qenv.yml requirements, and the runtime contract is unchanged from 1.28.2 and requires Onebox 33.8.0. From 1.28.1 or earlier, Cloudly 33.18.2 advances a node's network configuration only once the node's authenticated application receipt names its head. Upgrade every node to Pallet 36.3.0 or later, speaking @serve.zone/interfaces 32.50.0 or later, before this Cloudly update: a node that sends no application receipt, Pallet 36.2.1 or earlier, or whose session speaks @serve.zone/interfaces below 32.50.0, keeps the projection it holds and receives no successor. The receipt wire contract is unchanged. The changes of Cloudly 33.18.2 and the releases before it that an upgrade from 1.28.1 or earlier also takes are described in the changelog of cloudly@1.28.2, which repeats those of every earlier template; this changelog keeps their upgrade path only, because a catalog changelog is bounded at 65536 characters. From 1.2.1 or earlier, startup runs the forward-only data migration from 0.20.0 to 0.21.0 through the step retire-cluster-relay-acme and then the steps runtime-session-protocol-offer to 0.22.0, deployment-machine-user-grants to 0.23.0, corestore-control-token-state to 0.24.0, runtime-log-attempts to 0.25.0, cluster-node-hostname-intent to 0.26.0, deployment-operation-route-evidence to 0.27.0, service-secret-target-predecessor to 0.28.0, route-claim-absent-owners to 0.29.0 and bounded-runtime-history to 0.30.0; retire-cluster-relay-acme drops the relay certificate and DNS collections of Cloudly's retired Cloudflare and Let's Encrypt integration and logs every relay name whose A record Cloudly had written at Cloudflare: remove each such record at the provider. A cluster relay then requires a dcrouter that advertises dns.clientRecords and certificates.exactIssuance, SERVEZONE_SSLMODE=letsencrypt is refused (this template keeps external), and the typed API changes as the Breaking Changes of the Cloudly 33.0.0 changelog list; review them, stop Cloudly and take a verified backup before upgrading, because the 32.9 image refuses the migrated data and rollback requires a full restore. The upgrade is marked migration-required and backup-before-upgrade because hosts read these flags from the target version alone and every upgrade from 1.26.0 or earlier runs the 0.28.0 to 0.30.0 steps, one from 1.24.0 or earlier, and from 1.25.0 whose first start failed, the 0.25.0 to 0.28.0 steps before them, one from 1.19.0 or earlier the 0.24.0 to 0.25.0 step before that, one from 1.15.0 or earlier the 0.23.0 to 0.24.0 step before that, one from 1.8.0 or earlier the 0.22.0 to 0.23.0 step before that, one from 1.6.0 or earlier the 0.21.0 to 0.22.0 step before that and one from 1.2.1 or earlier the migration from 0.20.0 as well, and breaking and manual-review because every Cloudly before 33.17.3 refuses the 0.30.0 data, the 0.30.0 step deletes every archived deployment record, the 0.23.0 step removes stored grants forward-only, Cloudly 33.18.0 ignores APPSTORE_URL and loads the App Store catalog from its catalog registries and, from 1.7.0 or earlier, a username and password deployer, an administrator's deployment reservation and a machine identity or registry bearer issued earlier are refused. An installation on 1.1.10 or earlier also runs the 1.2.0 steps first, from 0.10.0, and requires completed Secrets v2 cutover retirement. Preserve existing user and storage identities, OIDC bindings, the encrypted keyring and WorkloadInit approval as dedicated root-readable files.",
+              "breaking": true,
+              "requiresManualReview": true,
+              "migrationRequired": true,
+              "backupBeforeUpgrade": true,
+              "requiresFeatures": [
+                "storage.object-storage.backup-exclusions"
+              ],
+              "healthCheck": {
+                "path": "/status",
+                "port": 3000,
+                "expectedStatus": 200
+              }
+            },
+            "configDigest": "sha256:18a8ac5cdff57251ae9733896054c9970e6e5ab3d5582911db5f702133fa7dfb"
           }
         ]
       },
@@ -8882,11 +9071,11 @@
       }
     ]
   },
-  "catalogDigest": "sha256:747ea478ea4c79aacc22e70a7c9a2568101fb78f8f50373dfbd5fee57fb63c16",
+  "catalogDigest": "sha256:d4aa2fa25d500b12293a9423cd614034a6fdc12f9bfebaa420320a8e790251b4",
   "signatures": [
     {
       "keyId": "sha256:58956c3537cdf542430d6685261824776d6128ebb9a93b3d77b53f2d031ba854",
-      "signature": "a2cb33cecda2ff7631961f9013b6f374071471a6d658d2c634ebcac96d1e976ef366e6e715b71bd74d334d6ae052b5cf2a94663ba174d27e1e98d29c0ac9690a"
+      "signature": "d794e925c37832259b698a95cdac3f5289fcf6b7ec97bd514fdd7a0d1d571327b156e3d7fc4164bef2bee586096a781c96bb30b3a348a192a730cafb3cc3cd04"
     }
   ]
 }

package/dist_catalog/catalog.json

@@ -4,9 +4,9 @@
     "major": 1,
     "minor": 0
   },
-  "revision": 33000007000000,
-  "publishedAt": "2026-10-14T01:06:51.000Z",
-  "sourceRevision": "29eab2aff549d4f31431c45f5308c068d48c3fa9",
+  "revision": 33000008000000,
+  "publishedAt": "2026-10-14T08:37:04.000Z",
+  "sourceRevision": "541e7a5e9bf996466d124f85301049499989c6c2",
   "apps": [
     {
       "info": {
@@ -59,7 +59,7 @@
           "Docs": "https://serve.zone"
         }
       },
-      "latestVersion": "1.29.0",
+      "latestVersion": "1.29.1",
       "versions": [
         {
           "version": "1.0.0",
@@ -8108,6 +8108,195 @@
             }
           },
           "configDigest": "sha256:1544df3a1e4e7d3413118e225e234f81061ed9fbc1d824b5827c4a14a31fd3c9"
+        },
+        {
+          "version": "1.29.1",
+          "config": {
+            "image": "code.foss.global/serve.zone/cloudly@sha256:fa7aaa4f68006421aa674ff8afef320919904091469909bbea5ab87bfe500296",
+            "targetPorts": [
+              {
+                "name": "web",
+                "port": 3000,
+                "protocol": "http",
+                "default": true
+              }
+            ],
+            "envVars": [
+              {
+                "key": "SERVEZONE_ENVIRONMENT",
+                "value": "production",
+                "description": "Cloudly runtime environment.",
+                "required": true
+              },
+              {
+                "key": "SERVEZONE_PUBLIC_ORIGIN",
+                "value": "https://${SERVICE_DOMAIN}",
+                "description": "Canonical public Cloudly origin including protocol and optional public port.",
+                "required": true
+              },
+              {
+                "key": "SERVEZONE_LISTENPORT",
+                "value": "3000",
+                "description": "Internal Cloudly HTTP port inside the container.",
+                "required": true
+              },
+              {
+                "key": "SERVEZONE_SSLMODE",
+                "value": "external",
+                "description": "Use external TLS termination through Onebox or dcrouter.",
+                "required": true
+              },
+              {
+                "key": "SERVEZONE_SECRET_KEYRING_PATH",
+                "value": "/run/secrets/cloudly-secrets-v2-keyring.json",
+                "description": "Path to the Cloudly Secrets v2 keyring mounted by Onebox; the keyring value is never placed in the public environment.",
+                "required": true
+              },
+              {
+                "key": "SERVEZONE_SECRET_KEYRING_FILE",
+                "description": "Existing Cloudly Secrets v2 keyring. Onebox stores it encrypted and mounts it only as a root-readable file; retain the existing keyring when upgrading.",
+                "required": true,
+                "secret": true,
+                "delivery": {
+                  "type": "file",
+                  "targetPath": "/run/secrets/cloudly-secrets-v2-keyring.json",
+                  "uid": 0,
+                  "gid": 0,
+                  "mode": 256
+                }
+              },
+              {
+                "key": "SERVEZONE_WORKLOADINIT_APPROVAL_PATH",
+                "value": "/run/secrets/cloudly-workloadinit-approval.json",
+                "description": "Private WorkloadInit release-approval artifact mounted by Onebox for Coreflow secret-runtime registration.",
+                "required": true
+              },
+              {
+                "key": "SERVEZONE_WORKLOADINIT_APPROVAL_FILE",
+                "description": "Operator-approved WorkloadInit release authority. Onebox stores it encrypted and mounts it only as a root-readable file.",
+                "required": true,
+                "secret": true,
+                "delivery": {
+                  "type": "file",
+                  "targetPath": "/run/secrets/cloudly-workloadinit-approval.json",
+                  "uid": 0,
+                  "gid": 0,
+                  "mode": 256
+                }
+              },
+              {
+                "key": "MONGODB_URL",
+                "value": "${MONGODB_URI}",
+                "description": "Authenticated MongoDB wire-protocol URL provisioned by Onebox Corestore.",
+                "required": true
+              },
+              {
+                "key": "MONGODB_NAME",
+                "value": "${MONGODB_DATABASE}",
+                "description": "MongoDB wire-protocol database name provisioned by Onebox Corestore.",
+                "required": true
+              },
+              {
+                "key": "MONGODB_USER",
+                "value": "${MONGODB_USERNAME}",
+                "description": "MongoDB wire-protocol username provisioned by Onebox Corestore.",
+                "required": true
+              },
+              {
+                "key": "MONGODB_PASS",
+                "value": "${MONGODB_PASSWORD}",
+                "description": "MongoDB wire-protocol password provisioned by Onebox Corestore.",
+                "required": true
+              },
+              {
+                "key": "S3_ENDPOINT",
+                "value": "onebox-corestore",
+                "description": "S3 endpoint host provisioned by Onebox Corestore.",
+                "required": true
+              },
+              {
+                "key": "S3_PORT",
+                "value": "9000",
+                "description": "S3 endpoint port provisioned by Onebox Corestore.",
+                "required": true
+              },
+              {
+                "key": "S3_USESSL",
+                "value": "false",
+                "description": "Use plain HTTP for internal Corestore S3 traffic on the Onebox network.",
+                "required": true
+              },
+              {
+                "key": "S3_BUCKET",
+                "value": "${S3_BUCKET}",
+                "description": "Cloudly's own S3 bucket, provisioned by Onebox Corestore: registry storage and kept workload logs.",
+                "required": true
+              },
+              {
+                "key": "S3_ACCESSKEY",
+                "value": "${S3_ACCESS_KEY}",
+                "description": "S3 access key provisioned by Onebox Corestore.",
+                "required": true
+              },
+              {
+                "key": "S3_SECRETKEY",
+                "value": "${S3_SECRET_KEY}",
+                "description": "S3 secret key provisioned by Onebox Corestore.",
+                "required": true
+              }
+            ],
+            "platformRequirements": {
+              "mongodb": true,
+              "s3": true
+            },
+            "backupExclusions": [
+              {
+                "storageRequest": "legacy-s3",
+                "keyPrefixes": [
+                  "runtime-logs/"
+                ],
+                "reason": "Workload logs; a restore brings back state, not past logs."
+              }
+            ],
+            "platformOidc": {
+              "redirectPath": "/auth/oidc/callback",
+              "roles": [
+                {
+                  "id": "admin",
+                  "label": "Administrator",
+                  "description": "Full Cloudly administration for this hosted instance."
+                },
+                {
+                  "id": "user",
+                  "label": "User",
+                  "description": "Standard authenticated Cloudly access for this hosted instance."
+                }
+              ],
+              "environmentVariables": {
+                "issuerUrl": "SERVEZONE_PLATFORM_OIDC_ISSUER",
+                "clientId": "SERVEZONE_PLATFORM_OIDC_CLIENT_ID",
+                "clientSecret": "SERVEZONE_PLATFORM_OIDC_CLIENT_SECRET",
+                "redirectUri": "SERVEZONE_PLATFORM_OIDC_REDIRECT_URI",
+                "audience": "SERVEZONE_PLATFORM_OIDC_AUDIENCE"
+              },
+              "clientAuthenticationMethod": "client_secret_basic"
+            },
+            "minOneboxVersion": "33.8.0",
+            "changelog": "Move Cloudly to 33.20.1, a fixes-only patch over 33.20.0: a Cloudly restart no longer blanks a cluster's ingress routes, a revocation withdraws exactly the routes it revokes, and a Pallet registration Cloudly cannot serve for a reason of its own is refused as momentary. After each cold start of Cloudly 33.20.0 the cluster relay applied two route tables of 0 routes before the full one, and every site of the cluster answered 502 for about 2.5 minutes. Two causes, both fixed. First, Cloudly published an empty revocation as its first table of a cluster, because it held no table of it yet; now the first pull or refresh compiles the plan and issues the full table, and a relay's read that comes before it waits up to 6 s for it and is otherwise answered { table: null }, so the relay keeps the table it holds and the full one follows as a push. Second, every node registers again after a restart, which opens a new session, and a route endpoint required an observation reported under the current session, so every route waited until each node reported anew. A reporter is now one node instance under one credential generation (nodeInstanceNonce, Pallet's per-process nonce, stored on every session record): a node instance that only registered again keeps its observations, its cutover uplink endpoint and its cluster ingress registration authority until it reports anew, while a new Pallet process, a new credential generation or a deleted node supersedes them at once. A route's identity names the node instance and credential generation instead of the session id and epoch, so such a registration changes no route. Route sources that cannot be read right after a start keep the relay's table for at most 180 s, logged once per cluster; then one line names the clusters still waiting and their routes are withdrawn as before. A revocation is now the held table less the revoked routes and port routes, issued and pushed at once: before, any revoked route identity made Cloudly publish an empty table for the whole cluster before it fetched certificates for the next one, so one node registering again or one changed route blanked every site of the cluster. A revoked ingress designation still withdraws everything, and route sources that cannot be read still withdraw every route. A Pallet registration that fails for a reason of Cloudly's own (a database connection lost, a write conflict, a race with a credential being applied, a controller not ready yet or any unnamed failure) is answered controller-unavailable and a spent budget controller-deadline, the @serve.zone/interfaces 32.57.0 runtime session refusal contract, so the node waits and registers again instead of ending its session; before, only a spent budget or a stopping controller carried the refusal payload, and the node read every other failure as a verdict. Only the named verdicts carry no payload: an inactive credential, a deleted, retiring (node-retiring) or foreign node, an unverifiable carrier, a replay that states another registration (registration-replay-mismatch), a moved controller identity, an invalid body. A credential check whose store cannot be read now fails instead of reading as a credential that does not verify, so a registration whose carrier Cloudly could not check is momentary too. A request is answered at its budget even when its work ignores the abort signal, and each registration refusal is logged once per node and reason, value-free, until the node's next accepted registration. Cloudly 33.20.1 adds no data step and keeps the data version 0.30.0, so an upgrade from 1.29.0 runs no data step; it adds no collection, changes neither the Dockerfile nor the qenv.yml requirements, and the runtime contract is unchanged from 1.29.0 and requires Onebox 33.8.0. Like 1.29.0, this template declares Cloudly's workload logs non-backup data: backupExclusions names the key prefix runtime-logs/ of Cloudly's own bucket (storage request legacy-s3), and requiresFeatures names storage.object-storage.backup-exclusions; a host that does not fulfil the feature keeps listing this version and refuses it as an install or upgrade target. From 1.29.0 the upgrade moves Cloudly from 33.20.0 to 33.20.1. From 1.28.6 or earlier it also takes the changes of Cloudly 33.20.0, including the first start that creates the collection cloudly_runtime_log_inline_payloads and the log ledger rows Cloudly 33.19.2 and earlier refuse, so a return to 1.28.6 or an earlier template needs the backup taken before the upgrade; from 1.28.5 or 1.28.3, which run the Cloudly 33.19.1 image, it also takes the changes of Cloudly 33.19.2, and from 1.28.4 or earlier the template also gains the declaration. The rest of this changelog is that of 1.29.0, the Cloudly 33.20.0 template, and describes the changes of Cloudly 33.20.0 and the releases before it that an upgrade from 1.28.6 or earlier also takes: Move Cloudly to 33.20.0, which serves non-public hostnames to dcrouter alone, writes workload logs as segments and lets a service whose retained rollout plan cannot run be redeployed. A node's cluster ingress serves non-public hostnames, restricted to dcrouter: before, Cloudly compiled only PUBLIC domains into the cluster route table, so a STANDARD hostname got a hub route and a private DNS answer while nothing on the node served it. A domain whose sourceProfileRef is not PUBLIC now compiles, on a moved and on a Pallet-born cluster, into a route whose sourceAllowList is the new setting dcrouterForwardSourceAddresses, the addresses dcrouter's forwarded connections arrive from at a node (set in the dashboard's dcrouter panel or with updateSettings; @serve.zone/interfaces 32.55.1 refuses IPv4-mapped IPv6 prefixes there, write the IPv4 prefix). A change of the setting revokes and reissues the route and refreshes every cluster ingress at once, and the ingress-to-workload grant covers every domain's target port. A non-public domain is left out by name while public routes are still served: while dcrouterForwardSourceAddresses is unset, while the cluster's ingress has not registered route-source-allowlist, and while the live cluster relay speaks @serve.zone/interfaces below 32.55.0. To serve a non-public hostname, the cluster's Coreflow and CoreTraffic must run releases on @serve.zone/interfaces 32.55.0 or later, and CoreTraffic must register the capability. Public port mappings stay public-only. Workload log payloads are written as segments, one object per capture every few minutes instead of one per batch: one object per batch grew Cloudly's bucket by about 7,000 objects an hour. A committed batch keeps its payload inline in the database, in the new collection cloudly_runtime_log_inline_payloads, so it reads as soon as it is committed; the drain then writes a capture's inline payloads as one segment, runtime-logs/<serviceId>/<assignmentId>/<captureId>/<first>-<last>.jsonl, and only after the write moves the ledger rows onto it and removes the inline payloads. A segment is due 5 minutes after its capture's oldest inline payload was committed, at 1 MiB, when the capture's final batch is inline, and early, largest capture first, once all inline payloads pass 16 MiB; at 32 MiB the drain commits nothing more until segments are written, so the queue's bounds refuse new reports with the retryable log-intake-saturated. A stop writes no segment early, and the next start writes the inline payloads when due. Rows committed before this version name their own object as before and age out under the 30-day retention. All payload objects stay below runtime-logs/, the prefix this template declares non-backup. The ledger collection cloudly_runtime_log_batches gains no index, so a start builds none over the existing ledger. A spec-owned service whose retained rollout plan the current runtime cannot execute, such as a Swarm-era promotion written before @serve.zone/interfaces 23.0.0 made the sealed containerInvocation required, can be redeployed: its runtime producer refuses the plan image-plan-missing instead of failing its build step on every pass, and while no deployment operation is in flight the reserve preflight answers GO with the auditable RECOVERY_REDEPLOY_OVER_FAILED_RUNTIME warning, which names the plan and the rules it breaks, instead of NO-GO with CURRENT_RUNTIME_DIGEST_MISSING and RUNTIME_ATTESTATION_MISMATCH. An attested runtime, a mutable current reference, a valid plan over an unattested runtime and an operation in flight still block as before. Cloudly 33.20.0 adds no data step and keeps the data version 0.30.0, so an upgrade from 1.28.6 runs no data step; it changes neither the Dockerfile nor the qenv.yml requirements, and the runtime contract is unchanged from 1.28.6 and requires Onebox 33.8.0. Its first start creates the collection cloudly_runtime_log_inline_payloads. On the platform Corestore 32.2.4 (NoSQLDB 10.6.1) a collection creation waits at most 5 s for the open transactions on the Cloudly database, and a transaction an earlier Cloudly left open by exiting before its stop completed (it logs that its shutdown deadline passed) holds that database until its session expires 30 minutes later, so that first start fails with EFENCE_BUSY until then: after an unclean stop of the previous Cloudly, start this version no earlier than 30 minutes after that exit. Every workload log batch this version commits names where its payload is, and Cloudly 33.19.2 and earlier refuse such a ledger row whenever they read it, so a return to 1.28.6 or an earlier template after this version committed a log batch requires the backup taken before the upgrade. Like 1.28.5 and 1.28.6, this template declares Cloudly's workload logs non-backup data: backupExclusions names the key prefix runtime-logs/ of Cloudly's own bucket, the object storage platformRequirements.s3 provisions (storage request legacy-s3), and requiresFeatures names storage.object-storage.backup-exclusions. A host that fulfils that feature leaves the objects under runtime-logs/ out of Cloudly's backups, and a restore brings back Cloudly's state, not its past workload logs: it neither restores nor removes objects under that prefix, and Cloudly reports a log batch whose payload it no longer has as a gap. A host that does not fulfil the feature keeps listing this version and refuses it as an install or upgrade target. From 1.28.6 the upgrade moves Cloudly from 33.19.2 to 33.20.0. From 1.28.5, which runs the Cloudly 33.19.1 image of 1.28.3, it also takes the changes of Cloudly 33.19.2; from 1.28.4 or earlier the template also gains the declaration. The rest of this changelog is that of 1.28.4, the Cloudly 33.19.2 template, and describes the changes of Cloudly 33.19.2 and the releases before it that an upgrade from 1.28.5 or earlier also takes: Move Cloudly to 33.19.2, a fixes-only patch over 33.19.1. A greenfield deployment, a legacy import's successor included, proves its namespace blank on the nodes of the Pallet clusters (phase switching or pallet) and nowhere else: a reservation whose service had no rollout plan yet asked every registered node, so a fleet that still keeps a Coreflow-owned Swarm cluster without a relay could never reserve one, and a legacy import's reserve preflight answered NO-GO with CORESTORE_INVENTORY_UNAVAILABLE and NAMESPACE_NOT_PROVEN_BLANK. The deployment preflight and reserveServiceDeployment read the same nodes; a Pallet node whose inventory cannot be read still blocks, a fleet with no Pallet cluster node refuses the reservation CORESTORE_INVENTORY_UNAVAILABLE, and an existing service's preflight still reads its plan's clusters. A greenfield rollout without a Pallet runtime spec is refused GREENFIELD_PALLET_RUNTIME_SPEC_REQUIRED, which is not retryable, instead of falling back to Coreflow placement onto Swarm nodes the namespace proof never covered: state the spec with setServiceRuntimeSpec, then promote again. A rollout with a Pallet spec plans onto the spec's nodes as before. A node's network compilation proves each workload lease once and each protected authority revision and allocation eligibility proof once per transaction (before, every lease was read up to three times per compilation and each read re-proved the same revision and proof, about 24 proof reads per endpoint), so a compilation with 100 endpoints reads 539 records instead of 2,839 (7.8 s instead of 19.8 s on Cloudly's test fixture), and every compilation signs and refuses exactly as before. Cloudly 33.19.2 adds no data step and no collection and keeps the data version 0.30.0, so an upgrade from 1.28.3 runs no data step; it changes neither the Dockerfile nor the qenv.yml requirements, and the runtime contract is unchanged from 1.28.3 and requires Onebox 33.8.0. From 1.28.2 or earlier, the changes of Cloudly 33.18.3, 33.18.4, 33.18.5, 33.19.0 and 33.19.1 apply too: Runtime log intake no longer holds a node's request while its payload uploads: an admitted reportRuntimeAssignmentLogs costs one append transaction that queues the batch with its payload in the new collection cloudly_runtime_log_queue, and the node is answered once it commits. A background drain writes the payloads, up to 32 at once, and commits the ledger rows and removes the queue rows in coalesced transactions that fence no controller record; every other node request under a request budget counts as critical work, and the drain waits for it at most 1 s per step. A resend of a batch the ledger or the queue already holds is answered replay with nothing written, and while the queue is at its bounds (1024 batches, 64 MiB, 8 concurrent intakes) a report is refused at once with the retryable log-intake-saturated. Object writes list nothing (@lossless.org/client 1.23.3), workload logs are kept for at least 30 days whatever shorter logs.retentionMs a spec states, and one runtime log intake line, logged on its own timer at most once a minute, reports intake latency, queue depth, the drain and refusals. A node that applied its network head is no longer handed a successor that only drops the head's withdrawals, and a successor of the projection a node's receipt names lists only the tombstones its own step retires, so a node with more than 256 retired leases or handoffs stays within the contract's bound. Route verification reads the certificate a route presents on a connection of its own, without pooling or TLS session resumption, so it records the route's subject alternative names; a connection that exposes no certificate fails with TLS_PEER_CERTIFICATE_UNAVAILABLE and a certificate without names with TLS_SUBJECT_ALT_NAMES_MISSING. readServiceDeploymentStatus (@serve.zone/interfaces 32.54.0) reads a deployment operation's status without changing it, and a failed service deletion answers SERVICE_DELETION_FAILED and logs its cause. The first start after 1.28.2 or earlier creates the collection cloudly_runtime_log_queue. On the platform Corestore 32.2.4 (NoSQLDB 10.6.1) a collection creation waits at most 5 s for the open transactions on the Cloudly database, and a transaction an earlier Cloudly left open by exiting before its stop completed (it logs that its shutdown deadline passed) holds that database until its session expires 30 minutes later, so that first start fails with EFENCE_BUSY until then: after an unclean stop of the previous Cloudly, start this version no earlier than 30 minutes after that exit. Cloudly 33.19.1 adds no data step and keeps the data version 0.30.0, so an upgrade from 1.28.2, 1.28.1 or 1.28.0 runs no data step; it changes neither the Dockerfile nor the qenv.yml requirements, and the runtime contract is unchanged from 1.28.2 and requires Onebox 33.8.0. From 1.28.1 or earlier, Cloudly 33.18.2 advances a node's network configuration only once the node's authenticated application receipt names its head. Upgrade every node to Pallet 36.3.0 or later, speaking @serve.zone/interfaces 32.50.0 or later, before this Cloudly update: a node that sends no application receipt, Pallet 36.2.1 or earlier, or whose session speaks @serve.zone/interfaces below 32.50.0, keeps the projection it holds and receives no successor. The receipt wire contract is unchanged. The changes of Cloudly 33.18.2 and the releases before it that an upgrade from 1.28.1 or earlier also takes are described in the changelog of cloudly@1.28.2, which repeats those of every earlier template; this changelog keeps their upgrade path only, because a catalog changelog is bounded at 65536 characters. From 1.2.1 or earlier, startup runs the forward-only data migration from 0.20.0 to 0.21.0 through the step retire-cluster-relay-acme and then the steps runtime-session-protocol-offer to 0.22.0, deployment-machine-user-grants to 0.23.0, corestore-control-token-state to 0.24.0, runtime-log-attempts to 0.25.0, cluster-node-hostname-intent to 0.26.0, deployment-operation-route-evidence to 0.27.0, service-secret-target-predecessor to 0.28.0, route-claim-absent-owners to 0.29.0 and bounded-runtime-history to 0.30.0; retire-cluster-relay-acme drops the relay certificate and DNS collections of Cloudly's retired Cloudflare and Let's Encrypt integration and logs every relay name whose A record Cloudly had written at Cloudflare: remove each such record at the provider. A cluster relay then requires a dcrouter that advertises dns.clientRecords and certificates.exactIssuance, SERVEZONE_SSLMODE=letsencrypt is refused (this template keeps external), and the typed API changes as the Breaking Changes of the Cloudly 33.0.0 changelog list; review them, stop Cloudly and take a verified backup before upgrading, because the 32.9 image refuses the migrated data and rollback requires a full restore. The upgrade is marked migration-required and backup-before-upgrade because hosts read these flags from the target version alone and every upgrade from 1.26.0 or earlier runs the 0.28.0 to 0.30.0 steps, one from 1.24.0 or earlier, and from 1.25.0 whose first start failed, the 0.25.0 to 0.28.0 steps before them, one from 1.19.0 or earlier the 0.24.0 to 0.25.0 step before that, one from 1.15.0 or earlier the 0.23.0 to 0.24.0 step before that, one from 1.8.0 or earlier the 0.22.0 to 0.23.0 step before that, one from 1.6.0 or earlier the 0.21.0 to 0.22.0 step before that and one from 1.2.1 or earlier the migration from 0.20.0 as well, and breaking and manual-review because every Cloudly before 33.17.3 refuses the 0.30.0 data, the 0.30.0 step deletes every archived deployment record, the 0.23.0 step removes stored grants forward-only, Cloudly 33.18.0 ignores APPSTORE_URL and loads the App Store catalog from its catalog registries and, from 1.7.0 or earlier, a username and password deployer, an administrator's deployment reservation and a machine identity or registry bearer issued earlier are refused. An installation on 1.1.10 or earlier also runs the 1.2.0 steps first, from 0.10.0, and requires completed Secrets v2 cutover retirement. Preserve existing user and storage identities, OIDC bindings, the encrypted keyring and WorkloadInit approval as dedicated root-readable files.",
+            "breaking": true,
+            "requiresManualReview": true,
+            "migrationRequired": true,
+            "backupBeforeUpgrade": true,
+            "requiresFeatures": [
+              "storage.object-storage.backup-exclusions"
+            ],
+            "healthCheck": {
+              "path": "/status",
+              "port": 3000,
+              "expectedStatus": 200
+            }
+          },
+          "configDigest": "sha256:18a8ac5cdff57251ae9733896054c9970e6e5ab3d5582911db5f702133fa7dfb"
         }
       ]
     },

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@serve.zone/appstore",
-  "version": "33.7.0",
+  "version": "33.8.0",
   "private": false,
   "description": "serve.zone App Store: the signed runtime catalog and its consumer loader.",
   "type": "module",

package/readme.md

@@ -37,7 +37,7 @@ The catalog lists 19 apps. Every image is pinned to the digest of its multi-plat
 | App ID | Name | App Store category | Latest | Image | Port |
 | --- | --- | --- | --- | --- | --- |
 | `adminer` | Adminer | Dev Tools | `1.0.0` | `adminer@sha256:74f29c416e148b98305e84446a18db7cd2c1038264dec464359d36774ef080ce` | `8080` |
-| `cloudly` | Cloudly | Dev Tools | `1.29.0` | `code.foss.global/serve.zone/cloudly@sha256:0fa3b6a1fcf0d003c2e7397fbc774dbc35815d5c75492b4ec12f2474961f683e` | `3000` |
+| `cloudly` | Cloudly | Dev Tools | `1.29.1` | `code.foss.global/serve.zone/cloudly@sha256:fa7aaa4f68006421aa674ff8afef320919904091469909bbea5ab87bfe500296` | `3000` |
 | `ghost` | Ghost | CMS | `1.0.0` | `ghost@sha256:9482099b1c8700764dc3d38c293e09c16d70f8bacc14af589c47d90b4bebd017` | `2368` |
 | `gitea` | Gitea | Dev Tools | `1.0.0` | `gitea/gitea@sha256:a9dc25679ebdd33356242d2e0667cc220043745e8339922879f187c20329a42e` | `3000` |
 | `grafana` | Grafana | Monitoring | `1.0.0` | `grafana/grafana@sha256:b28bae15e219c998fb0e0424ed724930cc61b1f61fb404d47c862f9a23f9e572` | `3000` |
@@ -68,7 +68,7 @@ Some templates ask Onebox to provision local platform services and inject connec
 
 | App ID | Requirements | Notes |
 | --- | --- | --- |
-| `cloudly` | MongoDB, S3 | Versions `1.29.0`, `1.28.6` and `1.28.5` declare Cloudly's workload logs non-backup data: `backupExclusions` names the key prefix `runtime-logs/` of the `legacy-s3` bucket, and `requiresFeatures` names `storage.object-storage.backup-exclusions`, so a host that does not fulfil it lists them and refuses them as targets. A backup on a host that fulfils it leaves those objects out, and a restore brings back Cloudly's state, not its past workload logs. Version `1.29.0` is `1.28.6` with the Cloudly `33.20.0` image and is the upgrade target of `1.28.6` and `1.28.5`. Cloudly `33.20.0` serves a non-public hostname from a node's cluster ingress to dcrouter alone once the setting `dcrouterForwardSourceAddresses` names dcrouter's forwarding sources (the cluster's Coreflow and CoreTraffic must run releases on `@serve.zone/interfaces` `32.55.0` or later), writes workload log payloads as one segment object per capture every few minutes, keeping a committed batch inline in the new collection `cloudly_runtime_log_inline_payloads` until its segment is written, and lets a service whose retained rollout plan cannot run be redeployed (`RECOVERY_REDEPLOY_OVER_FAILED_RUNTIME`). It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.6` runs none; its first start creates that collection, which after an unclean stop of the previous Cloudly fails with `EFENCE_BUSY` for up to 30 minutes on Corestore `32.2.4`. Cloudly `33.19.2` and earlier refuse the log ledger rows `33.20.0` writes, so a return to an earlier template needs the backup taken before the upgrade. `1.28.6` is `1.28.4` (Cloudly `33.19.2`) with that declaration; `1.28.5` is `1.28.3` (Cloudly `33.19.1`) with it, so a host on `1.28.3` moves its template without moving Cloudly. `1.28.5` sorts above `1.28.4` but runs the older image: from `1.28.4`, upgrade to `1.28.6`. Version `1.28.4` pins Cloudly `33.19.2` and keeps the `1.28.3` runtime contract, including the Onebox `33.8.0` minimum: its platform Corestore `32.2.4` runs NoSQLDB `10.6.1` with the D3 write-conflict fix, and on an earlier engine a node's network plan loses every commit to a `WriteConflict`, so Cloudly `33.15` must not run on it. Cloudly `33.19.2` is a fixes-only patch over `33.19.1`: a greenfield deployment, a legacy import's successor included, proves its namespace blank on the nodes of the Pallet clusters only, so a fleet that still keeps a Coreflow-owned Swarm cluster without a relay can reserve one, while a Pallet node whose inventory cannot be read still blocks; a greenfield rollout without a Pallet runtime spec is refused `GREENFIELD_PALLET_RUNTIME_SPEC_REQUIRED` instead of falling back to Coreflow placement (state the spec with `setServiceRuntimeSpec`, then promote again); and a node's network compilation proves each workload lease and each authority or eligibility proof once per transaction. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.3` runs none. The `1.28.4` changelog carries the changes of `33.19.2` and of `33.18.3` to `33.19.1` and the earlier releases' upgrade path only, because a catalog changelog is bounded at 65536 characters; `1.28.2`'s carries the rest. From `1.28.2` or earlier, Cloudly `33.19.1`'s changes apply too: Cloudly `33.19.1` carries `33.18.3` to `33.19.1`: a node's log report costs one append transaction that queues the batch with its payload in the new collection `cloudly_runtime_log_queue`, and a background drain writes the payloads and commits the ledger rows without holding the node's request or fencing the controller record (a full queue refuses a report with the retryable `log-intake-saturated`), object writes list nothing, workload logs are kept for at least 30 days, route verification reads the certificate each route presents on a connection of its own, and `readServiceDeploymentStatus` (interfaces `32.54.0`) reads a deployment operation without changing it. Its first start creates `cloudly_runtime_log_queue`; on NoSQLDB `10.6.1` that creation waits at most 5 s for the open transactions on the Cloudly database, and a transaction an earlier Cloudly left open by exiting before its stop completed holds the database until its session expires 30 minutes later, so after an unclean stop of the previous Cloudly start the upgrade no earlier than 30 minutes after that exit. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.2`, `1.28.1` or `1.28.0` runs none. From `1.28.1` or earlier, Cloudly `33.18.2`'s fixes apply too: Cloudly `33.18.2` is a fixes-only patch over `33.18.1`: a node's network configuration advances only once the node's application receipt names the projection it holds, so a node no longer receives a new version while it still applies the last one; upgrade every node to Pallet `36.3.0` or later (interfaces `32.50.0` or later) first, because a node that sends no receipt keeps the projection it holds and receives no successor. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.1` or `1.28.0` runs none. From `1.28.0` or earlier, Cloudly `33.18.1`'s fixes apply too: Cloudly `33.18.1` is a fixes-only patch over `33.18.0`: a node's log batches append without its node fences, so they no longer miss their budget against the node's producer passes, and the deployment preflight answers within 25 s, past which it answers the retryable `DEPLOYMENT_PREFLIGHT_UNAVAILABLE`. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.0` runs none. From `1.27.0` or earlier, Cloudly `33.18.0`'s changes apply too: Cloudly `33.18.0` loads the App Store catalog at runtime instead of bundling it, from the registries of its `appStoreCatalogRegistries` setting (empty means the npm registry), so a Cloudly template update needs an appstore release only; it ignores `APPSTORE_URL`, whose remote source is removed, and while no verified catalog is active it refuses App Store install details, upgrade preview and upgrade with `APPSTORE_CATALOG_UNAVAILABLE`. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.27.0` runs none. From `1.26.0` or earlier, Cloudly `33.17.3`'s changes apply too: Cloudly `33.17.3` moves the data version from `0.28.0` to `0.30.0`: every upgrade from `1.26.0` or earlier runs the forward-only data steps `route-claim-absent-owners` to `0.29.0` (the stored nulls of absent owners are removed from every route claim) and `bounded-runtime-history` to `0.30.0` (every archived deployment record is deleted, the index `archiveState_1` is dropped and the accepted Spark observations are pruned to the rows the census reads), and every Cloudly before `33.17.3` refuses the `0.30.0` data. Runtime history is bounded from then on: a Coreflow report deletes the deployment records it no longer names instead of archiving them, `getDeployments` answers the current records whether or not it is asked to `includeArchived`, `deploymentArchiveRetentionDays` is accepted but has no effect, and the dashboard no longer offers archived deployments. A deployment preflight no longer fails `DEPLOYMENT_PREFLIGHT_UNAVAILABLE` on a route claim that stores an absent owner as `null`, and the deployment and adoption preflights log the cause of an unexpected failure. A node's requests no longer contend on the controller fence and share a 6 s budget below the cluster relay's 8 s forward deadline, a request out of budget is refused `controller-deadline`, which the node retries, and a node deletion refuses what a racing write bound to the node. Cloudly consumes the projection application receipts of Pallet `36.3.0`, so a flow a node's projection withdrew is granted again on the same leases once the node reports that it applied the withdrawal; an earlier Pallet keeps every withdrawal carried. Cloudly `33.17.2` starts a database below data version `0.25.0` that holds a runtime slot: the data step `runtime-log-attempts` read through models startup has not prepared and was refused, and now reads and writes through the driver; an install whose first start failed there stayed at data version `0.24.0` and upgrades to `1.28.4` normally. Cloudly `33.17.1` fixes the first start of `33.17.0`, which crash-looped on a database holding Pallet-era nodes: its first data step `cluster-node-hostname-intent` stated each intent through the node and cluster models, which startup has not prepared while it runs the data steps, and was refused before its first write; the step now reads and writes the stored node and cluster documents directly, like the other data steps. The migration ledger does not advance past a failed step, so an install on `1.25.0` whose first start failed stayed unchanged at data version `0.25.0` and upgrades to `1.28.4` normally. Cloudly `33.17.1` keeps the data steps of `33.17.0`, which move the data version from `0.25.0` to `0.28.0`: every upgrade from `1.24.0` or earlier, and from `1.25.0` whose first start failed, runs the forward-only data steps `cluster-node-hostname-intent` to `0.26.0` (every Pallet-era node gets the intent of the hostname it reports, and nothing is renamed), `deployment-operation-route-evidence` to `0.27.0` (a routeless deployment operation that succeeded records its empty route evidence) and `service-secret-target-predecessor` to `0.28.0` (a secret target state an earlier service deletion left behind is removed) before the `0.28.0` to `0.30.0` steps. The template stays marked breaking, migration-required, backup-before-upgrade and manual-review, because hosts read the flags from the target version alone and every upgrade from `1.26.0` or earlier runs the `0.28.0` to `0.30.0` steps, after the `0.25.0` to `0.28.0` steps from `1.24.0` or earlier; every upgrade from `1.19.0` or earlier runs the forward-only data step `runtime-log-attempts` from `0.24.0` to `0.25.0` before them (the log record of every slot's current attempt is written), and every upgrade from `1.15.0` or earlier runs `corestore-control-token-state` from `0.23.0` to `0.24.0` before that (a Corestore control token stored earlier is recorded as revision 1); every Cloudly before `33.17.3` refuses the migrated data: stop Cloudly and take a verified backup first, as rollback is full-restore-only. From `1.24.0` or earlier, Cloudly `33.17.0`'s changes apply too: Cloudly `33.17.0` lets a legacy import reserve: it takes over every route claim its legacy service holds in the import's organization, the cluster ingress keeps serving each hostname for the legacy service until the import's service declares it, and cleanup or rollback before promotion hands the claims back while the legacy service exists; `ROUTE_UNAVAILABLE` and `DEPLOYMENT_ROUTE_COLLISION` name each conflicting hostname and the services holding it. It owns the hostname a node should carry (interfaces `32.52.0`, `cloudly-worker-<name>` by default): `setClusterNodeHostname` renames a Pallet-era node and `resetClusterNodeHostname` sets the default, Spark reads the intent at `POST /spark/nodes/host`, and a Pallet service's gateway routes and inbound mail follow the node its runtime spec pins by id, so a rename moves no traffic. A routeless deployment records its empty route evidence when it succeeds; deleting a cluster's designated ingress service is refused `SERVICE_IS_CLUSTER_INGRESS`, `designateClusterIngress` recovers a designation whose bearer is no longer active, and a deleted service's secret target state goes with its row, so a service recreated under its id writes its secrets again. From `1.23.0` or earlier, Cloudly `33.16.0`'s changes apply too: Cloudly `33.16.0` captures a legacy service's runtime attestation from an administrator's host observation, without a Coreflow session: `captureLegacyServiceRuntimeAttestation` takes `hostObservation`, the legacy Swarm service's spec image and mode and, per replica, its node, Swarm task, container, image store, image id, `RepoDigests`, state and health, read at most 15 minutes before the capture; Cloudly judges it against the one root every replica runs, which it requires whole in its registry and which the legacy tag need not still name, and refuses incomplete evidence without storing anything. The registry prune keeps an attested root until a release's `cloudly-legacy-import` evidence names the attestation or the legacy service is retired. From `1.22.0` or earlier, Cloudly `33.15.3`'s changes apply too: Cloudly `33.15.3` serves a promoted route: a route's identity carried the service's runtime write fence `runtimeAuthorityFenceGeneration`, which every runtime write of the service advances, in a Pallet-born cluster the route join's own workload lease read included, so the route plan read after the cluster ingress's certificate fetch never matched the one before it and every refresh that fetched a certificate withdrew the cluster's whole route table; a route's identity now carries only the versions that change with its authority (the route intent, the runtime policy revision, and per endpoint its assignment, its node's session and its target), while a service being deleted or holding no runtime authority is still refused by the route join, so a revoked route is withdrawn as before, by name. A table withdrawn because its authority changed or its route sources could not be read names that withdrawal as the reason the cluster ingress serves none of its routes, so a failing route verification states it; that reason states facts about its own route only, because one table carries the routes of several organizations: a route whose authority changed names its changed field, every other route only that another route's authority (or the ingress designation) changed during the certificate fetch, and a source-read failure only that the route sources could not be read, its error staying in the operator log. From `1.21.0` or earlier, Cloudly `33.15.2`'s changes apply too: Cloudly `33.15.2` lets a promoted route reach CoreTraffic: a route's identity carries the route intent version `routeIntentGeneration` instead of the whole-document `routeStateGeneration`, so the status writes during the cluster ingress's certificate fetch no longer withdraw the cluster's route table, a plan change during the fetch withdraws it only when it revokes a route the table is about to serve, and the change is named by route and input. A cluster ingress flow that a compensated promotion withdrew and a later promotion wants again no longer stalls the node's network projection: Cloudly holds that flow back with its withdrawal carried and signs the rest of the node's network, while a workload or router grant wanted again while withdrawn withholds the node's projection as `withdrawn-grant-reissued` and any other contract refusal as `projection-rejected`, each naming what it holds; the cluster ingress leaves a route whose grant is held out by name, and a failed route verification states why no route is served for its hostname. A held flow is granted again only once the target or CoreTraffic runs under a new lease. From `1.20.0` or earlier, Cloudly `33.15.1`'s changes apply too: Cloudly `33.15.1` fixes the deployment preflight of an existing service Pallet runs: it read the current runtime from Coreflow alone and answered NO-GO (`CURRENT_RUNTIME_DIGEST_MISSING`, `RUNTIME_ATTESTATION_MISMATCH`) for a service whose rollout status was verified, and now reads the runs of the service's current plan, the evidence its deployment status reads, when Pallet executes that plan on every cluster it targets. It also lets a route promotion survive the status writes of its own gateway sync: the promotion fenced on `routeStateGeneration`, the service's whole-document version, which the route status its gateway sync records, Coreflow's reconciliation reports and its own verified-route status all advance, so every promotion failed `DEPLOYMENT_ROUTE_FENCE_CONFLICT` with no route evidence; it now fences on `routeIntentGeneration`, which only a write that changes the service's domains advances, so a concurrent route change, a change and back included, still fails it fenced, and a promotion interrupted under `33.15.0` is recovered as fenced (retryable), without a rollback. From `1.19.0` or earlier, Cloudly `33.15.0`'s changes apply too: Cloudly `33.15.0` keeps workload logs: a Pallet node's log batches are kept per attempt, their payloads in Cloudly's own bucket (the template's `S3_BUCKET`) under `runtime-logs/`, bounded to 8 MiB and 8192 batches per attempt and the newest 8 attempts per slot, each batch for its config's `logs.retentionMs`, and read with `getServiceRuntimeLogAttempts` and `getRuntimeAssignmentLogs` (interfaces `32.47.0`) or the service page's Logs tab. A Pallet-executed rollout that succeeded stays succeeded when a replica later fails or is replaced; a failing service runtime pass names its step, and a database error its code name and number. A launcher secret written before a service's first image plan no longer leaves the service refused `secrets-unsupported`, because every image plan is written with the service's secret deployment plan in one transaction. The designated cluster ingress runs with `SERVEZONE_CLUSTER_RELAY_ORIGIN` and `SERVEZONE_CLUSTER_NODE_ID`, which Cloudly writes into that workload's environment (a cluster without a valid relay refuses it `cluster-relay-missing`); the template sets neither. From `1.18.0` or earlier, Cloudly `33.14.1` and `33.14.2`'s changes apply too: Cloudly `33.14.2` completes a rollout Pallet executes: Cloudly reports on every rollout whose target clusters are all on Pallet, one reporter per cluster (`pallet/<clusterId>`) over the runs of exactly the current desire, so it moves to `succeeded` (or `ready-for-route`) once its runs are ready at the requested digest and fails when a run fails, and `getServiceDeploymentStatus` reads its runtime tasks from the runtime manager instead of Coreflow, so tsdeploy no longer times out on a Pallet-attached service. A launcher-secret run admitted after a Cloudly restart, reconnect or credential rotation gets its material again instead of waiting `secret-mount-uncertain` (`secret-recipient-inactive`) until its node process restarts, because a node's secret recipient is active for every session of the node process that enrolled it. A failing producer pass is logged under its producer's subject once per change of cause, with how long it ran and a value-free cause naming the failing step and check (such as `network-projection-rejected`), and its recovery once; a failed sweep the timer started is no longer dropped. From `1.17.0` or earlier, Cloudly `33.14.0`'s changes apply too: it gives a platform-owned service's published ports infrastructure-administrator authority: an administrator designates a platform organization's service (an organization id carrying `:`) with `designatePlatformOwnedService`, withdraws it with `undesignatePlatformOwnedService` and reads it with `getServicePlatformOwnership` (interfaces `32.45.0`), after which only a verified administrator reads and writes its published ports, any other organization id is refused `organization-not-platform`, and the designation, stored on the service document, is read inside the published-port request's one admitted operation; every organization service keeps the organization-owner rule. It also names and logs every assignment dispatch outcome under a closed, value-free reason, pushes a refused or unanswered push again on the next pass, and keeps a run its node refused (cause `refused`) instead of replacing it on the failure backoff until its input changes. From `1.16.0` or earlier, Cloudly `33.13.3` and `33.13.4`'s changes apply too: Cloudly `33.13.4` delivers a run's secret material through the WorkloadInit launcher again (Corestore's `MONGODB_URI` included), remembers a successful WorkloadInit approval verification per artifact, and names the check of every refused material request with a closed, value-free `reason`; since Cloudly `33.13.3` a node's session is pushed each assignment revision once instead of on every delivery pass. From `1.15.0` or earlier, Cloudly `33.11.0` to `33.13.2`'s changes apply too: they let an administrator set the Corestore control token without an exec into Cloudly's container (`setCorestoreControlToken`, interfaces `32.42.0`, with its revision and value-free receipts in the new collections `cloudly_corestore_control_token_state` and `cloudly_corestore_control_token_receipts`, and `servezone corestore set-control-token`), keep every managed VPN member across a protected-authority step that only adds, retire a node and its egress owner (`retireClusterNode`, interfaces `32.43.0`), ask a retiring node on interfaces `32.44.0` to withdraw its allocation protection so that a stopped or offline node no longer blocks allocation, and keep a stopping workload's lease in its node's projection until the node's stop receipt is accepted. From `1.14.0` or earlier, Cloudly `33.10.0`'s changes apply too: it lets an operator obtain a cluster relay's bearer without an exec into Cloudly's process: `exportClusterRelayAuthorization` (interfaces `32.41.0`) requires a freshly signed-in administrator, proves the delivered credential version against the hash its generation issued, commits an audit receipt fenced at that generation before it answers the bearer, and never logs it; the receipts live in the new collection `cloudly_cluster_relay_authorization_exports`, installed at startup and listed by `getClusterRelayAuthorizationExports`, and `servezone relay export-authorization` (`@serve.zone/cli` `33.10.0`) writes the bearer as a new `0600` file. From `1.14.0` or earlier, Cloudly `33.9.0`'s changes apply too: a node's managed VPN member is kept across its network projection changes instead of being withdrawn from the hub network on every workload change (`bindGetRuntimeManagedVpnCredentialSuccessor`, interfaces `32.39.0`), the failure cause a node reports with a failed run (interfaces `32.40.0`) is stored and shown in the runtime assignment views and the dashboard's Cause column, and `deleteServiceById` answers success once a service is marked for deletion while another deleting service drains, naming its refusals `SERVICE_LIFECYCLE_BUSY`, `SERVICE_ORGANIZATION_AUTHORITY_REQUIRED` and `SERVICE_MANAGER_STOPPING` instead of an internal error. From `1.13.0` or earlier, Cloudly `33.8.3`'s fix applies too: it lets `deleteServiceById` retire a legacy service whose Corestore resources a successor adopted: its live Corestore bindings are tombstoned with the ordinary config push once every claimed legacy resource of each binding's capability has passed to the successor's live adopting binding, while a service without such a handover is still refused `CORESTORE_CLEANUP_TARGET_AUTHORITY_REQUIRED`. A legacy service whose runtime attestation a legacy-import operation still uses (`reserving`, `reserved`, `provisioning` or `awaiting-image`) is refused `SERVICE_LEGACY_IMPORT_PENDING` (retryable) and keeps running; import and deletion are serialized on the registry repository, so an import that starts after the check refuses as `LEGACY_IMPORT_SOURCE_MISMATCH`, and a deletion that proceeds deletes the attestations no release consumed. From `1.12.0` or earlier, Cloudly `33.8.2`'s fix applies too: Cloudly names why it refused a node's report: a refused `reportRuntimeNetworkProtection`, `reportRuntimeAssignmentTerminalReceipt`, `reportRuntimeAssignmentObservation` or `reportRuntimeAssignmentLogs` is logged as a warning once per node, report and distinct reason, and the node is answered with the unchanged error text plus `{ code: 'runtime-session-report-refused', reason }` as the error's data. From `1.11.0` or earlier, Cloudly `33.8.1`'s fix applies too: deploy before any Pallet that sends `discardedCaptures` or a `not-kept` log loss reaches a fleet node, since it accepts those log batch fields of interfaces `32.38.0`, which Cloudly `33.8.0` refuses on every retry, and still keeps no workload logs. From `1.10.0` or earlier, Cloudly `33.8.0`'s changes apply too: deploy before any Pallet that captures workload output reaches a fleet node, since Cloudly `33.8.0` answers a Pallet's `reportRuntimeAssignmentLogs` (interfaces `32.37.0`) with `not-kept`, which an earlier Cloudly leaves unhandled. Cloudly `33.8.0` lets a legacy `deployOnPush` service without deployable source move to a greenfield successor that runs the exact image it ran (`captureLegacyServiceRuntimeAttestation`, a `legacyImport` reservation under the capability `deployment:import-legacy`, and `importServiceDeploymentLegacyImage` in place of a push), judges promotion against the platforms of the service's rollout target, refuses a registry repository that belongs to another service as `REGISTRY_REPOSITORY_MISMATCH`, promotes and routes a service with a Corestore `database` or `objectstorage` binding again, and reads a node whose last report is older than 180 seconds as `offline`. Its web UI moved to path URLs and a regrouped menu, so bookmarks of the old UI locations open the not-found screen; server paths such as `/status` and the OIDC callback are unchanged. From `1.9.0` or earlier, Cloudly `33.7.0`'s changes apply: Cloudly `33.7.0` lets a fresh greenfield service take over a legacy service's Corestore database and bucket with their data and retention intact: an `adoptCorestoreLegacyResources` selection entry may name `successorServiceId` (interfaces `32.34.0`, carried out by Corestore `33.2.0` or later), the successor must share the owner's organization, be an immutable-deployment service, own no legacy resource of the capability and hold no other binding of it, and each legacy resource is adopted by one service only (`legacy-resource-adopted-elsewhere`, `successor-organization-mismatch`, `successor-not-greenfield`, `successor-owns-legacy-resource`). Since Cloudly `33.6.1`, a deployment reservation whose release tag the registry already holds is refused `DEPLOYMENT_RELEASE_TAG_OCCUPIED` instead of waiting for its image forever, an administrator's manifest push to a release tag a deployment operation reserved is refused `409 release-tag-reserved`, and a binding status report reaches its cluster relay only when it changed what the relay reads. From `1.8.0` or earlier, startup runs the forward-only data step `deployment-machine-user-grants` from `0.22.0` to `0.23.0`, which keeps on every deployment machine user only the grants a capability check reads and permanently removes the rest (such as the `{ serviceId, capabilities }` grants of Cloudly 11); stop Cloudly and take a verified backup first, as the way back is the pre-upgrade backup. Cloudly `33.6.0`'s changes apply too: Cloudly stops gracefully on SIGTERM and SIGINT within 8 seconds, below the 10-second stop grace of Docker and Swarm, so a stop or restart no longer leaves a database transaction held and the next start no longer crash-loops on `EFENCE_BUSY`; a process ended by SIGKILL still holds its database session until the server's session timeout. Every new service gets its own deployment namespace, so two services of one project reserve without `DEPLOYMENT_RESOURCE_COLLISION`, while a deployed service keeps its namespace, and `getDeploymentMachineUsers` lists users configured under Cloudly 11 again. From `1.7.0` or earlier, Cloudly `33.5.0`'s changes apply: Cloudly reserves a deployment only for a deployment machine user: `reserveServiceDeployment` refuses an administrator with `DEPLOYMENT_MACHINE_IDENTITY_REQUIRED`, including a deployer that signs in with a username and password (such as tsdeploy with `TSDEPLOY_CLOUDLY_USERNAME`/`TSDEPLOY_CLOUDLY_PASSWORD`), so before upgrading create a deployment machine user per deployer (`mutateDeploymentMachineUser`, or Security → Access → Deployment machine users in the dashboard), attach its grants with `configureServiceDeploymentMachineUser` and switch the deployer to its token; an existing `api` machine user with its grants already qualifies. A machine identity or registry bearer issued by an earlier release is refused until its holder exchanges its token again with `getIdentityByToken`, and removing a token now ends the sessions opened with it. Let no deployment run during the upgrade: a run in progress keeps its pre-upgrade identity and is refused for the rest of that run (rerun it, or resume it with `deploy-retry`). From `1.3.0` through `1.6.0`, startup runs the forward-only data step `runtime-session-protocol-offer` from `0.21.0` to `0.22.0`, which states `protocol: null` on every Pallet session row recorded before Cloudly stored its node's protocol offer; stop Cloudly and take a verified backup first, as rollback is full-restore-only. From `1.6.0`, Cloudly `33.4.0`'s changes apply: Cloudly `33.4.0` lets `setServicePublishedPorts` publish a contiguous range through `hostPortEnd` (at most 1024 ports in 64 entries per protocol) and same-port outbound through `symmetric: true`, which needs public egress (`published-port-symmetric-without-egress`) and owns its inside ports (`published-port-symmetric-inside-overlap`). New handoff leases translate outbound flows only into `49152-65535`, whose ports are refused `published-port-reserved`; a node keeps its `1024-65535` lease until its router is re-created. A policy write for a service being deleted is refused `service-deleting`. A projection carrying a range or a symmetric entry is withheld as `node-contract-behind` from a Pallet node whose session states a release below interfaces `32.31.0`, or an unrecorded one, so ranges and symmetric entries take effect on a node only once it runs a Pallet release built on interfaces `32.31.0` or later. From `1.5.0`, Cloudly `33.3.0`'s changes apply: Cloudly `33.3.0` lets an administrator set a service's network egress with `setServiceNetworkEgress` (public internet, and the platform endpoints of the protected authority), and refuses a workload whose database or object-storage binding sits on a node whose Corestore endpoint its network does not select as `database-endpoint-unselected` or `objectstorage-endpoint-unselected`; a replica already running keeps running while its target reports the refusal. From `1.4.0`, Cloudly `33.2.0`'s changes apply: Cloudly takes a dcrouter admin token once through `bootstrapExternalGatewayCredential` to provision its own gateway-client credential, starts a first enrollment with the gateway settings, and stages a gateway settings change the held credential cannot carry until the next bootstrap. From `1.2.1` or earlier, startup runs the forward-only migration from `0.20.0` to `0.21.0` before that step, dropping the relay certificate and DNS collections of Cloudly's retired Cloudflare and Let's Encrypt integration and logging every relay name whose A record Cloudly had written at Cloudflare, which the operator removes at the provider. dcrouter holds every cluster relay name's A record and certificate from then on, `SERVEZONE_SSLMODE=letsencrypt` is refused, and the typed API changes as the Cloudly 33.0.0 changelog lists. Stop Cloudly and take a verified backup before upgrading; rollback is full-restore-only. Roll out Cloudly `33.19.2` before Coreflow `33.2.0`; Coreflow `33.1.0` is the minimum. From `1.3.0`, also roll out Cloudly before Coreflow and Corestore `33.0.0`, and set every node's Corestore database endpoint, because a database binding on a node without one is not granted and its service is refused `database-endpoint-missing`, and existing database bindings return no credentials until they are granted again for the endpoint. From `1.1.10` or earlier, startup first runs the `1.2.0` steps from `0.10.0` and needs completed Secrets v2 cutover retirement. Both the existing Secrets v2 keyring and operator-approved WorkloadInit authority are required encrypted inputs delivered as separate root-owned `0400` files; the public environment contains only their fixed paths. |
+| `cloudly` | MongoDB, S3 | Versions `1.29.1`, `1.29.0`, `1.28.6` and `1.28.5` declare Cloudly's workload logs non-backup data: `backupExclusions` names the key prefix `runtime-logs/` of the `legacy-s3` bucket, and `requiresFeatures` names `storage.object-storage.backup-exclusions`, so a host that does not fulfil it lists them and refuses them as targets. A backup on a host that fulfils it leaves those objects out, and a restore brings back Cloudly's state, not its past workload logs. Version `1.29.1` is `1.29.0` with the Cloudly `33.20.1` image and is the upgrade target of `1.29.0`, `1.28.6`, `1.28.5` and `1.28.3`. Cloudly `33.20.1` no longer blanks a cluster's ingress routes when it restarts: its first table of a cluster is the full one, never an empty revocation, a node that only registers again keeps its routes because a route endpoint is a node instance's statement under its credential generation (`nodeInstanceNonce`), unreadable route sources keep the relay's table for at most 180 s after a start, and a revocation withdraws exactly the revoked routes. A Pallet registration Cloudly cannot serve for a reason of its own is refused as momentary (`controller-unavailable`, `controller-deadline`; @serve.zone/interfaces `32.57.0`), so the node registers again instead of ending its session, and a credential check whose store cannot be read fails closed. It adds no data step and no collection and keeps the data version `0.30.0`, so an upgrade from `1.29.0` runs none; an upgrade from `1.28.6` or earlier also takes the changes of Cloudly `33.20.0` below. Version `1.29.0` is `1.28.6` with the Cloudly `33.20.0` image and is the upgrade target of `1.28.6` and `1.28.5`. Cloudly `33.20.0` serves a non-public hostname from a node's cluster ingress to dcrouter alone once the setting `dcrouterForwardSourceAddresses` names dcrouter's forwarding sources (the cluster's Coreflow and CoreTraffic must run releases on `@serve.zone/interfaces` `32.55.0` or later), writes workload log payloads as one segment object per capture every few minutes, keeping a committed batch inline in the new collection `cloudly_runtime_log_inline_payloads` until its segment is written, and lets a service whose retained rollout plan cannot run be redeployed (`RECOVERY_REDEPLOY_OVER_FAILED_RUNTIME`). It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.6` runs none; its first start creates that collection, which after an unclean stop of the previous Cloudly fails with `EFENCE_BUSY` for up to 30 minutes on Corestore `32.2.4`. Cloudly `33.19.2` and earlier refuse the log ledger rows `33.20.0` writes, so a return to an earlier template needs the backup taken before the upgrade. `1.28.6` is `1.28.4` (Cloudly `33.19.2`) with that declaration; `1.28.5` is `1.28.3` (Cloudly `33.19.1`) with it, so a host on `1.28.3` moves its template without moving Cloudly. `1.28.5` sorts above `1.28.4` but runs the older image: from `1.28.4`, upgrade to `1.28.6`. Version `1.28.4` pins Cloudly `33.19.2` and keeps the `1.28.3` runtime contract, including the Onebox `33.8.0` minimum: its platform Corestore `32.2.4` runs NoSQLDB `10.6.1` with the D3 write-conflict fix, and on an earlier engine a node's network plan loses every commit to a `WriteConflict`, so Cloudly `33.15` must not run on it. Cloudly `33.19.2` is a fixes-only patch over `33.19.1`: a greenfield deployment, a legacy import's successor included, proves its namespace blank on the nodes of the Pallet clusters only, so a fleet that still keeps a Coreflow-owned Swarm cluster without a relay can reserve one, while a Pallet node whose inventory cannot be read still blocks; a greenfield rollout without a Pallet runtime spec is refused `GREENFIELD_PALLET_RUNTIME_SPEC_REQUIRED` instead of falling back to Coreflow placement (state the spec with `setServiceRuntimeSpec`, then promote again); and a node's network compilation proves each workload lease and each authority or eligibility proof once per transaction. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.3` runs none. The `1.28.4` changelog carries the changes of `33.19.2` and of `33.18.3` to `33.19.1` and the earlier releases' upgrade path only, because a catalog changelog is bounded at 65536 characters; `1.28.2`'s carries the rest. From `1.28.2` or earlier, Cloudly `33.19.1`'s changes apply too: Cloudly `33.19.1` carries `33.18.3` to `33.19.1`: a node's log report costs one append transaction that queues the batch with its payload in the new collection `cloudly_runtime_log_queue`, and a background drain writes the payloads and commits the ledger rows without holding the node's request or fencing the controller record (a full queue refuses a report with the retryable `log-intake-saturated`), object writes list nothing, workload logs are kept for at least 30 days, route verification reads the certificate each route presents on a connection of its own, and `readServiceDeploymentStatus` (interfaces `32.54.0`) reads a deployment operation without changing it. Its first start creates `cloudly_runtime_log_queue`; on NoSQLDB `10.6.1` that creation waits at most 5 s for the open transactions on the Cloudly database, and a transaction an earlier Cloudly left open by exiting before its stop completed holds the database until its session expires 30 minutes later, so after an unclean stop of the previous Cloudly start the upgrade no earlier than 30 minutes after that exit. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.2`, `1.28.1` or `1.28.0` runs none. From `1.28.1` or earlier, Cloudly `33.18.2`'s fixes apply too: Cloudly `33.18.2` is a fixes-only patch over `33.18.1`: a node's network configuration advances only once the node's application receipt names the projection it holds, so a node no longer receives a new version while it still applies the last one; upgrade every node to Pallet `36.3.0` or later (interfaces `32.50.0` or later) first, because a node that sends no receipt keeps the projection it holds and receives no successor. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.1` or `1.28.0` runs none. From `1.28.0` or earlier, Cloudly `33.18.1`'s fixes apply too: Cloudly `33.18.1` is a fixes-only patch over `33.18.0`: a node's log batches append without its node fences, so they no longer miss their budget against the node's producer passes, and the deployment preflight answers within 25 s, past which it answers the retryable `DEPLOYMENT_PREFLIGHT_UNAVAILABLE`. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.28.0` runs none. From `1.27.0` or earlier, Cloudly `33.18.0`'s changes apply too: Cloudly `33.18.0` loads the App Store catalog at runtime instead of bundling it, from the registries of its `appStoreCatalogRegistries` setting (empty means the npm registry), so a Cloudly template update needs an appstore release only; it ignores `APPSTORE_URL`, whose remote source is removed, and while no verified catalog is active it refuses App Store install details, upgrade preview and upgrade with `APPSTORE_CATALOG_UNAVAILABLE`. It adds no data step and keeps the data version `0.30.0`, so an upgrade from `1.27.0` runs none. From `1.26.0` or earlier, Cloudly `33.17.3`'s changes apply too: Cloudly `33.17.3` moves the data version from `0.28.0` to `0.30.0`: every upgrade from `1.26.0` or earlier runs the forward-only data steps `route-claim-absent-owners` to `0.29.0` (the stored nulls of absent owners are removed from every route claim) and `bounded-runtime-history` to `0.30.0` (every archived deployment record is deleted, the index `archiveState_1` is dropped and the accepted Spark observations are pruned to the rows the census reads), and every Cloudly before `33.17.3` refuses the `0.30.0` data. Runtime history is bounded from then on: a Coreflow report deletes the deployment records it no longer names instead of archiving them, `getDeployments` answers the current records whether or not it is asked to `includeArchived`, `deploymentArchiveRetentionDays` is accepted but has no effect, and the dashboard no longer offers archived deployments. A deployment preflight no longer fails `DEPLOYMENT_PREFLIGHT_UNAVAILABLE` on a route claim that stores an absent owner as `null`, and the deployment and adoption preflights log the cause of an unexpected failure. A node's requests no longer contend on the controller fence and share a 6 s budget below the cluster relay's 8 s forward deadline, a request out of budget is refused `controller-deadline`, which the node retries, and a node deletion refuses what a racing write bound to the node. Cloudly consumes the projection application receipts of Pallet `36.3.0`, so a flow a node's projection withdrew is granted again on the same leases once the node reports that it applied the withdrawal; an earlier Pallet keeps every withdrawal carried. Cloudly `33.17.2` starts a database below data version `0.25.0` that holds a runtime slot: the data step `runtime-log-attempts` read through models startup has not prepared and was refused, and now reads and writes through the driver; an install whose first start failed there stayed at data version `0.24.0` and upgrades to `1.28.4` normally. Cloudly `33.17.1` fixes the first start of `33.17.0`, which crash-looped on a database holding Pallet-era nodes: its first data step `cluster-node-hostname-intent` stated each intent through the node and cluster models, which startup has not prepared while it runs the data steps, and was refused before its first write; the step now reads and writes the stored node and cluster documents directly, like the other data steps. The migration ledger does not advance past a failed step, so an install on `1.25.0` whose first start failed stayed unchanged at data version `0.25.0` and upgrades to `1.28.4` normally. Cloudly `33.17.1` keeps the data steps of `33.17.0`, which move the data version from `0.25.0` to `0.28.0`: every upgrade from `1.24.0` or earlier, and from `1.25.0` whose first start failed, runs the forward-only data steps `cluster-node-hostname-intent` to `0.26.0` (every Pallet-era node gets the intent of the hostname it reports, and nothing is renamed), `deployment-operation-route-evidence` to `0.27.0` (a routeless deployment operation that succeeded records its empty route evidence) and `service-secret-target-predecessor` to `0.28.0` (a secret target state an earlier service deletion left behind is removed) before the `0.28.0` to `0.30.0` steps. The template stays marked breaking, migration-required, backup-before-upgrade and manual-review, because hosts read the flags from the target version alone and every upgrade from `1.26.0` or earlier runs the `0.28.0` to `0.30.0` steps, after the `0.25.0` to `0.28.0` steps from `1.24.0` or earlier; every upgrade from `1.19.0` or earlier runs the forward-only data step `runtime-log-attempts` from `0.24.0` to `0.25.0` before them (the log record of every slot's current attempt is written), and every upgrade from `1.15.0` or earlier runs `corestore-control-token-state` from `0.23.0` to `0.24.0` before that (a Corestore control token stored earlier is recorded as revision 1); every Cloudly before `33.17.3` refuses the migrated data: stop Cloudly and take a verified backup first, as rollback is full-restore-only. From `1.24.0` or earlier, Cloudly `33.17.0`'s changes apply too: Cloudly `33.17.0` lets a legacy import reserve: it takes over every route claim its legacy service holds in the import's organization, the cluster ingress keeps serving each hostname for the legacy service until the import's service declares it, and cleanup or rollback before promotion hands the claims back while the legacy service exists; `ROUTE_UNAVAILABLE` and `DEPLOYMENT_ROUTE_COLLISION` name each conflicting hostname and the services holding it. It owns the hostname a node should carry (interfaces `32.52.0`, `cloudly-worker-<name>` by default): `setClusterNodeHostname` renames a Pallet-era node and `resetClusterNodeHostname` sets the default, Spark reads the intent at `POST /spark/nodes/host`, and a Pallet service's gateway routes and inbound mail follow the node its runtime spec pins by id, so a rename moves no traffic. A routeless deployment records its empty route evidence when it succeeds; deleting a cluster's designated ingress service is refused `SERVICE_IS_CLUSTER_INGRESS`, `designateClusterIngress` recovers a designation whose bearer is no longer active, and a deleted service's secret target state goes with its row, so a service recreated under its id writes its secrets again. From `1.23.0` or earlier, Cloudly `33.16.0`'s changes apply too: Cloudly `33.16.0` captures a legacy service's runtime attestation from an administrator's host observation, without a Coreflow session: `captureLegacyServiceRuntimeAttestation` takes `hostObservation`, the legacy Swarm service's spec image and mode and, per replica, its node, Swarm task, container, image store, image id, `RepoDigests`, state and health, read at most 15 minutes before the capture; Cloudly judges it against the one root every replica runs, which it requires whole in its registry and which the legacy tag need not still name, and refuses incomplete evidence without storing anything. The registry prune keeps an attested root until a release's `cloudly-legacy-import` evidence names the attestation or the legacy service is retired. From `1.22.0` or earlier, Cloudly `33.15.3`'s changes apply too: Cloudly `33.15.3` serves a promoted route: a route's identity carried the service's runtime write fence `runtimeAuthorityFenceGeneration`, which every runtime write of the service advances, in a Pallet-born cluster the route join's own workload lease read included, so the route plan read after the cluster ingress's certificate fetch never matched the one before it and every refresh that fetched a certificate withdrew the cluster's whole route table; a route's identity now carries only the versions that change with its authority (the route intent, the runtime policy revision, and per endpoint its assignment, its node's session and its target), while a service being deleted or holding no runtime authority is still refused by the route join, so a revoked route is withdrawn as before, by name. A table withdrawn because its authority changed or its route sources could not be read names that withdrawal as the reason the cluster ingress serves none of its routes, so a failing route verification states it; that reason states facts about its own route only, because one table carries the routes of several organizations: a route whose authority changed names its changed field, every other route only that another route's authority (or the ingress designation) changed during the certificate fetch, and a source-read failure only that the route sources could not be read, its error staying in the operator log. From `1.21.0` or earlier, Cloudly `33.15.2`'s changes apply too: Cloudly `33.15.2` lets a promoted route reach CoreTraffic: a route's identity carries the route intent version `routeIntentGeneration` instead of the whole-document `routeStateGeneration`, so the status writes during the cluster ingress's certificate fetch no longer withdraw the cluster's route table, a plan change during the fetch withdraws it only when it revokes a route the table is about to serve, and the change is named by route and input. A cluster ingress flow that a compensated promotion withdrew and a later promotion wants again no longer stalls the node's network projection: Cloudly holds that flow back with its withdrawal carried and signs the rest of the node's network, while a workload or router grant wanted again while withdrawn withholds the node's projection as `withdrawn-grant-reissued` and any other contract refusal as `projection-rejected`, each naming what it holds; the cluster ingress leaves a route whose grant is held out by name, and a failed route verification states why no route is served for its hostname. A held flow is granted again only once the target or CoreTraffic runs under a new lease. From `1.20.0` or earlier, Cloudly `33.15.1`'s changes apply too: Cloudly `33.15.1` fixes the deployment preflight of an existing service Pallet runs: it read the current runtime from Coreflow alone and answered NO-GO (`CURRENT_RUNTIME_DIGEST_MISSING`, `RUNTIME_ATTESTATION_MISMATCH`) for a service whose rollout status was verified, and now reads the runs of the service's current plan, the evidence its deployment status reads, when Pallet executes that plan on every cluster it targets. It also lets a route promotion survive the status writes of its own gateway sync: the promotion fenced on `routeStateGeneration`, the service's whole-document version, which the route status its gateway sync records, Coreflow's reconciliation reports and its own verified-route status all advance, so every promotion failed `DEPLOYMENT_ROUTE_FENCE_CONFLICT` with no route evidence; it now fences on `routeIntentGeneration`, which only a write that changes the service's domains advances, so a concurrent route change, a change and back included, still fails it fenced, and a promotion interrupted under `33.15.0` is recovered as fenced (retryable), without a rollback. From `1.19.0` or earlier, Cloudly `33.15.0`'s changes apply too: Cloudly `33.15.0` keeps workload logs: a Pallet node's log batches are kept per attempt, their payloads in Cloudly's own bucket (the template's `S3_BUCKET`) under `runtime-logs/`, bounded to 8 MiB and 8192 batches per attempt and the newest 8 attempts per slot, each batch for its config's `logs.retentionMs`, and read with `getServiceRuntimeLogAttempts` and `getRuntimeAssignmentLogs` (interfaces `32.47.0`) or the service page's Logs tab. A Pallet-executed rollout that succeeded stays succeeded when a replica later fails or is replaced; a failing service runtime pass names its step, and a database error its code name and number. A launcher secret written before a service's first image plan no longer leaves the service refused `secrets-unsupported`, because every image plan is written with the service's secret deployment plan in one transaction. The designated cluster ingress runs with `SERVEZONE_CLUSTER_RELAY_ORIGIN` and `SERVEZONE_CLUSTER_NODE_ID`, which Cloudly writes into that workload's environment (a cluster without a valid relay refuses it `cluster-relay-missing`); the template sets neither. From `1.18.0` or earlier, Cloudly `33.14.1` and `33.14.2`'s changes apply too: Cloudly `33.14.2` completes a rollout Pallet executes: Cloudly reports on every rollout whose target clusters are all on Pallet, one reporter per cluster (`pallet/<clusterId>`) over the runs of exactly the current desire, so it moves to `succeeded` (or `ready-for-route`) once its runs are ready at the requested digest and fails when a run fails, and `getServiceDeploymentStatus` reads its runtime tasks from the runtime manager instead of Coreflow, so tsdeploy no longer times out on a Pallet-attached service. A launcher-secret run admitted after a Cloudly restart, reconnect or credential rotation gets its material again instead of waiting `secret-mount-uncertain` (`secret-recipient-inactive`) until its node process restarts, because a node's secret recipient is active for every session of the node process that enrolled it. A failing producer pass is logged under its producer's subject once per change of cause, with how long it ran and a value-free cause naming the failing step and check (such as `network-projection-rejected`), and its recovery once; a failed sweep the timer started is no longer dropped. From `1.17.0` or earlier, Cloudly `33.14.0`'s changes apply too: it gives a platform-owned service's published ports infrastructure-administrator authority: an administrator designates a platform organization's service (an organization id carrying `:`) with `designatePlatformOwnedService`, withdraws it with `undesignatePlatformOwnedService` and reads it with `getServicePlatformOwnership` (interfaces `32.45.0`), after which only a verified administrator reads and writes its published ports, any other organization id is refused `organization-not-platform`, and the designation, stored on the service document, is read inside the published-port request's one admitted operation; every organization service keeps the organization-owner rule. It also names and logs every assignment dispatch outcome under a closed, value-free reason, pushes a refused or unanswered push again on the next pass, and keeps a run its node refused (cause `refused`) instead of replacing it on the failure backoff until its input changes. From `1.16.0` or earlier, Cloudly `33.13.3` and `33.13.4`'s changes apply too: Cloudly `33.13.4` delivers a run's secret material through the WorkloadInit launcher again (Corestore's `MONGODB_URI` included), remembers a successful WorkloadInit approval verification per artifact, and names the check of every refused material request with a closed, value-free `reason`; since Cloudly `33.13.3` a node's session is pushed each assignment revision once instead of on every delivery pass. From `1.15.0` or earlier, Cloudly `33.11.0` to `33.13.2`'s changes apply too: they let an administrator set the Corestore control token without an exec into Cloudly's container (`setCorestoreControlToken`, interfaces `32.42.0`, with its revision and value-free receipts in the new collections `cloudly_corestore_control_token_state` and `cloudly_corestore_control_token_receipts`, and `servezone corestore set-control-token`), keep every managed VPN member across a protected-authority step that only adds, retire a node and its egress owner (`retireClusterNode`, interfaces `32.43.0`), ask a retiring node on interfaces `32.44.0` to withdraw its allocation protection so that a stopped or offline node no longer blocks allocation, and keep a stopping workload's lease in its node's projection until the node's stop receipt is accepted. From `1.14.0` or earlier, Cloudly `33.10.0`'s changes apply too: it lets an operator obtain a cluster relay's bearer without an exec into Cloudly's process: `exportClusterRelayAuthorization` (interfaces `32.41.0`) requires a freshly signed-in administrator, proves the delivered credential version against the hash its generation issued, commits an audit receipt fenced at that generation before it answers the bearer, and never logs it; the receipts live in the new collection `cloudly_cluster_relay_authorization_exports`, installed at startup and listed by `getClusterRelayAuthorizationExports`, and `servezone relay export-authorization` (`@serve.zone/cli` `33.10.0`) writes the bearer as a new `0600` file. From `1.14.0` or earlier, Cloudly `33.9.0`'s changes apply too: a node's managed VPN member is kept across its network projection changes instead of being withdrawn from the hub network on every workload change (`bindGetRuntimeManagedVpnCredentialSuccessor`, interfaces `32.39.0`), the failure cause a node reports with a failed run (interfaces `32.40.0`) is stored and shown in the runtime assignment views and the dashboard's Cause column, and `deleteServiceById` answers success once a service is marked for deletion while another deleting service drains, naming its refusals `SERVICE_LIFECYCLE_BUSY`, `SERVICE_ORGANIZATION_AUTHORITY_REQUIRED` and `SERVICE_MANAGER_STOPPING` instead of an internal error. From `1.13.0` or earlier, Cloudly `33.8.3`'s fix applies too: it lets `deleteServiceById` retire a legacy service whose Corestore resources a successor adopted: its live Corestore bindings are tombstoned with the ordinary config push once every claimed legacy resource of each binding's capability has passed to the successor's live adopting binding, while a service without such a handover is still refused `CORESTORE_CLEANUP_TARGET_AUTHORITY_REQUIRED`. A legacy service whose runtime attestation a legacy-import operation still uses (`reserving`, `reserved`, `provisioning` or `awaiting-image`) is refused `SERVICE_LEGACY_IMPORT_PENDING` (retryable) and keeps running; import and deletion are serialized on the registry repository, so an import that starts after the check refuses as `LEGACY_IMPORT_SOURCE_MISMATCH`, and a deletion that proceeds deletes the attestations no release consumed. From `1.12.0` or earlier, Cloudly `33.8.2`'s fix applies too: Cloudly names why it refused a node's report: a refused `reportRuntimeNetworkProtection`, `reportRuntimeAssignmentTerminalReceipt`, `reportRuntimeAssignmentObservation` or `reportRuntimeAssignmentLogs` is logged as a warning once per node, report and distinct reason, and the node is answered with the unchanged error text plus `{ code: 'runtime-session-report-refused', reason }` as the error's data. From `1.11.0` or earlier, Cloudly `33.8.1`'s fix applies too: deploy before any Pallet that sends `discardedCaptures` or a `not-kept` log loss reaches a fleet node, since it accepts those log batch fields of interfaces `32.38.0`, which Cloudly `33.8.0` refuses on every retry, and still keeps no workload logs. From `1.10.0` or earlier, Cloudly `33.8.0`'s changes apply too: deploy before any Pallet that captures workload output reaches a fleet node, since Cloudly `33.8.0` answers a Pallet's `reportRuntimeAssignmentLogs` (interfaces `32.37.0`) with `not-kept`, which an earlier Cloudly leaves unhandled. Cloudly `33.8.0` lets a legacy `deployOnPush` service without deployable source move to a greenfield successor that runs the exact image it ran (`captureLegacyServiceRuntimeAttestation`, a `legacyImport` reservation under the capability `deployment:import-legacy`, and `importServiceDeploymentLegacyImage` in place of a push), judges promotion against the platforms of the service's rollout target, refuses a registry repository that belongs to another service as `REGISTRY_REPOSITORY_MISMATCH`, promotes and routes a service with a Corestore `database` or `objectstorage` binding again, and reads a node whose last report is older than 180 seconds as `offline`. Its web UI moved to path URLs and a regrouped menu, so bookmarks of the old UI locations open the not-found screen; server paths such as `/status` and the OIDC callback are unchanged. From `1.9.0` or earlier, Cloudly `33.7.0`'s changes apply: Cloudly `33.7.0` lets a fresh greenfield service take over a legacy service's Corestore database and bucket with their data and retention intact: an `adoptCorestoreLegacyResources` selection entry may name `successorServiceId` (interfaces `32.34.0`, carried out by Corestore `33.2.0` or later), the successor must share the owner's organization, be an immutable-deployment service, own no legacy resource of the capability and hold no other binding of it, and each legacy resource is adopted by one service only (`legacy-resource-adopted-elsewhere`, `successor-organization-mismatch`, `successor-not-greenfield`, `successor-owns-legacy-resource`). Since Cloudly `33.6.1`, a deployment reservation whose release tag the registry already holds is refused `DEPLOYMENT_RELEASE_TAG_OCCUPIED` instead of waiting for its image forever, an administrator's manifest push to a release tag a deployment operation reserved is refused `409 release-tag-reserved`, and a binding status report reaches its cluster relay only when it changed what the relay reads. From `1.8.0` or earlier, startup runs the forward-only data step `deployment-machine-user-grants` from `0.22.0` to `0.23.0`, which keeps on every deployment machine user only the grants a capability check reads and permanently removes the rest (such as the `{ serviceId, capabilities }` grants of Cloudly 11); stop Cloudly and take a verified backup first, as the way back is the pre-upgrade backup. Cloudly `33.6.0`'s changes apply too: Cloudly stops gracefully on SIGTERM and SIGINT within 8 seconds, below the 10-second stop grace of Docker and Swarm, so a stop or restart no longer leaves a database transaction held and the next start no longer crash-loops on `EFENCE_BUSY`; a process ended by SIGKILL still holds its database session until the server's session timeout. Every new service gets its own deployment namespace, so two services of one project reserve without `DEPLOYMENT_RESOURCE_COLLISION`, while a deployed service keeps its namespace, and `getDeploymentMachineUsers` lists users configured under Cloudly 11 again. From `1.7.0` or earlier, Cloudly `33.5.0`'s changes apply: Cloudly reserves a deployment only for a deployment machine user: `reserveServiceDeployment` refuses an administrator with `DEPLOYMENT_MACHINE_IDENTITY_REQUIRED`, including a deployer that signs in with a username and password (such as tsdeploy with `TSDEPLOY_CLOUDLY_USERNAME`/`TSDEPLOY_CLOUDLY_PASSWORD`), so before upgrading create a deployment machine user per deployer (`mutateDeploymentMachineUser`, or Security → Access → Deployment machine users in the dashboard), attach its grants with `configureServiceDeploymentMachineUser` and switch the deployer to its token; an existing `api` machine user with its grants already qualifies. A machine identity or registry bearer issued by an earlier release is refused until its holder exchanges its token again with `getIdentityByToken`, and removing a token now ends the sessions opened with it. Let no deployment run during the upgrade: a run in progress keeps its pre-upgrade identity and is refused for the rest of that run (rerun it, or resume it with `deploy-retry`). From `1.3.0` through `1.6.0`, startup runs the forward-only data step `runtime-session-protocol-offer` from `0.21.0` to `0.22.0`, which states `protocol: null` on every Pallet session row recorded before Cloudly stored its node's protocol offer; stop Cloudly and take a verified backup first, as rollback is full-restore-only. From `1.6.0`, Cloudly `33.4.0`'s changes apply: Cloudly `33.4.0` lets `setServicePublishedPorts` publish a contiguous range through `hostPortEnd` (at most 1024 ports in 64 entries per protocol) and same-port outbound through `symmetric: true`, which needs public egress (`published-port-symmetric-without-egress`) and owns its inside ports (`published-port-symmetric-inside-overlap`). New handoff leases translate outbound flows only into `49152-65535`, whose ports are refused `published-port-reserved`; a node keeps its `1024-65535` lease until its router is re-created. A policy write for a service being deleted is refused `service-deleting`. A projection carrying a range or a symmetric entry is withheld as `node-contract-behind` from a Pallet node whose session states a release below interfaces `32.31.0`, or an unrecorded one, so ranges and symmetric entries take effect on a node only once it runs a Pallet release built on interfaces `32.31.0` or later. From `1.5.0`, Cloudly `33.3.0`'s changes apply: Cloudly `33.3.0` lets an administrator set a service's network egress with `setServiceNetworkEgress` (public internet, and the platform endpoints of the protected authority), and refuses a workload whose database or object-storage binding sits on a node whose Corestore endpoint its network does not select as `database-endpoint-unselected` or `objectstorage-endpoint-unselected`; a replica already running keeps running while its target reports the refusal. From `1.4.0`, Cloudly `33.2.0`'s changes apply: Cloudly takes a dcrouter admin token once through `bootstrapExternalGatewayCredential` to provision its own gateway-client credential, starts a first enrollment with the gateway settings, and stages a gateway settings change the held credential cannot carry until the next bootstrap. From `1.2.1` or earlier, startup runs the forward-only migration from `0.20.0` to `0.21.0` before that step, dropping the relay certificate and DNS collections of Cloudly's retired Cloudflare and Let's Encrypt integration and logging every relay name whose A record Cloudly had written at Cloudflare, which the operator removes at the provider. dcrouter holds every cluster relay name's A record and certificate from then on, `SERVEZONE_SSLMODE=letsencrypt` is refused, and the typed API changes as the Cloudly 33.0.0 changelog lists. Stop Cloudly and take a verified backup before upgrading; rollback is full-restore-only. Roll out Cloudly `33.19.2` before Coreflow `33.2.0`; Coreflow `33.1.0` is the minimum. From `1.3.0`, also roll out Cloudly before Coreflow and Corestore `33.0.0`, and set every node's Corestore database endpoint, because a database binding on a node without one is not granted and its service is refused `database-endpoint-missing`, and existing database bindings return no credentials until they are granted again for the endpoint. From `1.1.10` or earlier, startup first runs the `1.2.0` steps from `0.10.0` and needs completed Secrets v2 cutover retirement. Both the existing Secrets v2 keyring and operator-approved WorkloadInit authority are required encrypted inputs delivered as separate root-owned `0400` files; the public environment contains only their fixed paths. |
 | `ghost` | MariaDB | Uses MySQL-style `database__connection__*` environment variables. |
 | `gitea` | MariaDB | Uses Gitea database environment variables with `${MARIADB_*}` placeholders. |
 | `nextcloud` | MariaDB, Valkey | Uses MySQL-compatible database variables. Valkey is declared as a platform requirement. |