@toa.io/operations 1.0.0-alpha.34 → 1.0.0-alpha.340

package/readme.md

@@ -1,9 +1,188 @@
 # Toa Operations
 
+## Installing
+
+`toa deploy`, `toa build`, `toa push`, `toa env`, `toa export` and `toa conceal` need this package
+beside the CLI. The runtime does not depend on it, so an application that deploys lists it with the
+runtime:
+
+```shell
+$ npm i -D @toa.io/runtime @toa.io/operations
+```
+
+A machine that only deploys needs no runtime and no extension:
+
+```shell
+$ npm i @toa.io/cli @toa.io/operations
+$ npx toa deploy production -p application
+```
+
+What the extensions declare is read from `@toa.io/definitions`, which the CLI brings; what they run
+is not needed to render a chart. `registry.services: build` is the exception: it builds an
+extension's service image from the installed package, so it needs the runtime installed beside
+this one. `published` does not.
+
+## Compositions
+
+A composition is deployed as one pod. Beside its components it may run extension services,
+which are otherwise each deployed on their own:
+
+```yaml
+# context.toa.yaml
+
+compositions:
+  - name: edge
+    components:
+      - todos.tasks
+    services:
+      - exposition
+```
+
+See [compositions](../documentation/compositions.md).
+
+## Evicted
+
+What Toa does not deploy for a context, whatever a composition lists. It stays part of the
+context, and is deployed by other means:
+
+```yaml
+# context.toa.yaml
+
+evicted:
+  components:
+    - todos.stats
+  services:
+    - exposition
+```
+
+Nothing an evicted component alone required is deployed either — its storage, the extensions it
+declared. `evicted@production` states it for one environment.
+
+See [compositions](../documentation/compositions.md#evicted).
+
+## Secrets
+
+A value that must not be in the chart is rendered as a reference to a Kubernetes secret, and the
+secret is the application's to deploy. A deploy reads the namespace it deploys into before it builds
+anything, and refuses where a key a workload would read is not there:
+
+```
+$ toa deploy production
+Secrets are not deployed: toa-mongodb.default/username, toa-mongodb.default/password
+```
+
+Required is what `toa export secrets` lists, less the keys it marks `(optional)`, plus the
+`registry.credentials` secret where a context names one. `toa deploy --dry` reads no cluster, and
+neither does anything else that reads a context: `toa env`, `toa export`, `toa build` and `toa push`.
+
+See [variables and secrets](../documentation/deployment.md#variables-and-secrets).
+
 ## Context
 
 ### Container Registry
 
+Deploy images default to `FROM ghcr.io/toa-io/runtime:<runtime.version>`
+(published with each Toa release). The base image already has `@toa.io/runtime`
+installed; composition/service Dockerfiles only install component dependencies.
+
+To use a custom base image, set `registry.build.image` (or `composition.image`).
+That image must provide the `toa` CLI, or install it via `registry.build.run`. The binary is
+`@toa.io/cli`'s, which the runtime depends on, so a local install puts it on the `PATH`:
+
+```yaml
+# context.toa.yaml
+
+registry:
+  build:
+    image: node:24.22.1-alpine3.24
+    run: |
+      npm i --prefix /toa @toa.io/runtime --omit=dev
+      ln -s /toa/node_modules/.bin/toa /usr/local/bin/toa
+```
+
+`/toa` is where it goes: what an extension needs for what a component declares is installed
+there, beside the extension that reads the declaration.
+
+#### Composition Images
+
+A composition is two images in one repository. `composition-<name>:deps-<hash>` is what its
+components depend on, installed on the base image: it is tagged by everything the install
+reads — the runtime version, the base image, the build options, each component's
+`package.json` (and `package-lock.json`, where there is one), and what the extensions install
+for what the components declare — and is built only when one of those changes. `composition-<name>:<hash>` is the sources laid over it in a single linked
+layer, so a deploy that changes code alone builds and pushes that layer, and neither
+downloads nor uploads the dependencies again. The same holds for `mono`.
+
+The sources are copied as files, so a component that links a directory in — sources it shares with
+another component — carries what the link points to, and a link the build cannot read fails the
+build. What is linked is source alone: dependencies are installed per component, from its own
+`package.json`.
+
+Beside each component's sources the layer carries `manifest.toa.json`, the manifest as this
+build normalised it. A composition reads it instead of normalising again, so it loads no
+bridge to read what a module declares; a workspace has no such file and is read as it always
+was. A component an extension ships has none either — its package is published long before any
+image is built — and is read from the digest `@toa.io/definitions` carries, which a deploy has
+always read. So nothing that runs reads a module's source, and `oxc-parser` is installed only
+where a manifest is normalised: a workspace and a deploy, both through `@toa.io/operations`. What a build cannot express there — a prototype in a directory of the application's own
+rather than in a package — fails the build, where before it failed the container.
+
+A pushed image builds on the `toa` container builder, whose registry exporter is what lays
+an image over a base it never pulled; `toa build` loads on the daemon's own.
+
+A dependency named by a moving git ref is installed when the dependencies image is built and
+not again until its manifest changes, so pin such a dependency to a commit and bump it there.
+
+An extension's heavy dependency is not installed with the extension: `@toa.io/extensions.storages`
+declares the AWS and Cloudinary SDKs as optional peers, so the base image carries neither. What a
+component declares is what a deploy installs — the storages named in its `manifest.toa.yaml`,
+resolved through the context's annotation to their providers' packages, installed into `/toa`
+beside the extension that reads them. A composition whose components declare no `s3` storage
+carries no AWS SDK, whatever the context declares for another composition. A workspace installs
+the same set with `toa npm`.
+
+#### Extension Service Images
+
+`registry.services` says where an extension service's image comes from.
+
+`build`, the default, builds one per service into `registry.base`. Its tag carries the
+runtime version, so every Toa release rebuilds them all.
+
+`published` deploys the image the extension ships and builds nothing:
+
+```yaml
+# context.toa.yaml
+
+registry:
+  services: published
+```
+
+The images are tagged with the runtime version:
+
+```
+ghcr.io/toa-io/extension-exposition-gateway:1.0.0-alpha.285
+ghcr.io/toa-io/extension-introspection-explorer:1.0.0-alpha.285
+ghcr.io/toa-io/extension-configuration-values:1.0.0-alpha.285
+```
+
+The cluster pulls them from `ghcr.io` rather than from `registry.base`. They are public and
+`registry.credentials` does not apply; a cluster that cannot reach `ghcr.io` needs `build`.
+
+The field is environment-scoped like the rest:
+
+```yaml
+# context.toa.yaml
+
+registry:
+  base@local: localhost:5000
+  base@production: registry.digitalocean.com/acme
+  services: published
+  services@local: build
+```
+
+An extension that states no image is built whatever this says, as is a service a
+composition runs in its own pod.
+
 #### Build Options
 
 ```yaml
@@ -40,3 +219,48 @@ a secret containing required credentials can be specified using `registry.creden
 registry:
   credentials: docker-credentials-secret-name
 ```
+
+The secret itself is not Toa's to create, and a deploy that does not find it in the cluster is
+[refused](#secrets).
+
+#### Toa images
+
+Each release publishes `ghcr.io/toa-io/runtime:<version>` and, beside it, `:alpha` or `:latest`.
+The extension services share that version:
+
+```
+ghcr.io/toa-io/extension-exposition-gateway:1.0.0-alpha.285
+ghcr.io/toa-io/extension-introspection-explorer:1.0.0-alpha.285
+ghcr.io/toa-io/extension-configuration-values:1.0.0-alpha.285
+```
+
+Untagged manifests are deleted. A prerelease older than 90 days, and not among the last 30, is
+deleted with the four service images of that version. `:alpha`, `:latest`, and a version without
+a pre-release suffix stay.
+
+An application still on a deleted prerelease cannot pull `registry.services: published` and cannot
+rebuild `FROM` that runtime. Images already in `registry.base` keep running. Stay inside the
+window, or pin a graduated release.
+
+#### Userspace images
+
+`toa deploy production` writes two tags on the same composition image (`mono` and a service
+`registry.services: build` built are the same):
+
+- `<name>:<hash>` — what the chart pins. A hash of the runtime and the sources. `toa push` writes
+  this tag and nothing else.
+- `<name>:production` — moved onto that image. The next production deploy moves it again.
+
+`deps-<hash>` is not tagged with the environment. Do not name an environment eight hex digits, or
+`deps-` plus eight: a retention job would treat it as a content tag.
+
+A job the application runs deletes every `<hash>` and `deps-<hash>` older than 7 days, except the
+last 5, except the image `:production` (or `:staging`, or `:latest`) currently names. Then:
+
+```shell
+$ doctl registries garbage-collection start --include-untagged-manifests <registry>
+```
+
+Rollback is only those 7 days, those last 5, and whatever the environment tag still names. Turn
+the job on after the first deploy that writes that tag. The first garbage collection returns the
+space the old tags held, and blocks pushes while it runs.

package/src/deployment/.deployment/.describe/components.js

@@ -1,10 +1,8 @@
-'use strict'
-
 /**
  * @param {toa.deployment.Composition[]} compositions
  * @returns {string[]}
  */
-const components = (compositions) => {
+export const components = (compositions) => {
   /** @type {Set<string>} */
   const components = new Set()
 
@@ -16,5 +14,3 @@ const components = (compositions) => {
 
   return Array.from(components)
 }
-
-exports.components = components

package/src/deployment/.deployment/.describe/compositions.js

@@ -1,11 +1,30 @@
-'use strict'
+import { addVariables } from './variables.js'
+import { addMounts } from './mounts.js'
+import { fold } from './fold.js'
 
-const { addVariables } = require('./variables')
-
-function compositions (compositions, variables) {
+export function compositions(compositions, dependency) {
   for (const composition of compositions) {
-    addVariables(composition, variables)
+    const claimed = (dependency.services ?? []).filter(
+      (service) => service.workload?.includes(composition.name) === true
+    )
+
+    // a service brings components of its own — the identity components inside the gateway —
+    // and their variables are keyed by their own labels, not by the composition's
+    const keys = composition.components.concat(
+      ...claimed.map((service) => service.components ?? [])
+    )
+
+    addVariables(composition, dependency.variables, keys)
+    addMounts(composition, dependency.mounts, keys)
+
+    if (claimed.length > 0) {
+      // the label each hosted service's own Service selects on
+      composition.hosted = claimed.map((service) => service.name)
+
+      fold(composition, claimed, dependency)
+    }
+
+    if (dependency.probe !== undefined && dependency.probe !== false)
+      composition.probe ??= dependency.probe
   }
 }
-
-exports.compositions = compositions

package/src/deployment/.deployment/.describe/dependencies.js

@@ -1,10 +1,8 @@
-'use strict'
-
 /**
  * @param {toa.deployment.dependency.Reference[]} references
  * @returns {*}
  */
-const dependencies = (references) => {
+export const dependencies = (references) => {
   return references?.reduce((map, reference) => {
     const { name, alias, values } = reference
 
@@ -13,6 +11,3 @@ const dependencies = (references) => {
     return map
   }, {})
 }
-
-
-exports.dependencies = dependencies

package/src/deployment/.deployment/.describe/events.js

@@ -0,0 +1,63 @@
+/**
+ * Tells each component which of its events something consumes. An event nobody consumes gets
+ * no emitter, no exchange and no outbox row, and a component none of whose events are consumed
+ * gets no outbox at all.
+ *
+ * A component that declares events always gets the variable, empty when nothing is consumed —
+ * an absent variable means every event, which is what a run without a deployment gets.
+ *
+ * @param {toa.norm.Context} context
+ * @param {toa.deployment.Dependency} dependency
+ */
+export function events(context, dependency) {
+  const components = deployed(context)
+  const consumed = collect(components, context.events, dependency.events)
+
+  for (const { locator, events } of components) {
+    if (events === undefined) continue
+
+    const labels = Object.keys(events).filter((label) =>
+      consumed.has(locator.id + '.' + label)
+    )
+
+    dependency.variables[locator.label] ??= []
+
+    dependency.variables[locator.label].push({
+      name: VARIABLE + locator.uppercase,
+      value: labels.join(' ')
+    })
+  }
+}
+
+/**
+ * Every component this context deploys, its own and those its extensions bring. `components`
+ * holds only the former; the latter reach a deployment as instances of what they depend on,
+ * which is also how they are given the rest of their variables.
+ */
+function deployed(context) {
+  const components = new Map()
+
+  for (const instances of Object.values(context.dependencies ?? {}))
+    for (const { component } of instances) components.set(component.locator.id, component)
+
+  return [...components.values()]
+}
+
+/**
+ * Everything that consumes an event of this context: the receivers of its own components, what
+ * a dependency declares it consumes, and what the context says is consumed outside it.
+ */
+function collect(components, declared, contributed) {
+  const consumed = new Set(declared ?? [])
+
+  for (const label of contributed ?? []) consumed.add(label)
+
+  for (const { receivers } of components)
+    for (const [label, receiver] of Object.entries(receivers ?? {}))
+      // a receiver with a source consumes from a foreign broker, not from this context
+      if (receiver.source === undefined) consumed.add(label)
+
+  return consumed
+}
+
+const VARIABLE = 'TOA_EVENTS_'

package/src/deployment/.deployment/.describe/fold.js

@@ -0,0 +1,42 @@
+/**
+ * What a workload takes on by running services in its own pod: their variables, the ports
+ * they bind, and a readiness probe answered by one of them rather than by the workload.
+ *
+ * What it does *not* take on is how they are reached. A composition-hosted service keeps
+ * its own Service and Ingress — only the pods behind them change — so the ingress union
+ * belongs to `mono`, the one workload that fronts every service under a single name.
+ *
+ * @param {object} workload
+ * @param {toa.deployment.dependency.Service[]} services
+ * @param {toa.deployment.Dependency} dependency
+ */
+export function fold(workload, services, dependency) {
+  workload.variables ??= []
+
+  for (const service of services) {
+    if (service.variables !== undefined)
+      for (const variable of service.variables)
+        if (!workload.variables.some((item) => item.name === variable.name))
+          workload.variables.push(variable)
+
+    // every declared port is bound by the single process, none is primary
+    if (service.port !== undefined)
+      (workload.backends ??= []).push({
+        port: service.port,
+        path: service.ingress?.path ?? '/'
+      })
+
+    if (service.probe !== undefined && service.probe !== false)
+      workload.probe = service.probe
+  }
+
+  if (
+    workload.probe === undefined &&
+    dependency.probe !== undefined &&
+    dependency.probe !== false
+  )
+    workload.probe = dependency.probe
+
+  // the more specific prefix must come first, whatever the controller's tie-break
+  workload.backends?.sort((a, b) => b.path.length - a.path.length)
+}

package/src/deployment/.deployment/.describe/fold.test.js

@@ -0,0 +1,108 @@
+import { describe, it } from 'node:test'
+import assert from 'node:assert/strict'
+
+import { fold } from './fold.js'
+
+const service = (name, extra = {}) => ({ group: 'group', name, version: '0', ...extra })
+
+it('should collect the ports of every service', () => {
+  const workload = {}
+
+  fold(workload, [service('one', { port: 8000 }), service('two', { port: 8002 })], {})
+
+  assert.deepStrictEqual(workload.backends, [
+    { port: 8000, path: '/' },
+    { port: 8002, path: '/' }
+  ])
+})
+
+it('should take the path a service declares', () => {
+  const workload = {}
+
+  fold(workload, [service('one', { port: 8002, ingress: { path: '/explorer' } })], {})
+
+  assert.deepStrictEqual(workload.backends, [{ port: 8002, path: '/explorer' }])
+})
+
+it('should put the more specific prefix first', () => {
+  const workload = {}
+
+  fold(
+    workload,
+    [
+      service('one', { port: 8000, ingress: { path: '/' } }),
+      service('two', { port: 8002, ingress: { path: '/explorer' } })
+    ],
+    {}
+  )
+
+  assert.deepStrictEqual(
+    workload.backends.map((backend) => backend.path),
+    ['/explorer', '/']
+  )
+})
+
+it('should leave a service without a port out of the backends', () => {
+  const workload = {}
+
+  fold(workload, [service('one')], {})
+
+  assert.strictEqual(workload.backends, undefined)
+})
+
+describe('variables', () => {
+  it('should take those of every service', () => {
+    const workload = {}
+
+    fold(
+      workload,
+      [
+        service('one', { variables: [{ name: 'A', value: '1' }] }),
+        service('two', { variables: [{ name: 'B', value: '2' }] })
+      ],
+      {}
+    )
+
+    assert.deepStrictEqual(workload.variables, [
+      { name: 'A', value: '1' },
+      { name: 'B', value: '2' }
+    ])
+  })
+
+  it('should keep the first of a name', () => {
+    const workload = { variables: [{ name: 'A', value: 'own' }] }
+
+    fold(workload, [service('one', { variables: [{ name: 'A', value: 'theirs' }] })], {})
+
+    assert.deepStrictEqual(workload.variables, [{ name: 'A', value: 'own' }])
+  })
+})
+
+describe('probe', () => {
+  const probe = { path: '/.ready', port: 8004 }
+  const fallback = { path: '/.ready', port: 8001 }
+
+  it('should take the probe of a service over the default', () => {
+    const workload = {}
+
+    fold(workload, [service('one', { port: 8000, probe })], { probe: fallback })
+
+    assert.deepStrictEqual(workload.probe, probe)
+  })
+
+  it('should fall back to the default', () => {
+    const workload = {}
+
+    fold(workload, [service('one', { port: 8000 })], { probe: fallback })
+
+    assert.deepStrictEqual(workload.probe, fallback)
+  })
+
+  it('should ignore a disabled probe', () => {
+    const workload = {}
+
+    fold(workload, [service('one', { port: 8000, probe: false })], { probe: false })
+
+    assert.strictEqual(workload.probe, undefined)
+  })
+})

package/src/deployment/.deployment/.describe/index.js

@@ -1,13 +1,4 @@
-'use strict'
-
-const { dependencies } = require('./dependencies')
-const { components } = require('./components')
-const { compositions } = require('./compositions')
-const { variables } = require('./variables')
-const { services } = require('./services')
-
-exports.dependencies = dependencies
-exports.components = components
-exports.compositions = compositions
-exports.variables = variables
-exports.services = services
+export { dependencies } from './dependencies.js'
+export { components } from './components.js'
+export { compositions } from './compositions.js'
+export { services } from './services.js'

package/src/deployment/.deployment/.describe/mounts.js

@@ -0,0 +1,17 @@
+export function addMounts(composition, mounts, keys = composition.components) {
+  if (mounts === undefined) return
+
+  const used = new Set()
+
+  for (const [key, mount] of Object.entries(mounts)) {
+    if (key !== 'global' && !keys?.includes(key)) continue
+
+    for (const { name, path, claim } of mount) {
+      if (used.has(name)) continue
+
+      composition.mounts ??= []
+      composition.mounts.push({ name, path, claim })
+      used.add(name)
+    }
+  }
+}

package/src/deployment/.deployment/.describe/resources.js

@@ -0,0 +1,129 @@
+/**
+ * Every deployment states what it may take. One that states nothing is `BestEffort`: the
+ * first evicted under memory pressure and the last given CPU under contention — which on
+ * a busy node is slow enough for its own startup probe to kill it, in a loop, while the
+ * chart reports nothing wrong. That is not a thing to find out in production.
+ *
+ * A deployment takes its own declaration, or the context's `resources` where it has none.
+ * Deploying without any is a decision rather than an omission, and it is spelled
+ * `resources: null` — at either place.
+ */
+export function resources(context, values) {
+  for (const unit of units(values, evicted(context))) {
+    // `null` is an answer and `undefined` is not one, so the fallback reads only the latter
+    if (unit.deployment.resources === undefined)
+      unit.deployment.resources = context.resources
+
+    if (unit.deployment.resources === undefined)
+      throw new Error(
+        `${unit.subject} declares no resources. ` +
+          "Declare them on it or as the context's 'resources', " +
+          "or 'resources: null' to deploy it without any."
+      )
+
+    heap(unit.deployment)
+  }
+}
+
+/**
+ * The memory limit sizes the heap. Node reads the machine's memory rather than the
+ * container's, so without this the heap grows past the limit and the pod is killed instead
+ * of collected. What is left of the limit is the process itself: its code, its buffers and
+ * its threads. A deployment that states `NODE_OPTIONS` of its own keeps them.
+ */
+function heap(deployment) {
+  const limit = deployment.resources?.memory?.[1]
+
+  if (limit === undefined) return
+
+  deployment.variables ??= []
+
+  if (deployment.variables.some((variable) => variable.name === NODE_OPTIONS)) return
+
+  const megabytes = Math.floor((quantity(limit) * HEAP_SHARE) / 2 ** 20)
+
+  deployment.variables.push({
+    name: NODE_OPTIONS,
+    value: `--max-old-space-size=${megabytes}`
+  })
+}
+
+/**
+ * A Kubernetes quantity, in bytes.
+ *
+ * @param {string | number} value
+ * @returns {number}
+ */
+export function quantity(value) {
+  const match = String(value).match(/^(\d+(?:\.\d+)?)([KMGTPE]i?|[kmun])?$/)
+
+  if (match === null) throw new Error(`'${value}' is not a quantity`)
+
+  const [, number, suffix = ''] = match
+  const scale = SUFFIXES[suffix]
+
+  if (scale === undefined) throw new Error(`'${value}' is not a memory quantity`)
+
+  return Number(number) * scale
+}
+
+const NODE_OPTIONS = 'NODE_OPTIONS'
+const HEAP_SHARE = 0.75
+
+const SUFFIXES = {
+  '': 1,
+  K: 1e3,
+  M: 1e6,
+  G: 1e9,
+  T: 1e12,
+  P: 1e15,
+  E: 1e18,
+  Ki: 2 ** 10,
+  Mi: 2 ** 20,
+  Gi: 2 ** 30,
+  Ti: 2 ** 40,
+  Pi: 2 ** 50,
+  Ei: 2 ** 60
+}
+
+/**
+ * The labels of the components the context evicts, whether or not the eviction was applied:
+ * `toa env -c` does not apply it, since a local run of an evicted component still needs its
+ * variables.
+ *
+ * @returns {Set<string>}
+ */
+function evicted(context) {
+  const ids = new Set(context.evicted?.components ?? [])
+
+  return new Set(
+    (context.components ?? [])
+      .filter(({ locator }) => ids.has(locator.id))
+      .map(({ locator }) => locator.label)
+  )
+}
+
+function* units(values, evicted) {
+  if (values.mono !== undefined)
+    yield { deployment: values.mono, subject: 'The mono deployment' }
+
+  for (const composition of values.compositions ?? []) {
+    // a composition of what the context evicts is here only for a local run of it, and Toa
+    // deploys none of it, so nothing asks what it may take
+    if (
+      composition.components?.length > 0 &&
+      composition.components.every((label) => evicted.has(label))
+    )
+      continue
+
+    yield { deployment: composition, subject: `Composition '${composition.name}'` }
+  }
+
+  for (const service of values.services ?? []) {
+    // a service a composition runs has no deployment of its own to size; the composition
+    // it runs in states what the pod may take
+    if (service.workload !== undefined) continue
+
+    yield { deployment: service, subject: `Service '${service.name}'` }
+  }
+}