@olivervea/pipelines 2.0.0-beta.2 → 2.0.0-beta.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +125 -62
- package/manifest.schema.json +208 -11
- package/package.json +6 -4
- package/src/assets.d.ts +2 -2
- package/src/build/build.ts +121 -51
- package/src/build/cli.ts +29 -22
- package/src/build/elf.ts +8 -7
- package/src/build/hashes.ts +82 -20
- package/src/build/project.ts +95 -28
- package/src/builder/pipeline.ts +344 -88
- package/src/builder/refs.ts +15 -5
- package/src/core/brands.ts +7 -5
- package/src/core/config-refs.ts +103 -0
- package/src/core/config.ts +224 -0
- package/src/core/external.ts +86 -0
- package/src/core/handlers.ts +61 -15
- package/src/core/models.ts +32 -15
- package/src/core/origin.ts +23 -0
- package/src/core/secrets.ts +2 -2
- package/src/core/steps.ts +89 -38
- package/src/core/types.ts +5 -1
- package/src/core/webhooks.ts +21 -7
- package/src/ctx/files.ts +111 -40
- package/src/ctx/index.ts +84 -31
- package/src/ctx/sh.ts +105 -44
- package/src/ctx/types.ts +12 -4
- package/src/describe/describe.ts +701 -133
- package/src/describe/manifest-schema.ts +258 -61
- package/src/describe/mapper-source.ts +249 -104
- package/src/describe/record.ts +67 -37
- package/src/dev/cli.ts +72 -35
- package/src/harness/env.ts +45 -25
- package/src/harness/execute.ts +58 -29
- package/src/harness/index.ts +22 -9
- package/src/harness/main.ts +239 -78
- package/src/harness/result.ts +38 -22
- package/src/harness/validate.ts +77 -15
- package/src/index.ts +124 -11
- package/src/library/checks.ts +28 -0
- package/src/library/compose-deploy.ts +113 -0
- package/src/library/github-source.ts +100 -47
- package/src/library/helm-deploy-ssh.ts +452 -108
- package/src/library/helm-deploy.ts +449 -0
- package/src/library/http-health.ts +7 -5
- package/src/library/image-build.ts +499 -105
- package/src/library/index.ts +48 -12
- package/src/library/models.ts +33 -9
- package/src/library/ssh.ts +67 -0
- package/src/testing/index.ts +74 -36
- package/src/util/canonical.ts +7 -4
- package/src/util/handler-hash.ts +17 -7
- package/src/util/hash.ts +5 -5
- package/src/util/tar.ts +86 -36
- package/src/library/script.ts +0 -85
package/README.md
CHANGED
|
@@ -1,25 +1,25 @@
|
|
|
1
1
|
# @olivervea/pipelines
|
|
2
2
|
|
|
3
|
-
The [Olve.Pipelines](https://github.com/OliverVea/Olve.Pipelines) v2 SDK
|
|
3
|
+
The [Olve.Pipelines](https://github.com/OliverVea/Olve.Pipelines) v2 SDK. It has four parts:
|
|
4
4
|
|
|
5
5
|
- the **authoring API**: models, steps, handlers and the typed pipeline builder;
|
|
6
6
|
- the **step harness**: the runtime compiled into each pipeline's single executable (`describe`, `run`, `map-trigger`);
|
|
7
|
-
- the **build tool** (`olve-build`), which turns a pipeline directory into `step.tar.
|
|
7
|
+
- the **build tool** (`olve-build`), which turns a pipeline directory into `step.tar.zst` and `manifest.json`;
|
|
8
8
|
- the **dev runner** (`olve-dev`), which runs the same harness path from source (used by `pl dev`).
|
|
9
9
|
|
|
10
|
-
Design: [`docs/design/pipelines-v2.md`](https://github.com/OliverVea/Olve.Pipelines/blob/
|
|
10
|
+
Design: [`docs/design/pipelines-v2.md`](https://github.com/OliverVea/Olve.Pipelines/blob/main/docs/design/pipelines-v2.md). Spike findings: [`pipelines-v2-spikes.md`](https://github.com/OliverVea/Olve.Pipelines/blob/main/docs/design/pipelines-v2-spikes.md).
|
|
11
11
|
The manifest format is in [`manifest.schema.json`](manifest.schema.json), generated from [`src/describe/manifest-schema.ts`](src/describe/manifest-schema.ts).
|
|
12
12
|
|
|
13
13
|
```
|
|
14
|
-
sdk/
|
|
15
|
-
src/index.ts authoring API (`@
|
|
16
|
-
src/core/ models, secrets, steps, implement()
|
|
14
|
+
src/sdk/
|
|
15
|
+
src/index.ts authoring API (`@olivervea/pipelines`)
|
|
16
|
+
src/core/ models, secrets, config, steps, implement()
|
|
17
17
|
src/builder/ pipeline builder (types + runtime records), Ref<T>
|
|
18
|
-
src/library/ library steps (
|
|
18
|
+
src/library/ library steps (gitHubSource, imageBuild, helmDeployOverSsh, helmDeployInCluster, composeDeploy), their exported helpers (httpHealthCheck, sshSession, checkSpa) and SDK models; public API only
|
|
19
19
|
src/describe/ describe: manifest, checks, mapper recording + source analysis, manifest schema
|
|
20
|
-
src/harness/ harness CLI modes, env contract, envelope, result upload (`@
|
|
20
|
+
src/harness/ harness CLI modes, env contract, envelope, result upload (`@olivervea/pipelines/harness`)
|
|
21
21
|
src/ctx/ the ctx handed to handlers
|
|
22
|
-
src/testing/ testHandler (`@
|
|
22
|
+
src/testing/ testHandler (`@olivervea/pipelines/testing`)
|
|
23
23
|
src/build/ olve-build
|
|
24
24
|
src/dev/ olve-dev
|
|
25
25
|
examples/self-deploy/ fixture pipeline (the self-deploy shape) used by the tests
|
|
@@ -30,13 +30,12 @@ sdk/
|
|
|
30
30
|
|
|
31
31
|
## Install
|
|
32
32
|
|
|
33
|
-
The package is `@olivervea/pipelines` on npm;
|
|
34
|
-
in the pipeline directory's `package.json`:
|
|
33
|
+
The package is `@olivervea/pipelines` on npm; a pipeline directory's `package.json` depends on it by that name:
|
|
35
34
|
|
|
36
35
|
```json
|
|
37
36
|
{
|
|
38
37
|
"type": "module",
|
|
39
|
-
"dependencies": { "@
|
|
38
|
+
"dependencies": { "@olivervea/pipelines": "2.0.0-beta.3" },
|
|
40
39
|
"devDependencies": { "@types/bun": "1.4.3", "typescript": "7.0.2" }
|
|
41
40
|
}
|
|
42
41
|
```
|
|
@@ -47,11 +46,9 @@ bunx olve-dev describe .
|
|
|
47
46
|
bunx olve-build . --out .olve/out
|
|
48
47
|
```
|
|
49
48
|
|
|
50
|
-
- Keep the alias name: `olve-build` generates imports of `@olve/pipelines/harness`, so an unaliased
|
|
51
|
-
`@olivervea/pipelines` dependency does not build.
|
|
52
49
|
- The package ships TypeScript sources (Bun runs them directly; `harness.sdkHash` hashes them), so the
|
|
53
|
-
same version gives the same `sdkHash` and handler hashes from npm or from `file:../sdk`.
|
|
54
|
-
- Inside this repo, `.pipelines/` and `.pipelines-test/` use `"@
|
|
50
|
+
same version gives the same `sdkHash` and handler hashes from npm or from `file:../src/sdk`.
|
|
51
|
+
- Inside this repo, `.pipelines/` and `.pipelines-test/` use `"@olivervea/pipelines": "file:../src/sdk"`.
|
|
55
52
|
|
|
56
53
|
### Releases
|
|
57
54
|
|
|
@@ -67,7 +64,7 @@ A pipeline directory (the binding path, `.pipelines/` by default) looks like thi
|
|
|
67
64
|
```
|
|
68
65
|
models.ts data models (contract file)
|
|
69
66
|
pipeline.ts steps + `export default pipeline(…)` (contract file)
|
|
70
|
-
handlers
|
|
67
|
+
handlers/*.ts one per custom step: `export default implement(step, fn)` (pairs by the step, not the file name)
|
|
71
68
|
handlers/lib/… shared helpers (not handlers)
|
|
72
69
|
scripts/*.sh embedded with `import s from "../scripts/x.sh" with { type: "file" }`
|
|
73
70
|
```
|
|
@@ -77,7 +74,7 @@ Contract files hold contracts only. Describe evaluates them without loading hand
|
|
|
77
74
|
### Models, secrets, steps
|
|
78
75
|
|
|
79
76
|
```ts
|
|
80
|
-
import { model, z, fileRef, optional, secret, step, steps, Source } from "@
|
|
77
|
+
import { model, z, fileRef, optional, external, secret, step, steps, Source } from "@olivervea/pipelines";
|
|
81
78
|
|
|
82
79
|
export const BuildArtifact = model("BuildArtifact", z.object({
|
|
83
80
|
version: z.string(),
|
|
@@ -90,16 +87,12 @@ export const deployBeta = step({
|
|
|
90
87
|
name: "deploy-beta", // identity: lower-case kebab, ≤ 63 chars
|
|
91
88
|
description: "Deploys to beta.", // required (may be ""); shown in the UI, not versioned
|
|
92
89
|
image: "alpine:latest", // any image; the harness brings its own runtime
|
|
93
|
-
input: { artifact: BuildArtifact }, // model | optional(model) | plain zod schema
|
|
90
|
+
input: { artifact: BuildArtifact }, // model | optional(model) | plain zod schema | external(model, …)
|
|
94
91
|
output: BetaDeployment, // a model, or omitted (no output)
|
|
95
92
|
secrets: [SSH_PRIVATE_KEY], // only these are mounted; ctx.secrets is typed to them
|
|
96
93
|
timeoutSeconds: 900, // optional
|
|
97
94
|
retries: 1, // optional: a failed attempt is retried (fresh pod) up to n times
|
|
98
95
|
});
|
|
99
|
-
|
|
100
|
-
import testSh from "./scripts/test.sh" with { type: "file" };
|
|
101
|
-
export const test = steps.script({ name: "test", description: "Runs the tests.", image: "…", script: testSh, input: { source: Source }, output: TestReport,
|
|
102
|
-
env: { TAG: ({ source }, run) => `${run.version}-${source.sha.slice(0, 7)}` } }); // env values may be functions of the inputs
|
|
103
96
|
```
|
|
104
97
|
|
|
105
98
|
Rules:
|
|
@@ -110,49 +103,108 @@ Rules:
|
|
|
110
103
|
- **Trigger mappers** produce `z.input`, so fields with defaults may be left out. **Handlers, mapper refs and `when`** see `z.output`.
|
|
111
104
|
- A **plain-schema input** has no model to match on, so the step needs a mapper.
|
|
112
105
|
- **`optional(M)`** means "maybe skipped": `M` must still have an earlier producer.
|
|
106
|
+
- **`external(M, { title, description?, from? })`** is answered from outside the run; `from` lists `callback()` (another system POSTs to the URL an earlier step gets from `ctx.inputUrl(step, input)`), `poll({ url, valuePath, intervalSeconds? })` and `person()` (the default). It needs no producer and no mapper entry; the step waits until it is answered, then the value (validated against `M`) is in its input like any other. Failure handlers can't have one.
|
|
113
107
|
|
|
114
|
-
|
|
108
|
+
### Config
|
|
115
109
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
110
|
+
Values that differ per deployment (hosts, namespaces, which identity provider) are **config**: typed entries the pipeline declares, whose values live on the instance (one YAML document per pipeline, edited with `pl config` or the dashboard), not in the repo.
|
|
111
|
+
|
|
112
|
+
```ts
|
|
113
|
+
import { config, secret, secretRef, step, z } from "@olivervea/pipelines";
|
|
114
|
+
|
|
115
|
+
export const DEX_CLIENT_SECRET = secret("DEX_CLIENT_SECRET", "Dex client secret.");
|
|
116
|
+
export const AUTH = config("auth", "How users authenticate.", z.discriminatedUnion("mode", [
|
|
117
|
+
z.object({ mode: z.literal("none") }),
|
|
118
|
+
z.object({ mode: z.literal("dex"), issuer: z.string().url(), clientSecret: secretRef(DEX_CLIENT_SECRET) }),
|
|
119
|
+
]));
|
|
120
|
+
export const TARGET = config("target", "Where the release is deployed.",
|
|
121
|
+
z.object({ host: z.string(), namespace: z.string().default("olve-pipelines") }));
|
|
122
|
+
|
|
123
|
+
export const deploy = step({ name: "deploy", description: "Deploys the release.", image: "alpine:3.22",
|
|
124
|
+
config: [AUTH, TARGET], // ctx.config is typed to exactly these
|
|
125
|
+
secrets: [DEX_CLIENT_SECRET] }); // a secretRef's secret must be declared too
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
```yaml
|
|
129
|
+
# the instance's config document for this pipeline
|
|
130
|
+
auth: { mode: dex, issuer: https://dex.example.com, clientSecret: { secret: DEX_CLIENT_SECRET } }
|
|
131
|
+
target: { host: deploy@box }
|
|
132
|
+
```
|
|
121
133
|
|
|
122
|
-
|
|
134
|
+
- `config(name, description, schema)` mirrors `secret(...)`. Names match `^[a-z][a-zA-Z0-9]*$` (they are YAML keys).
|
|
135
|
+
- An entry whose schema accepts `undefined` (`.optional()`, `.default(…)`) is optional; every other entry is required. Defaults are applied when config is read, never written into the document.
|
|
136
|
+
- Handlers see `ctx.config` with the output view (defaults applied); a discriminated union narrows as usual.
|
|
137
|
+
- `secretRef(S)` is a field whose value is `{ secret: "<S.name>" }`: it names a secret, never holds its value. The handler reads the value from `ctx.secrets`. A step that receives an entry containing a `secretRef` must declare that secret.
|
|
138
|
+
- Config is per **run**: the run pins a config snapshot at start; continues and step reruns reuse it. Secrets are per attempt.
|
|
139
|
+
- Library steps (and any step's `settings`) take **config references** in place of values: `host: TARGET.host`, `target: TARGET.beta`. A reference is a typed proxy (`ConfigRef<T>`, `T` = the field's output type; `Configurable<T> = T | ConfigRef<T>`), so a `number` field or an optional field can't go where a `string` is required. Fields named `kind`, `name`, `description` or `schema` (the entry's own properties) are reached through `$` (`TARGET.$.name`); `TARGET.$` and `TARGET` itself reference the whole entry. The step's `config` gets every entry referenced in its `settings` (`configEntries` in the manifest); the manifest records each reference as `{"$config": "<entry>.<path>"}` (field names must not contain `.`; literal keys starting with `$` are reserved); the handler resolves them with `resolveConfigRefs(value, ctx.config)` (defaults applied). `describe` and reconcile reject a reference to an entry the step doesn't receive or a field the entry doesn't have.
|
|
140
|
+
- `clusterAccess: true` on a step asks for the instance's step ServiceAccount token (`Kubernetes:StepServiceAccount`; manifest `clusterAccess: true`, omitted otherwise). Without the instance setting the pod gets no token, like every step.
|
|
123
141
|
|
|
124
142
|
### Library steps
|
|
125
143
|
|
|
126
|
-
Library steps carry their own implementation and image; they take typed
|
|
144
|
+
Library steps carry their own implementation and image; they take typed settings and use the SDK models `Source`, `BuiltImage`, `PushedImage` and `HelmRelease`. Each is a function returning `implement(step({ kind, settings, … }), handler)`, built on the public API only (a test enforces it), so a step package outside the SDK is written the same way (see [Reusable steps](#reusable-steps)). The set grows from real use: a step or an option exists because several pipelines use it; what one pipeline needs stays a custom step there.
|
|
127
145
|
|
|
128
146
|
| Step | Image | Input → output | Config |
|
|
129
147
|
|---|---|---|---|
|
|
130
|
-
| `steps.
|
|
131
|
-
| `steps.
|
|
132
|
-
| `steps.
|
|
133
|
-
| `steps.helmDeployOverSsh` | `alpine:3.
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
- `
|
|
137
|
-
-
|
|
138
|
-
-
|
|
148
|
+
| `steps.gitHubSource` | `alpine:3.22` (work happens in the harness) | `{ repo: "owner/name", sha }` (plain schemas: needs a mapper) → `Source` | `token?: Secret` (sent as `Authorization: Bearer` when set; unset = anonymous; becomes the step's secret), `apiUrl?` (default `https://api.github.com`) |
|
|
149
|
+
| `steps.imageBuild` | `gcr.io/kaniko-project/executor:debug` | `{ source: Source }` → `BuiltImage { repository, tag, digest, tar }`; with `push`: `PushedImage { repository, tag, digest }` | `repository` (image name; tag = run version, `-attempt<n>` on reruns), `context?` (dir in the source tree with the `Dockerfile`; default the root), `push?` (push to `repository` instead of a tarball), `contentTag?` (with `push`: tag `src-<context hash>`, skip when the registry has it), `registryMap?` (kaniko `--registry-map`: registry → mirror prefix; default none), `stallTimeoutSeconds?` (default 300, 0 = off), `retries?` (default 2), `registryAuth?` (`{ username, password: Secret }`: log in to the `repository` registry; or a config reference to `{ username, password: { secret: NAME } }`, with `NAME` in `secrets`), `insecureRegistry?` (that registry is plain HTTP), `secrets?` (declared secrets a config `registryAuth` may name), `enabled?` (`false` skips the step) |
|
|
150
|
+
| `steps.helmDeployOverSsh` | `alpine:3.22` (+ `apk add openssh-client` at run time) | `{ source: Source, <key>: BuiltImage }` per `images` (`PushedImage` with `fromRegistry`) → `HelmRelease { release, namespace, image? }` (`image`: the first image; absent without images) | `host` (`user@host`), `sshKey: Secret`, `release`, `namespace`, `chart` (dir in the source), `values?` (chart-relative `-f`), `set?`, `images?` (input key → values prefix; default `{ image: "image" }`; `{}` = the chart alone), `fromRegistry?`, `imageRefs?` (`PushedImage` input key → values prefix: set as `<prefix>.repository`/`.tag`/`.digest`, not pulled; works next to tarball `images`), `deployments?` (default `[release]`, none without images), `rolloutTimeoutSeconds?` (default 120), `dependencies?` (pull `Chart.yaml`'s dependencies on the host), `helmWait?` / `atomic?` (`helm --wait` / `--atomic`), `healthUrl?` (URL or `{ url, attempts?, delaySeconds? }`) |
|
|
151
|
+
| `steps.helmDeployInCluster` (and the `target` form of `helmDeployOverSsh`) | `alpine/helm:3.16.2` (ssh: `alpine:3.22`) | `{ source: Source, <key>: PushedImage }` per `images`, `<key>: optional(PushedImage)` per `imageRefs` → `HelmRelease` (`image`: `repository:tag@digest` of the first image) | `target` (a `select` case's reference to a `helmTarget()` variant: `{ kind: "inCluster", namespace, release, values, set, url? }` for helmDeployInCluster, the same plus `host` with `kind: "ssh"` for helmDeployOverSsh; `url` is where step pods reach the release, default its in-cluster Service URL, and is passed on in `HelmRelease.url`), `sshKey: Secret` (ssh only), `chart`, `inlineValues?` (objects written as values files after the target's), `images?`, `imageRefs?`, `deployments?` (ssh), `rolloutTimeoutSeconds?` (default 120), `healthPath?`; helmDeployInCluster declares `clusterAccess` |
|
|
152
|
+
| `steps.composeDeploy` | `alpine:3.22` (work happens in the harness) | `{ image: PushedImage }` → `HelmRelease { release: <service>, namespace: "compose", image: <repository>@<digest> }` | `target` (a `select` case's reference to the compose variant of `deployTarget()`: `{ kind: "compose", deployerUrl, service, token: { secret }, healthUrl? }`), `token: Secret` (must be the secret the target names; the step's only secret), `timeoutSeconds?` |
|
|
153
|
+
|
|
154
|
+
- `gitHubSource` downloads `<apiUrl>/repos/<repo>/tarball/<sha>` (retrying 5xx), strips the top-level directory and outputs it as `tree`.
|
|
155
|
+
- `imageBuild` copies the tree to `/kaniko/build-context` (kaniko wipes other directories between multi-stage builds), builds `--context=/kaniko/build-context[/<context>] --dockerfile=<that>/Dockerfile` with `--no-push --tar-path --digest-file`, and outputs the tarball. A missing context or Dockerfile fails the step before kaniko starts. `repository` in the output is fully qualified the way a runtime names a loaded image (`x` → `docker.io/library/x`). A **stall watchdog** kills kaniko when it prints nothing for `stallTimeoutSeconds` (a base-image pull can stall on an idle connection at "Unpacking rootfs", and kaniko has no read timeout), failing the attempt; `retries` then starts a fresh attempt. The retry is deliberately a new pod, never a second kaniko in the same container: kaniko unpacks the base image over the container root and does not clean up when killed.
|
|
156
|
+
- **Caches never block a build.** `registryMap` maps registries to pull-through caches (e.g. `{ "mcr.microsoft.com": "mcr-cache.example.com" }`; kaniko under gVisor can stall pulling from mcr.microsoft.com directly, a nearby cache doesn't). There is none by default. Before kaniko starts, every mirror host gets a 5 s `GET https://<host>/v2/` (200 or 401 = up); one that doesn't answer is dropped with a log line (`<cache> unreachable; pulling from <registry> directly`). After that, kaniko itself copes: it falls back to the original registry when a mirror can't serve an image's manifest (`RetrieveRemoteImage`; `--skip-default-registry-fallback` is never set). A cache that dies mid-download (after the manifest came from it) fails or stalls the attempt; `retries` starts a fresh one, which probes again.
|
|
157
|
+
- **Registry mode** (`push: true`, for images too big for the step's ~1 GiB scratch space and Garage): kaniko runs without `--no-push`/`--tar-path`, pushes to `repository` (it must start with a registry host; anonymous unless `registryAuth`) and the output is a `PushedImage` with the digest from `--digest-file`. With `contentTag`, the tag is `src-` + the first 12 hex of the sha256 of the context directory's deterministic tar (the bytes `ctx.output.dir` would write), computed from the fetched tree; a `HEAD https://<host>/v2/<path>/manifests/<tag>` (answering a Basic or Bearer challenge: anonymous token, or the `registryAuth` login; an anonymous lookup that is denied counts as "not there") that answers a `Docker-Content-Digest` skips kaniko and outputs that digest. With `registryAuth`, the step declares the password secret and writes `$DOCKER_CONFIG/config.json` (`/kaniko/.docker` in the kaniko image) for that registry before kaniko runs, so pushes and base-image pulls from it are logged in. `insecureRegistry` makes the lookups plain HTTP and adds kaniko `--insecure --insecure-registry=<host>`.
|
|
158
|
+
- `helmDeployOverSsh` gets each image into the host's **k3s** containerd: a `BuiltImage` tar is streamed in (`nerdctl --address /run/k3s/containerd/containerd.sock --namespace k8s.io load`), a `PushedImage` (`fromRegistry`) is pulled there (`… pull -q`, so the rollout doesn't wait on the pull). It copies the chart to `/tmp/<release>-helm-<namespace>`, runs `helm upgrade --install` from inside it (values, then `set`, then per image `<prefix>.repository`/`.tag`/`.pullPolicy` — `Never` for tarballs, `IfNotPresent` for registry images — which always win), waits for `kubectl rollout status` of each of `deployments` in order (`deploy/<name>` for a bare name), and with `healthUrl` polls it (5 × 5 s unless given) from the harness. `images: {}` deploys the chart alone: the step's only input is `source`, no image is imported and no image flag set, and nothing is waited on unless `deployments` names it. `dependencies: true` runs `helm pull <name> --repo <repository> --version <version>` (or the `oci://` reference) into the copied chart's `charts/` for each dependency in `Chart.yaml`; `helmWait` / `atomic` add `--wait` / `--atomic` with `--timeout=<rolloutTimeoutSeconds>s` and make `deployments` default to none. Several images (one chart, two Deployments):
|
|
159
|
+
|
|
160
|
+
```ts
|
|
161
|
+
steps.helmDeployOverSsh({ name: "deploy-beta", description: "", host: "oliver@bulwark-m2", sshKey: SSH_PRIVATE_KEY,
|
|
162
|
+
release: "olve-slimerepublics", namespace: "apps-beta", chart: "helm",
|
|
163
|
+
images: { server: "image", web: "web.image" }, deployments: ["olve-slimerepublics-server", "olve-slimerepublics-web"] });
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
- **Deploy targets.** `helmTarget(defaults?, extend?)` is the `inCluster | ssh` union. `deployTarget({ namespace?, release?, extend?, composeToken, optional? })` adds `compose` (`{ deployerUrl, service, token: secretRef(composeToken), healthUrl?, …extend }`) and, with `optional: true`, `none` (`{ kind: "none" }`, no other fields; its `select` case usually has no steps). A library step that references a target narrowed to one case needs only that case's secrets: the helm steps don't declare the compose `token`.
|
|
167
|
+
- `composeDeploy` resolves its target, checks `kind` is `compose` and that `token.secret` is the declared secret, then `POST <deployerUrl>/v1/deploy` `{ service, image: "<repository>@<digest>" }` with `Authorization: Bearer <token>`. Each line of the answer's `log` is logged as `deployer: …`; anything but HTTP 2xx with `status: "deployed"` fails the step with the deployer's status and message. With `healthUrl` it then runs `httpHealthCheck` (12 × 5 s). The deployer itself (`compose/deployer`) and its API: [docs/setup/compose-deploy.md](../docs/setup/compose-deploy.md).
|
|
139
168
|
|
|
140
169
|
**Helpers.** General functions the library steps are built on, exported for custom handlers:
|
|
141
170
|
|
|
142
171
|
| Helper | What it does |
|
|
143
172
|
|---|---|
|
|
144
|
-
| `httpHealthCheck(ctx, url, { attempts?, delaySeconds?, headers? })` | GETs `url` until it returns 2xx (redirects followed), retrying network errors and non-2xx statuses; defaults 5 attempts, 5 s apart
|
|
173
|
+
| `httpHealthCheck(ctx, url, { attempts?, delaySeconds?, headers? })` | GETs `url` until it returns 2xx (redirects followed), retrying network errors and non-2xx statuses; defaults 5 attempts, 5 s apart. Throws the last error when the attempts run out. Runs in the harness, so the image needs no curl. `helmDeployOverSsh`'s `healthUrl` uses it. |
|
|
174
|
+
| `serviceUrl(release)` | `http://<release>.<namespace>.svc.cluster.local` for a `HelmRelease` (charts that name the Service after the release). |
|
|
175
|
+
| `checkSpa(ctx, url)` | `GET <url>/` must reference an `/assets/*.js` bundle (Vite's layout) and the bundle must be served as JavaScript; returns `{ html, bundle }` for app-specific checks. |
|
|
176
|
+
| `sshSession(ctx, { host, key })` | Installs openssh-client when the image has no ssh (apk or apt-get), writes the key to `<tmpDir>/ssh/id` (trailing newline, mode 600; host key not verified) and returns `{ run(remote), text(remote), copy(local, remote), shell(remote) }`; `shell` is the quoted ssh command for pipes (`ctx.sh(`printf '%s' "$V" \| ${s.shell("kubectl apply -f -")}`)`). `helmDeployOverSsh` uses it. |
|
|
145
177
|
|
|
146
178
|
```ts
|
|
147
|
-
import { httpHealthCheck, implement } from "@
|
|
179
|
+
import { httpHealthCheck, implement } from "@olivervea/pipelines";
|
|
148
180
|
export default implement(smoke, async (_, ctx) => { await httpHealthCheck(ctx, "http://app.ns.svc.cluster.local/api/health", { attempts: 10 }); });
|
|
149
181
|
```
|
|
150
182
|
|
|
183
|
+
### Reusable steps
|
|
184
|
+
|
|
185
|
+
A step several pipelines use is a function returning `implement(step(…), handler)`, staged directly: it carries its handler, so it has no `handlers/` file. The library steps are written this way, so a module in a repo or a package of its own works the same.
|
|
186
|
+
|
|
187
|
+
```ts
|
|
188
|
+
import { implement, resolveConfigRefs, serviceUrl, step, HelmRelease, type Configurable, type LiteralName } from "@olivervea/pipelines";
|
|
189
|
+
|
|
190
|
+
export function smokeTest<const N extends string>(def: { name: LiteralName<N>; description: string; path: Configurable<string> }) {
|
|
191
|
+
return implement(
|
|
192
|
+
step({ name: def.name, description: def.description, kind: "smokeTest", image: "alpine:3.22", input: { release: HelmRelease }, settings: { path: def.path } }),
|
|
193
|
+
async ({ release }, ctx) => void (await ctx.http.expectOk(`${release.url ?? serviceUrl(release)}${resolveConfigRefs(def.path, ctx.config)}`)),
|
|
194
|
+
);
|
|
195
|
+
}
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
- **`kind`**: a label, required for a step that carries its implementation (`custom` is a step with a handler file). Prefix a package's kinds (`acme/smokeTest`).
|
|
199
|
+
- **`settings`**: what the step was set up with, recorded as the step's manifest `config` (so part of its version). JSON values and config references; the step receives every entry its settings reference.
|
|
200
|
+
- **Identity**: `implement` and `step` record the module that called them (from the stack; a tail call can name a module further up, which only adds to what is hashed). The build bundles those modules, minus `pipeline.ts` and `models.ts`, like a handler, and the kind's hash (`kindHashes` in the generated `handlers.ts`) goes into each such step's `handlerHash`. A step that carries its implementation and has no other module (defined and implemented in a contract file) is a build error.
|
|
201
|
+
- `implement` refuses a step that already carries an implementation; a `handlers/*.ts` file for such a step is a describe error.
|
|
202
|
+
|
|
151
203
|
### Handlers
|
|
152
204
|
|
|
153
205
|
```ts
|
|
154
206
|
// handlers/deploy-beta.ts
|
|
155
|
-
import { implement } from "@
|
|
207
|
+
import { implement } from "@olivervea/pipelines";
|
|
156
208
|
import { deployBeta } from "../pipeline";
|
|
157
209
|
import deployScript from "../scripts/deploy.sh" with { type: "file" };
|
|
158
210
|
|
|
@@ -165,7 +217,7 @@ export default implement(deployBeta, async ({ artifact }, ctx) => {
|
|
|
165
217
|
|
|
166
218
|
| `ctx.` | |
|
|
167
219
|
|---|---|
|
|
168
|
-
| `sh(cmd \| script, { env, cwd, args })`, `sh.text(…)` | Runs a command string, or an embedded script, through the image's shell. The script is copied to `$OLVE_DIR/tmp/scripts/` first. The shell is the first of `/bin/sh`, `/busybox/sh`, `/bin/bash` that exists; a shell-less image gets a clear `ShellNotFoundError`. A script's shebang is honoured when its interpreter exists. A string counts as an embedded script when it is a `/$bunfs/…` path, or an absolute path to an existing `*.sh` or `*.bash` file. Pass `{ script }` to force script mode. A non-zero exit throws a `CommandError` that carries `exitCode`. |
|
|
220
|
+
| `sh(cmd \| script, { env, cwd, args })`, `sh.text(…)` | Runs a command string, or an embedded script, through the image's shell. The script is copied to `$OLVE_DIR/tmp/scripts/` first. The shell is the first of `/bin/sh`, `/busybox/sh`, `/bin/bash` that exists; a shell-less image gets a clear `ShellNotFoundError`. A script's shebang is honoured when its interpreter exists. A string counts as an embedded script when it is a `/$bunfs/…` path, or an absolute path to an existing `*.sh` or `*.bash` file. Pass `{ script }` to force script mode. `args` are the script's arguments, or `$1`, `$2`… of a command string. A non-zero exit throws a `CommandError` that carries `exitCode`. |
|
|
169
221
|
| `exec([argv…], { env, cwd, idleTimeoutSeconds })` | Runs a binary directly, with no shell (works on distroless). `idleTimeoutSeconds` (also on `sh`) is an inactivity watchdog: no stdout/stderr output for that long → SIGTERM, SIGKILL 10 s later, and a `CommandStalledError`. |
|
|
170
222
|
| `readText(path)`, `readJson(path)` | Paths are relative to the cwd. |
|
|
171
223
|
| `files.fetch(ref)` | Downloads a file reference and verifies its sha256. Directory refs are extracted. Returns a local path. |
|
|
@@ -173,6 +225,7 @@ export default implement(deployBeta, async ({ artifact }, ctx) => {
|
|
|
173
225
|
| `http.fetch`, `http.getJson(url, { headers })`, `http.expectOk(url, { method, headers })` | Bound to `ctx.signal`. A non-2xx status throws a readable error, with the URL redacted after `?`. |
|
|
174
226
|
| `retry({ attempts, delaySeconds, backoff? }, fn)` | Retries `fn`. Can be cancelled. |
|
|
175
227
|
| `secrets.NAME` | Typed to the step's declared secrets. Read from the env var `NAME`. Throws if the variable is missing. |
|
|
228
|
+
| `config.<entry>` | Typed to the step's declared config entries (output view, defaults applied). From input.json's `$config`. An optional entry that is not set is `undefined`. |
|
|
176
229
|
| `log(message, fields?)` | Writes a `message k=v …` line to stderr. |
|
|
177
230
|
| `signal` | Aborted on SIGTERM/SIGINT. Running commands are sent SIGTERM. |
|
|
178
231
|
| `run` | `{ pipeline, id, number, attempt, step, version }` (from the env, see below). |
|
|
@@ -199,19 +252,21 @@ export default pipeline({ name: "Olve.Pipelines", description: "…", input: M.D
|
|
|
199
252
|
- **The `from` argument.** It is keyed by step name, plus `input` for the pipeline input. It holds `Ref<T>` values, so only property access and literals compile. Use `from["deploy-beta"].url` for names that contain hyphens.
|
|
200
253
|
- **Describe checks the mapper twice.** It records the mapper with proxies, and it statically analyses the mapper's source. The source must be an object literal of member chains, literals, constant arrays and nested object literals. Ternaries, `&&`, `||`, `??`, `!`, comparisons, arithmetic, template strings, calls, spread and optional chaining are all rejected. Free identifiers such as module constants are allowed; they are recorded as `{const}`.
|
|
201
254
|
- **Triggers.** A pipeline may call `onPush` at most once. Each `onPoll` and `onWebhook` needs a distinct `name` (`onPoll` defaults to `"poll"`). `onWebhook({ name, verify }, ({ headers, body }) => input | null)` maps a delivery to `/api/hooks/<pipeline id>/<name>`; `verify` is `hmacSha256({ secret, header, prefix? })`, `githubSignature(secret)` or `bearer(secret)`, and `null` ignores the delivery.
|
|
255
|
+
- **`select` (branch on config).** `.select(TARGET.prod, { ssh: (p, prod) => p.stage(…), inCluster: (p, prod) => p.stage(…) })`: the first argument references a `z.discriminatedUnion` in a config entry (`DiscriminatorOf<U>` is its one literal field; a union with no or several literal fields is a type error). Cases are exhaustive and checked at compile time; each gets the reference narrowed to its variant (`ConfigNode<VariantOf<U, D, V>>`), recorded as `{"$config": "target.prod", "$case": "target.prod.kind=ssh"}` and usable only inside that case. The manifest records `{select, discriminator, cases}`; the service resolves it per run from the pinned config snapshot (`when` is per run input). The same step name may appear in several cases (same output model; not two different custom steps, since handlers pair by name); a step every case has is definitely available afterwards, one only some have is maybe-available. The harness picks the variant of a same-named step from `OLVE_CASES` (a JSON array of the case conditions the step sits in, set by the service).
|
|
256
|
+
- **Retention.** `pipeline({ …, retention: { keepLatestRuns?, minRunAgeDays?, buildGraceDays? } })` overrides the instance's retention thresholds for this pipeline's runs and binaries, field by field (non-negative integers; unset = the instance's value). Recorded on the manifest root as `retention`; the service applies the active version's.
|
|
202
257
|
- **Failure handlers.** `.onFailure(handler, { steps? })` takes a step whose inputs are all `FailureContext`. When a run fails, the service runs each handler whose scope (default: any step) contains a failed step, once per failure: `run <handler>` in the handler's image with its declared secrets, from the run's pinned binary. It is untracked and best-effort — not an attempt of the run, never retried, and its own failure triggers nothing; its log is kept at `runs/<p>/<run>/_failure/<handler>/<id>/log`. `FailureContext` carries `pipeline {id, name}`, `run {id, number, address, versionStamp, trigger, url}`, `version {number, commit}`, the run's `input`, a one-line `reason`, and `failedSteps[]` (`step, attempt, address, reason, logUrl`) — only the steps in the handler's scope, never empty.
|
|
203
258
|
|
|
204
259
|
### Testing handlers
|
|
205
260
|
|
|
206
261
|
```ts
|
|
207
|
-
import { testHandler, testFileRef } from "@
|
|
262
|
+
import { testHandler, testFileRef } from "@olivervea/pipelines/testing";
|
|
208
263
|
import deployBeta from "../handlers/deploy-beta";
|
|
209
264
|
|
|
210
|
-
const r = await testHandler(deployBeta, { artifact }, { secrets: { SSH_PRIVATE_KEY: "k" }, ctx: { http: fakeHttp }, run: { number: 412 } });
|
|
265
|
+
const r = await testHandler(deployBeta, { artifact }, { secrets: { SSH_PRIVATE_KEY: "k" }, config: { target: { host: "h" } }, ctx: { http: fakeHttp }, run: { number: 412 } });
|
|
211
266
|
expect(r).toMatchObject({ status: "succeeded", output: { release: "olve-pipelines" } }); // r.filesDir, r.logs, r.exitCode
|
|
212
267
|
```
|
|
213
268
|
|
|
214
|
-
`testHandler` uses the same validation path as a real run. Invalid input gives `status: "failed"` with `kind: "input"`; an invalid return value gives `kind: "output"`. Any part of `ctx` can be replaced; whatever you don't replace is the real implementation. `testHandler`
|
|
269
|
+
`testHandler` uses the same validation path as a real run. Invalid input gives `status: "failed"` with `kind: "input"`; an invalid return value gives `kind: "output"`. Any part of `ctx` can be replaced; whatever you don't replace is the real implementation. `testHandler` takes any implementation: a handler module's default export, a library step or a reusable step.
|
|
215
270
|
|
|
216
271
|
## Harness contract (service ⇄ step binary)
|
|
217
272
|
|
|
@@ -235,7 +290,7 @@ This section is everything the C# side needs in order to run the binary.
|
|
|
235
290
|
|
|
236
291
|
| Path | Who | What |
|
|
237
292
|
|---|---|---|
|
|
238
|
-
| `input.json` | service (via olve-init) | **run**: a JSON object with exactly the step's input keys, built from the manifest's `input` mappings. Leave out a key whose ref root step was skipped; this is allowed only for `optional` inputs. Unknown keys → exit 65. **map-trigger push**:
|
|
293
|
+
| `input.json` | service (via olve-init) | **run**: a JSON object with exactly the step's input keys, built from the manifest's `input` mappings. Leave out a key whose ref root step was skipped; this is allowed only for `optional` inputs. Unknown keys → exit 65. For a step with `configEntries`, the reserved key `"$config"` holds `{ <entry>: <value> }` from the run's config (entries that are not set are left out); each entry is parsed with its schema, and a missing required entry or an invalid value → exit 65 (`config.<entry>…`). Other steps ignore `$config`. **map-trigger push**: `{"ref": "refs/heads/<branch>", "after": "<sha>", "repository": {"full_name": "<owner>/<repo>"}}` (`PushEvent`). **map-trigger poll**: `{"value": "<extracted value as string>"}`. **map-trigger webhook**: `{"headers": {…}, "body": …}` (`WebhookEvent`). |
|
|
239
294
|
| `work/output.json` | harness | The envelope. Any stale copy is deleted at start, and the new file is written atomically. |
|
|
240
295
|
| `work/files/<object path>` | harness | Output files and directory tars (see "File references"). |
|
|
241
296
|
| `tmp/` | harness | Scratch space. Point `TMPDIR` here. |
|
|
@@ -252,13 +307,15 @@ This section is everything the C# side needs in order to run the binary.
|
|
|
252
307
|
| `OLVE_RUN` | run | Run number (integer ≥ 0). Written into produced file refs. Default 0. |
|
|
253
308
|
| `OLVE_ATTEMPT` | run | Attempt number (integer ≥ 1). Written into produced file refs. Default 1. |
|
|
254
309
|
| `OLVE_VERSION_STAMP` | run | Derived version, e.g. `2026.10.21.412` → `ctx.run.version`. |
|
|
310
|
+
| `OLVE_CASES` | run | Only for a step inside a `select`: a JSON array of the case conditions it sits in (`["target.prod.kind=ssh"]`); `run <step>` uses it to pick the variant of a name several cases use. |
|
|
255
311
|
| `<SECRET_NAME>` | per step | Each declared secret as an env var of the **same name**: `secretKeyRef` key = name. Mount only the step's `secrets` (for poll triggers, the poll's `secrets`). Child processes inherit them. The mount is optional: an unset secret leaves the variable unset and `ctx.secrets.<NAME>` throws on access, so a handler can check `process.env.<NAME>` first and skip. |
|
|
256
312
|
| `OLVE_RESULT_URL` | optional | Pre-signed **PUT** for `result.tar.gz` (see below). Unset means no upload. |
|
|
313
|
+
| `OLVE_CALLBACK_URLS` | optional | JSON object mapping `<step>/<input>` → this run's callback URL of that external input (`ctx.inputUrl`). Set when the pipeline has `callback()` inputs and the instance has a public base URL. |
|
|
257
314
|
| `OLVE_FILE_URLS` | optional | JSON object mapping file key → pre-signed **GET** URL, one entry per file ref in the input. The key is `<run>/<step>/<attempt>/<path>` (`path` exactly as in the ref, including a trailing `/` for directories). The URL points at the stored object (see the storage rule). |
|
|
258
315
|
| `OLVE_FILES_BASE_URL` (+ `OLVE_FILES_TOKEN`) | optional | Fallback: `GET <base>/<run>/<step>/<attempt>/<object path>` with an optional `Authorization: Bearer <token>`. |
|
|
259
316
|
| `OLVE_FILES_DIR` | dev | Local pool laid out as `<run>/<step>/<attempt>/files/<object path>`. |
|
|
260
317
|
|
|
261
|
-
`OLVE_RESULT_URL`, `OLVE_FILE_URLS` and `
|
|
318
|
+
`OLVE_RESULT_URL`, `OLVE_FILE_URLS`, `OLVE_FILES_TOKEN` and `OLVE_CALLBACK_URLS` are removed from the environment of child processes. URLs are never logged past `?`.
|
|
262
319
|
|
|
263
320
|
`ctx.files.fetch` looks for a file in this order:
|
|
264
321
|
|
|
@@ -286,7 +343,7 @@ For `map-trigger`, `output` is the pipeline input after it has been parsed by th
|
|
|
286
343
|
| 0 | ok (succeeded or skipped) | yes |
|
|
287
344
|
| 1 | the handler or trigger mapper threw | failed / `handler` |
|
|
288
345
|
| 64 | usage: unknown mode, step or trigger | none |
|
|
289
|
-
| 65 | `input.json` missing, not JSON, or invalid for the step (or event) | failed / `input` |
|
|
346
|
+
| 65 | `input.json` missing, not JSON, or invalid for the step (or event), including its `$config` | failed / `input` |
|
|
290
347
|
| 66 | the return value failed strict validation, or an own file ref is missing or changed | failed / `output` |
|
|
291
348
|
| 74 | the `result.tar.gz` upload failed after retries. Used only when the step itself succeeded; otherwise the step's own code wins. | yes (printed) |
|
|
292
349
|
| 78 | the pipeline is invalid (describe problems, or a missing or wrong handler module) | none |
|
|
@@ -320,6 +377,7 @@ Exact format: [`manifest.schema.json`](manifest.schema.json) (JSON Schema 2020-1
|
|
|
320
377
|
- **`apiVersion`** is `"2.0"`.
|
|
321
378
|
- **`harness`** is `{ bun, sdk, sdkHash }`.
|
|
322
379
|
- **`models[name]`** is `{ output, input? }`.
|
|
380
|
+
- **Step `kind`** is `custom` (a handler file) or the kind of a step that carries its implementation (`imageBuild`, `composeDeploy`, a package's `acme/…`). To the service it is a label: any non-empty kind is accepted, so a new library or package step needs no service change.
|
|
323
381
|
- `output` is the `z.output` view, with `additionalProperties: false` on every object. It validates outputs, the pool and trigger results.
|
|
324
382
|
- `input` is the `z.input` view (defaults optional). Use it for trigger forms and manual input. It is emitted only when it differs from `output` beyond `additionalProperties`.
|
|
325
383
|
- **Step `input`** maps each input key to a mapping:
|
|
@@ -328,10 +386,11 @@ Exact format: [`manifest.schema.json`](manifest.schema.json) (JSON Schema 2020-1
|
|
|
328
386
|
- a nested object of mappings.
|
|
329
387
|
|
|
330
388
|
An object whose only key is `ref` (with a string value) or `const` is a tag. Describe never emits a nested object with exactly that key set. Model-matched inputs are emitted already resolved (`{"ref": "build"}`).
|
|
389
|
+
- **Step `externalInputs`** (when present) maps each external input key to `{model, title, description, from}`, `from` being `{kind: "person"}`, `{kind: "callback"}` or `{kind: "poll", url, valuePath, intervalSeconds}`. Those keys are not in `input` or `inputTypes`; the service merges the answers into `input.json`.
|
|
331
390
|
- **`inputTypes`** declares each input key: `{model, optional}`, or `{schema: {output, input?}}` for a plain schema.
|
|
332
|
-
- **`handlerHash`** is the step's implementation identity. A step version can be computed from image + `inputTypes`/model versions + `output` model version + `secrets` + `input` mappings + `handlerHash`. Stage placement, `when` and the step's `description` are not part of it.
|
|
333
|
-
- Custom steps: `sha256(canonical {bundle, harness})`. `bundle` hashes the handler bundled on its own, with `pipeline.ts`, `models.ts` and `@
|
|
334
|
-
-
|
|
391
|
+
- **`handlerHash`** is the step's implementation identity. A step version can be computed from image + `inputTypes`/model versions + `output` model version + `secrets` + `configEntries` (and those entries' schemas) + `input` mappings + `handlerHash`. Stage placement, `when` and the step's `description` are not part of it.
|
|
392
|
+
- Custom steps: `sha256(canonical {bundle, harness})`. `bundle` hashes the handler bundled on its own, with `pipeline.ts`, `models.ts` and `@olivervea/pipelines` external. It is tree-shaken and includes embedded scripts, so it changes only when that handler's code (or code it uses) changes.
|
|
393
|
+
- Steps that carry their implementation: `sha256(canonical {library: kind, config, harness, code})`, `code` being the hash of the modules that define and implement steps of that kind (see [Reusable steps](#reusable-steps)).
|
|
335
394
|
- Because `harness` is part of both hashes, a Bun or SDK bump changes every handler.
|
|
336
395
|
- **`stages`** is an array whose items are either `{steps: [Step…]}` (parallel, with a barrier) or `{when: {field: [values]}, stages: [...]}`.
|
|
337
396
|
- **`triggers`**:
|
|
@@ -339,18 +398,22 @@ Exact format: [`manifest.schema.json`](manifest.schema.json) (JSON Schema 2020-1
|
|
|
339
398
|
- `{kind: "poll", name, url, headers, secrets, valuePath, intervalSeconds}`, where header values may contain `$SECRET:NAME`;
|
|
340
399
|
- `{kind: "webhook", name, verify: {scheme: "hmac-sha256", secret, header, prefix} | {scheme: "bearer", secret}}`. The `map-trigger` event is `{headers, body}`; a `null` mapping yields a `skipped` envelope (the delivery is ignored).
|
|
341
400
|
- **`failureHandlers`** is a list of `{step: Step, steps: [names] | null}`. Each handler's inputs are all FailureContext (`{"ref": "failure"}`).
|
|
401
|
+
- **`retention`** (when present) is `{keepLatestRuns?, minRunAgeDays?, buildGraceDays?}`, non-negative integers that replace the instance's thresholds for this pipeline. Omitted when the definition sets none.
|
|
342
402
|
- **`secrets`** lists every secret used by steps, polls and webhooks, sorted, as `{name, description}`.
|
|
403
|
+
- **`config`** lists every config entry steps declare, sorted by name, as `{name, description, required, schema}`. `schema` is the `z.input` view (defaults listed); it validates the entry's value in the config document. `required` is false when the schema accepts a missing value. A `secretRef` field is marked `"x-olve-kind": "secretRef"`, with the secret's name as `properties.secret.const`. Omitted when there are no entries.
|
|
404
|
+
- **Step `configEntries`** names the entries the step receives (sorted; omitted when none). It is part of the step's content, like `secrets`. The step key `config` is the step's `settings`.
|
|
343
405
|
|
|
344
406
|
Describe runs these checks; reconcile should re-run the manifest-level ones:
|
|
345
407
|
|
|
346
408
|
- step name format and reserved names (`input`, `failure`), and duplicate step names;
|
|
347
409
|
- one schema per model name, and JSON-representable models;
|
|
348
|
-
- missing handler file,
|
|
410
|
+
- missing handler file, a handler file without a step (or for a step that carries its implementation), and a step that carries its implementation without a kind;
|
|
349
411
|
- availability, ambiguity, maybe-available as required, `optional` without a producer, plain schema without a mapper;
|
|
350
412
|
- mapper source analysis and recording, unknown fields in refs, mapper keys that are missing or extra;
|
|
351
413
|
- `when` on an unknown or non-literal field, or with an invalid value;
|
|
352
414
|
- a poll header referencing an undeclared secret, duplicate triggers;
|
|
353
415
|
- conflicting secret descriptions;
|
|
416
|
+
- config names, one description and schema per config name, a step receiving a `secretRef` without declaring its secret, and input keys starting with `$` (reserved);
|
|
354
417
|
- an `onFailure` scope naming an unknown step, and failure-handler inputs.
|
|
355
418
|
|
|
356
419
|
## Build tool
|
|
@@ -360,13 +423,13 @@ olve-build <pipelineDir> --out <dir> [--target bun-linux-x64|bun-linux-x64-musl]
|
|
|
360
423
|
olve-build <pipelineDir> --hash-only # print handler hashes (diagnostics)
|
|
361
424
|
```
|
|
362
425
|
|
|
363
|
-
1.
|
|
426
|
+
1. Pair each `handlers/*.ts` with the step its `implement(step, …)` names, and hash it (see `handlerHash`); hash the modules of each kind of step that carries its implementation.
|
|
364
427
|
2. Generate `<pipelineDir>/.olve/gen/handlers.ts` (a registry of literal `import()` paths, plus hashes and harness info) and `entry.ts`. Add `.olve/` to `.gitignore`.
|
|
365
428
|
3. Run `bun build --compile --target=bun-linux-x64 --no-compile-autoload-dotenv --no-compile-autoload-bunfig --bytecode --format=esm` → `<out>/step`. The output is never minified, because mapper source analysis needs real source.
|
|
366
429
|
4. Check that `PT_INTERP` is `/lib64/ld-linux-x86-64.so.2`. On alpine, Bun silently produces a musl binary; the check fails it.
|
|
367
430
|
5. Run `<out>/step describe` → `<out>/manifest.json`. A describe problem is a build failure with exit 78. **A custom step without a handler file is an error.**
|
|
368
|
-
6. Write `<out>/step.tar.
|
|
369
|
-
7. With `--upload-url`, PUT `step.tar.
|
|
431
|
+
6. Write `<out>/step.tar.zst` (a deterministic tar, `step` at the root, mode 0755, zstd level 3: it unpacks in the step pod far faster than gzip; harness-base also still unpacks the gzip archives of older SDKs; from 2.0.0-beta.7 the SDK needs an instance whose harness-base unpacks zstd, i.e. one deployed on or after 2026-10-25) and `<out>/build.json` (diagnostics).
|
|
432
|
+
7. With `--upload-url`, PUT `step.tar.zst`, using the same retry rules as the result upload. With `--sentinels`, print the manifest between the manifest sentinels.
|
|
370
433
|
|
|
371
434
|
Exit codes: 0 ok, 1 build failure, 64 usage, 78 pipeline invalid.
|
|
372
435
|
|
|
@@ -378,11 +441,11 @@ bun install --frozen-lockfile
|
|
|
378
441
|
bunx olve-build .pipelines --out /work/out --upload-url "$STEP_PUT_URL" --sentinels
|
|
379
442
|
```
|
|
380
443
|
|
|
381
|
-
From npm
|
|
444
|
+
From npm, nothing else is needed. A local `file:`/`link:` dependency (`"@olivervea/pipelines": "file:../src/sdk"`, as Olve.Pipelines' own `.pipelines/` does) resolves to its **source directory**, not the copy in `node_modules`, so its own dependencies must be installed there too: the build Job runs `bun install --frozen-lockfile --production` in every local dependency of the binding's `package.json` that is inside the repo, before building.
|
|
382
445
|
|
|
383
|
-
The repo depends on the SDK as a normal package. It resolves through `exports` (`@
|
|
446
|
+
The repo depends on the SDK as a normal package. It resolves through `exports` (`@olivervea/pipelines`, `@olivervea/pipelines/harness`, `@olivervea/pipelines/testing`), and the tests cover this with a `file:` install. `bun` must be on `PATH`, because the bins use `#!/usr/bin/env bun`. A consumer `tsconfig.json` has to include the dot-directory explicitly (`"include": [".pipelines/**/*.ts"]`), and should exclude `.pipelines/.olve`.
|
|
384
447
|
|
|
385
|
-
How the service runs it (reconcile, `BuildJobBuilder`): one Job per (pipeline, config commit), named `olve-build-<pipeline12>-<sha12>`. The repo tarball comes from `codeload.github.com/<repo>/tar.gz/<sha>` anonymously (public repos), or from the GitHub API tarball endpoint with the binding's credentials key from the pipeline Secret. The
|
|
448
|
+
How the service runs it (reconcile, `BuildJobBuilder`): one Job per (pipeline, config commit), named `olve-build-<pipeline12>-<sha12>`. The repo tarball comes from `codeload.github.com/<repo>/tar.gz/<sha>` anonymously (public repos), or from the GitHub API tarball endpoint with the binding's credentials key from the pipeline Secret. The archive is PUT to `versions/<pipeline>/builds/<sha>/step.tar.gz` (the key predates zstd; the init container tells the formats apart by content), and a version added from that build keeps this key as its `BinaryKey`. The in-repo fixture `.pipelines-test/` (bound on v2 beta) uses exactly this layout.
|
|
386
449
|
|
|
387
450
|
The manifest is the text between the sentinels in the pod log. The manifest text is identical across checkouts. The binary and archive bytes are identical for the same checkout layout.
|
|
388
451
|
|
|
@@ -390,11 +453,11 @@ The manifest is the text between the sentinels in the pod log. The manifest text
|
|
|
390
453
|
|
|
391
454
|
```
|
|
392
455
|
olve-dev describe <dir> [--sentinels] # same bytes as the compiled binary's describe
|
|
393
|
-
olve-dev run <dir> <step> [--input-file f] [--olve-dir d] [--secret NAME=value]...
|
|
456
|
+
olve-dev run <dir> <step> [--input-file f] [--config-file f] [--olve-dir d] [--secret NAME=value]...
|
|
394
457
|
olve-dev map-trigger <dir> <trigger> --event-file f [--olve-dir d]
|
|
395
458
|
```
|
|
396
459
|
|
|
397
|
-
`olve-dev` runs handlers from source, through the same validation path as the binary. `$OLVE_DIR` defaults to `<dir>/.olve`, and outputs land in `<dir>/.olve/work/`. Set `OLVE_FILES_DIR` to a local pool to fetch file refs.
|
|
460
|
+
`olve-dev` runs handlers from source, through the same validation path as the binary. `$OLVE_DIR` defaults to `<dir>/.olve`, and outputs land in `<dir>/.olve/work/`. Set `OLVE_FILES_DIR` to a local pool to fetch file refs. `--config-file` (a JSON object of config entries) becomes the input's `$config`.
|
|
398
461
|
|
|
399
462
|
## Development
|
|
400
463
|
|