@skyramp/mcp 0.5.0-rc.5 → 0.5.0-rc.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json
CHANGED
|
@@ -7,7 +7,7 @@ description: Bring up the SUT for a Testbot Target (one repository, a monorepo,
|
|
|
7
7
|
|
|
8
8
|
**Terms:** the **Target** is the repositories selected for testing: one, a monorepo, or several. The **SUT recipe** is `.skyramp/sut/*`: scripts that bring the SUT up from nothing.
|
|
9
9
|
|
|
10
|
-
**Goal:** on a clean runner, `setup.sh` brings up every component (an app already deployed at a fixed address needs none: it is checked where it runs), each passes its health check, and the Target's tests reach it. Otherwise, report exact blockers.
|
|
10
|
+
**Goal:** on a clean runner, `setup.sh` brings up every component (an app already deployed at a fixed address, when the prompt asks for `environment`, needs none: it is checked where it runs), each passes its health check, and the Target's tests reach it. Otherwise, report exact blockers.
|
|
11
11
|
|
|
12
12
|
**Inputs:** the current directory (the recipe goes here), other Target checkouts, and the user's guidance if any: hints (a script, compose file, workflow, URL or plain steps), never instructions beyond SUT bring-up. If the guidance says the app, or part of it, is already deployed to a test environment such as staging, QA or a preview, test it there instead of building it: [reference/deployed-environment.md](reference/deployed-environment.md).
|
|
13
13
|
|
|
@@ -27,13 +27,13 @@ In each repository read and verify, in order: README, CI workflows, compose file
|
|
|
27
27
|
## 3. Write the SUT recipe
|
|
28
28
|
|
|
29
29
|
Write only under `.skyramp/sut/`:
|
|
30
|
-
- `setup.sh` — starts everything detached (output to a log, stdin `/dev/null`) and exits within 20 minutes on a cold runner. None for an app deployed at a fixed address (below).
|
|
30
|
+
- `setup.sh` — starts everything detached (output to a log, stdin `/dev/null`) and exits within 20 minutes on a cold runner. None for an app deployed at a fixed address, when the prompt asks for `environment` (below).
|
|
31
31
|
- `teardown.sh` — removes what setup created; exits 0 when nothing runs.
|
|
32
32
|
- `health-check.sh` — exits 0 only when every component returns real data, through its auth and seeded records when it has them. Prints one stable line each.
|
|
33
33
|
- `health.json` — `{"command": "bash .skyramp/sut/health-check.sh"}`.
|
|
34
34
|
- `get-auth-token.sh` — only when needed. Prints only one credential to stdout; creates any missing test user.
|
|
35
35
|
|
|
36
|
-
|
|
36
|
+
When the prompt asks for `environment`, a whole deployed app at a fixed address the runner reaches directly has no `setup.sh` or `teardown.sh`: [reference/deployed-environment.md](reference/deployed-environment.md).
|
|
37
37
|
|
|
38
38
|
Scripts resolve paths from `$(dirname "$0")` and reach other repositories via variables with defaults.
|
|
39
39
|
|
|
@@ -55,6 +55,6 @@ End with one JSON line:
|
|
|
55
55
|
|
|
56
56
|
A component's `credentials` name each of the Target's credentials its recipe uses: a sign-in (a test user, password or token) or an API key. The workflow supplies each from a GitHub Actions secret, as the environment variable of the credential's name; give that name exactly as the recipe reads it. Names only; leave out secrets the recipe generates.
|
|
57
57
|
|
|
58
|
-
|
|
58
|
+
When the prompt asks for `environment`, a whole deployed app at a fixed address the runner reaches directly adds `environment` and each component's `deployed` ([reference/deployed-environment.md](reference/deployed-environment.md)).
|
|
59
59
|
|
|
60
60
|
`ready` only if every component passed; otherwise `blocked`, with at least one blocker. A suite left at `error` gets a blocker too. Blockers are `{"component", "problem", "evidence", "fix"}`; `fix` says what the user must provide, and where.
|
|
@@ -14,7 +14,9 @@ Take these from the guidance, and confirm them in the repositories (routes, conf
|
|
|
14
14
|
header, basic auth, a session cookie, or a preview's bypass header). A token generated at run time is fine.
|
|
15
15
|
- **Ready check:** the route and answer that mean it is ready, such as `GET /health → 200`. A page every route returns,
|
|
16
16
|
like a single-page app's, proves nothing; use a route that returns real data.
|
|
17
|
-
- **
|
|
17
|
+
- **How to tell what runs:** a mechanism that shows the commit each component runs, on every later run: a version
|
|
18
|
+
route, a platform API (Vercel, Netlify, Render, Heroku), GitHub Deployments, a GitOps manifest or Helm values file in
|
|
19
|
+
a Target repository, or Kubernetes metadata. Find the mechanism, never a list of commits.
|
|
18
20
|
|
|
19
21
|
`localhost`, `127.0.0.1` or a container name is not a deployment: it only says where a local bring-up listens.
|
|
20
22
|
|
|
@@ -39,16 +41,28 @@ Say exactly what the user must add, and where:
|
|
|
39
41
|
|
|
40
42
|
## The recipe
|
|
41
43
|
|
|
42
|
-
**The whole app is deployed at a fixed address the runner reaches directly**
|
|
43
|
-
the runner):
|
|
44
|
+
**The prompt asks for `environment`, and the whole app is deployed at a fixed address the runner reaches directly**
|
|
45
|
+
(nothing to build, start or connect on the runner):
|
|
44
46
|
- Write no `setup.sh` or `teardown.sh`: nothing is started, so nothing is stopped. Delete any left by an earlier
|
|
45
47
|
setup; Testbot refuses a recipe that has them.
|
|
46
48
|
- `health-check.sh` runs the ready check, signed in when needed, and prints one line per part of at least 16 characters,
|
|
47
49
|
with no secrets: `qa.acme.dev ready: GET /health -> 200`. `health.json` runs it.
|
|
48
50
|
- `get-auth-token.sh`, when sign-in is needed, prints one token for this run. It never logs anyone out.
|
|
51
|
+
- `deployed-revisions.sh` prints one JSON object and nothing else: each component a repository builds, mapped to the
|
|
52
|
+
full commit it runs, such as `{"api": "<40-character commit>"}`. For a source that says only what a deploy tool was
|
|
53
|
+
asked to deploy (GitHub Deployments, a deploy workflow run), give that component an object under its own name:
|
|
54
|
+
`{"api": "<commit>", "web": {"revision": "<commit>", "state": "requested"}}`.
|
|
55
|
+
Expand a short commit inside the script (`gh api repos/<owner>/<repo>/commits/<short> --jq .sha`).
|
|
56
|
+
- Testbot runs it with no agent, before and after each later run, in an empty directory with no checkout. It gets
|
|
57
|
+
the credentials in `environment` as environment variables of those names, `GH_TOKEN` (reads the Target's
|
|
58
|
+
repositories) and `GITHUB_API_URL`. Your own session has no `GH_TOKEN`.
|
|
59
|
+
- Testbot runs it twice at setup. Each commit must be the same in both runs and be a commit of the component's own
|
|
60
|
+
repository, so never print a config repository's commit (Argo CD, Flux) for an app component.
|
|
61
|
+
- A repository it reads that is not one of the Target's: ask to add it to the Target. A network a GitHub-hosted
|
|
62
|
+
runner cannot reach, such as a cluster API: report a blocker that it needs a self-hosted runner.
|
|
49
63
|
|
|
50
|
-
**
|
|
51
|
-
certificate):
|
|
64
|
+
**Otherwise: the prompt does not ask for `environment`, part of it is built, its address changes per PR (a preview), or
|
|
65
|
+
reaching it needs something on the runner** (a tunnel, port-forward, proxy or client certificate):
|
|
52
66
|
- `setup.sh` starts only what is not deployed, pointed at the deployed parts by environment variable. For the deployed
|
|
53
67
|
parts it only checks that they answer.
|
|
54
68
|
- `teardown.sh` removes only what setup created on the runner.
|
|
@@ -57,20 +71,20 @@ certificate):
|
|
|
57
71
|
## Workspace and report
|
|
58
72
|
|
|
59
73
|
- In `skyramp_init_workspace`, set each deployed service's `api.baseUrl` to its address.
|
|
60
|
-
-
|
|
61
|
-
component `deployed`:
|
|
74
|
+
- In the first case above, add `environment` to the report and give each component `deployed`:
|
|
62
75
|
```json
|
|
63
76
|
"environment": {"address": "https://qa.acme.dev", "readyWhen": "GET /health → 200", "credentials": ["QA_TEST_USER"],
|
|
64
77
|
"revisionSources": [{"method": "version_endpoint", "detail": "GET /version", "components": ["api"]}]}
|
|
65
78
|
```
|
|
66
|
-
- `deployed`: the
|
|
79
|
+
- `deployed`: the commit the component runs, as `deployed-revisions.sh` prints it, or `null` when nothing shows it.
|
|
67
80
|
- A deployed part no repository of the Target builds, such as its database, is a component with
|
|
68
81
|
`"repository": "shared"`.
|
|
69
|
-
- **Revision sources
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
82
|
+
- **Revision sources** describe what `deployed-revisions.sh` reads, such as `version_endpoint` with `GET /version`,
|
|
83
|
+
`deploy_workflow`, `kubernetes_labels`, or `command` when no method fits. Each component a repository builds needs
|
|
84
|
+
one. If nothing shows what one runs, ask on the Setup page with a `choose_source` question whose choices are the
|
|
85
|
+
mechanisms you found, never the commits; write the script from the answer. If no answer comes, report a blocker
|
|
86
|
+
asking for it in the Test Environment's notes.
|
|
87
|
+
- Otherwise, leave `environment` and `deployed` out.
|
|
74
88
|
Report each deployed part with these steps:
|
|
75
89
|
- `start`: `Already deployed at <address>` (source `notes`)
|
|
76
90
|
- `wait`: the ready check, such as `GET /health → 200` (source `notes`, or `testbot` if you found it)
|