@yunuseon/kayra 0.12.0 → 0.13.0

package/agent/skills/kayra-lab/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: kayra-lab
+description: Run kayra's upgrade lab (test/lab/upgrade.sh) - a throwaway Multipass server brought up at the last release, upgraded to the kayra checkout, and taken through every deploy path - and read its result. Use this before merging any change to kayra's platform (ansible/, compose/, scripts/, cli/src/deployer/, terraform/), after a release to prove the full path from it, whenever the user asks to "test this on a VM", "run the lab", "does the upgrade work", or when a bug was found on a real server and needs a reproduction.
+---
+
+# The upgrade lab
+
+What broke on kayra's real servers broke on **upgrades**, never on the fresh
+VMs it used to be tested on: Authelia kept rules it had never loaded, an app
+was left unregistered, every image was rebuilt on every run. The lab
+reproduces the upgrade path end to end:
+
+1. A temp fleet pinned to the last release (or `--from <version>`) brings a
+   Multipass server up and deploys a fixture app (`test/lab/app`) by a
+   signed webhook.
+2. The fleet's kayra becomes this checkout; `kayra deploy` (which ends with
+   `kayra verify`).
+3. A new commit by webhook; the same commit again must be `CACHED`.
+4. A commit the compose policy refuses: the run fails naming the bind mount,
+   and the old release keeps serving.
+5. `kayra app rollback` to the first release, the registry GC, `kayra verify`.
+
+## Before you start
+
+- Run it from a kayra checkout, or a worktree of one, that holds the change
+  under test. The lab links the fleet to **that directory's working tree**,
+  uncommitted changes included: say which commit and which changes it tests.
+- `multipass list` shows no `kayra-lab` VM. A kept one from an earlier
+  failure is torn down first (its command was printed at the end of that run).
+- Multipass, mkcert, OpenTofu, Ansible, Node 24 and python3 on the Mac.
+
+## Running it
+
+About 15 minutes. Run it in the background with its output in a log, and
+follow the phases rather than waiting blind:
+
+```bash
+bash test/lab/upgrade.sh [--from <version>] [--keep] > <log> 2>&1; echo "lab exit $?" >> <log>
+```
+
+Watch for lines starting `── ` (a phase), `✓` and `✗`, `kept for`, and
+`lab exit`. Keep one watcher per log: two watchers on the same file report
+every event twice. A release older than the one that allowed `http://` app
+repos on multipass prints "cannot clone over http://: the app arrives with
+the upgrade". The release then sets up only the platform, which is expected
+for that range.
+
+## When it fails
+
+A failure keeps everything, the VM (`kayra-lab`), the temp fleet and the log,
+and prints where. Look before you change anything:
+
+- The lab's own log: `grep -nE 'fatal:|FAILED|✗|kayra:|rror' <lab log>`.
+- What the server did: from the kept fleet directory,
+  `node node_modules/@yunuseon/kayra/bin/kayra.js app runs lab sample` and
+  `… app logs lab sample`.
+- Then decide: **a kayra bug or a lab bug?** Most first-run failures were the
+  lab's. Each was fixed in the lab, and twice the fix showed something kayra
+  should handle:
+  - git's dumb HTTP cannot serve the shallow clones kayra makes, so the lab
+    uses smart HTTP (`test/lab/git-http.py`);
+  - a repo URL without `owner/name` matches no forge, so its webhooks were
+    refused;
+  - a bind mount appended to the end of the compose file landed under
+    `networks:`, so compose rejected the file before the policy saw it;
+  - `verify` did not yet understand an app registered but never deployed:
+    that was a gap in kayra, fixed there, with a case added.
+- Tear the kept VM down with the printed command before the next run.
+
+## After a pass
+
+The VM and the temp fleet are destroyed unless `--keep`. Report the five
+phases and the final `kayra verify`, and what the run covered. A bug found on
+a real server, which the lab did not catch, gets a step here: the lab is
+only as good as the history it encodes.

package/agent/skills/kayra-release/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: kayra-release
+description: Cut a release of kayra (the npm package @yunuseon/kayra) - version bump PR, checks, merge, tag vX.Y.Z, the release workflow publishing to npmjs.org, and checking the published package. Use this whenever kayra changes are merged and need to reach fleets, when the user says "release", "cut a release", "tag it", "publish kayra", "ship 0.x", or asks how to get merged kayra work onto servers - a fleet only gets kayra changes through a release.
+---
+
+# Releasing kayra
+
+A fleet runs the kayra version its `package.json` pins, from npmjs.org, so
+merged work reaches no server until it is released. A release is a version
+bump on `main`, a tag `v<version>` on the bump's merge commit, and
+`.github/workflows/release.yml`, which publishes that tag to npm.
+
+## Before anything: don't touch the user's checkout
+
+The user works in the same checkouts in parallel: their own branch,
+uncommitted changes, sometimes mid-task. Do all of this in a **separate git
+worktree off `origin/main`**, and run every step with absolute paths inside
+it. Write the steps as one script under `set -euo pipefail` with one command
+per line. A guard such as `[ -z "$(git status --short)" ] && …` protects only
+what is inside its `&&` chain: anything after a `;` runs regardless, and that
+is how writes once landed in the user's working tree. If a step needs the
+user's checkout (it rarely does here), stop and ask.
+
+## 1. What goes into this release
+
+- `git log --oneline v<last>..origin/main` and the merged PRs since the last tag.
+- Check that every PR you count is really on `main`: a PR stacked on another
+  branch merges into **that** branch (`gh pr view <n> --json baseRefName`).
+  It happened once: the PR was merged, its work never reached `main`.
+- **Stop when there is nothing to release.** If the log since the last tag is
+  empty, or a PR you were told is merged is still open (`gh pr view <n> --json
+  state`), the release would be empty: stop and say so, even when asked to
+  release. Make the script fail here (`test -n "$(git log …)"`, `test "$state"
+  = MERGED`), not print and go on: once a run printed `#41 OPEN` and went on to
+  open an empty release PR.
+- Pick the version (0.x):
+  - **patch** for fixes;
+  - **minor** for a new command, a new setting, a behaviour change (for example
+    `deploy` newly ending with `verify`), and anything breaking for fleets.
+  - A breaking change lists exactly what a fleet must change, in the release
+    commit and the PR. Fleets read it when they bump.
+
+## 2. The bump PR
+
+In the worktree:
+
+```bash
+git -C "$W" switch -c release-<version>
+npm --prefix "$W" version <version> --no-git-tag-version   # package.json + package-lock.json only
+git -C "$W" add package.json package-lock.json
+git -C "$W" commit -s -F <message file>                      # DCO sign-off; the message summarises the PRs
+git -C "$W" push -u origin release-<version>
+gh pr create --repo yunuseon/kayra --base main --head release-<version> --title "<version>" --body-file <file>
+gh pr checks release-<version> --repo yunuseon/kayra --watch --interval 20
+```
+
+Write commit messages and PR bodies to a file first (quoted heredoc):
+backticks inside `-m "…"` or a double-quoted `--body` are executed by the
+shell.
+
+## 3. Merge and tag
+
+`main` is protected: it changes only through a PR with green `checks`, for
+admins too. Merging and pushing a tag both ask the operator (kayra's guard);
+that is expected, say what you are doing.
+
+```bash
+gh pr merge <n> --repo yunuseon/kayra --merge --delete-branch
+merge="$(gh pr view <n> --repo yunuseon/kayra --json mergeCommit --jq .mergeCommit.oid)"
+git -C "$W" fetch -q origin && git -C "$W" checkout -q "$merge"
+grep '"version"' "$W/package.json"          # must be the new version
+git -C "$W" tag v<version>                  # lightweight, on the merge commit, like every earlier tag
+git -C "$W" push origin v<version>
+```
+
+Release tags are protected (a ruleset): once pushed, a tag cannot be moved or
+deleted. A wrong tag is fixed by the next patch release, not by retagging.
+So check the commit before you tag it.
+
+## 4. Watch it publish
+
+```bash
+run="$(gh run list --repo yunuseon/kayra --workflow release.yml --limit 1 --json databaseId --jq '.[0].databaseId')"
+gh run watch "$run" --repo yunuseon/kayra --exit-status
+npm view @yunuseon/kayra@<version> version   # may lag a minute after the run: retry for ~2 minutes
+```
+
+Then look inside the published package for what this release added, not
+just at its version. The fleets get the package, not the repo, and only the
+paths in `package.json`'s `files` ship:
+
+```bash
+d="$(mktemp -d)" && (cd "$d" && npm pack @yunuseon/kayra@<version> --silent && tar -tzf *.tgz | grep -E '<new files>')
+```
+
+Test-only paths (`test/`, `ci/`) are meant to stay out.
+
+## 5. Afterwards
+
+- Remove the worktree and the local branch.
+- The release reaches servers through each fleet: that is the **kayra-rollout**
+  skill. Say so, and say whether servers need a deploy at all (a release that
+  only changes the CLI or the agent tooling changes nothing on servers).

package/agent/skills/kayra-rollout/SKILL.md

@@ -0,0 +1,92 @@
+---
+name: kayra-rollout
+description: Bring a kayra fleet's servers to a new kayra release - the version bump PR in each fleet (in a separate git worktree), its checks, the merge, then kayra deploy server by server in a safe order with kayra verify after each. Use this whenever a kayra release exists that the servers don't run yet, when the user says "roll it out", "bump the fleets", "deploy the new kayra", "update the servers", "Renovate opened a kayra bump", or after the kayra-release skill finishes.
+---
+
+# Rolling kayra out to the fleets
+
+A fleet pins kayra in its `package.json`; its servers run what that pinned
+kayra copies to them. Rolling out means: bump the pin in the fleet, merge,
+then deploy each server, least important first, checking each before the
+next.
+
+## The rule that matters most: separate worktrees
+
+The user works in the same checkouts in parallel: their own branch and
+uncommitted changes. Make every change in a **separate git worktree off
+`origin/main`**, with absolute paths, as one script under
+`set -euo pipefail`, one command per line. Never let a step that writes run
+after `;` behind a guard: `[ -z "$(git status --short)" ] && git switch -c x;
+npm install …` ran the `npm install` in the user's tree. Record the user's
+`git status --short` before and check it is unchanged at the end.
+
+Deploying is different: `require-main.sh` refuses a real server unless **the
+fleet's checkout** is a clean `main` equal to `origin/main`, with the pinned
+kayra installed (not a link). If the user's checkout isn't (their branch,
+their changes), don't switch it: tell them, and deploy when they say it is
+on `main`.
+
+## 1. Which fleets, which servers, which order
+
+- The fleets are the repositories that pin `@yunuseon/kayra`. Their own
+  CLAUDE.md or README names them, how their PRs work (GitHub `gh`, Forgejo
+  `tea`), and the rollout order. Read that and follow it.
+- Without an order given: multipass and test servers first, then home
+  servers, then real ones (hetzner), the least important first. Platform
+  servers get `kayra deploy`; edge and NAS servers get `kayra setup`.
+- Read the release notes between the fleet's version and the new one. A
+  breaking release says exactly what a fleet must change (settings renamed or
+  removed, new secrets): those changes go into the same bump PR.
+
+## 2. The bump PR, per fleet
+
+In a worktree of the fleet:
+
+```bash
+npm install --prefix "$W" --silent -D -E @yunuseon/kayra@<version>
+# …the fleet changes the release asks for (vault edits: decrypt in memory, never print a value)
+cd "$W" && kayra validate
+```
+
+Run the fleet's checks the way its CI does, against **tracked files only**
+(untracked local folders such as a lab server must not count): copy
+`git ls-files` plus `node_modules` into a temp context and
+`docker build -f <kayra checkout>/ci/checks/Dockerfile --build-context fleet=<context> --build-arg CHECKS="--fleet /fleet" <kayra checkout>`,
+with the kayra checkout at the released tag.
+
+Commit (the message names the release and anything the fleet changed),
+push, open the PR the fleet's way, wait for its checks (GitHub:
+`gh pr checks --watch`; Forgejo: `tea api /repos/<owner>/<repo>/commits/<sha>/status`
+until `success`), then merge. Merges ask the operator (kayra's guard); that
+is expected. Remove the worktree and the local branch.
+
+## 3. Deploy, server by server
+
+From the fleet's own checkout once it is on the merged `main` (`git pull`,
+then `npm ci`, so `node_modules` is the released version):
+
+- `kayra deploy <platform server>`, **not** `kayra setup`: a new version may
+  change the app role, which only `apps` runs. `deploy` ends with
+  `kayra verify` and fails if the server is broken. Read every `✗`.
+- `kayra setup <edge or nas server>`. A NAS's SSH may be on only on demand:
+  if it refuses the connection, ask the user to switch it on. Don't retry in
+  a loop.
+- One server at a time, in order. Stop at the first failure, find the cause,
+  and fix it in kayra if it is kayra's, before touching the next server.
+
+Things that have bitten before:
+- **A new platform hostname needs DNS before the deploy.** Traefik asks
+  Let's Encrypt once; a name without DNS keeps Traefik's default certificate
+  until Traefik restarts. `verify` reports it as `certificate by TRAEFIK
+  DEFAULT CERT`.
+- **A Docker upgrade by the docker role restarts every container.** That's
+  normal; check the server's public routes afterwards.
+- **Changed platform config on an upgraded server:** `verify` catches what
+  did not reload (a 403 from Authelia meant rules it had never loaded).
+
+## 4. Report
+
+Per server: the version it runs now, the deploy's recap (nothing failed?),
+the `kayra verify` result, and anything left for the user (a DNS apply, a
+switch in a vendor UI, a manual cleanup). Then whether the user's checkouts
+are as they left them.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yunuseon/kayra",
-  "version": "0.12.0",
+  "version": "0.13.0",
   "description": "A self-hosted app platform and the CLI that runs it: Traefik, Authelia, a registry, rootless BuildKit and a small deployer per server, set up with OpenTofu and Ansible.",
   "license": "MIT",
   "author": "Yunus Öztürk",
@@ -22,6 +22,7 @@
     "compose/",
     "terraform/",
     "scripts/",
+    "agent/",
     "cloud-init.yml.tpl",
     "hosts.conf",
     "schema/"