@evident-ai/runner-synchroniser 0.1.0

package/README.md

@@ -0,0 +1,185 @@
+# runner-synchroniser
+
+Keeps an agent runner's model credentials alive across task replacement.
+
+The Evident runner is a container that can be stopped and replaced at any time — it
+scales to zero when idle, and its host task is disposable. Anything that lives only on
+its disk is lost with it, including the two things that are expensive to recreate: the
+credentials a human bootstrapped interactively (a Claude subscription login, an OpenCode
+provider auth) and the agent's session history.
+
+This package is the credential half of that durability story: it restores both
+credential stores from an object store at boot, uploads them again whenever they change,
+and answers the "does this runner have any way to authenticate a model?" question that
+the boot script gates on. It also renders the `litestream.yml` that covers the session
+database, so both halves read their bucket, prefix and paths from one config module
+instead of two.
+
+It is a CLI, not a library: the runner's `entrypoint.sh` invokes it as a single
+self-contained bundle (`node cli.js <command>`).
+
+## Install
+
+The package is published to npm as `@evident-ai/runner-synchroniser`, exposing one
+`runner-synchroniser` binary and requiring Node >= 22:
+
+```bash
+npx @evident-ai/runner-synchroniser@dev env
+```
+
+`dist/cli.js` is a single self-contained ESM bundle with **no runtime dependencies** —
+the AWS SDK is bundled at build time, so installing it pulls nothing else in.
+
+> Publishing is being rolled out (#612). The `dev` tag is published on every merge to
+> `main` that touches this package; the first publish is a human bootstrap (#694), and
+> the `latest`-on-release path is not wired up yet. Nothing in this repo installs the
+> package: the runner image and the MicroVM still build the bundle from source, and will
+> keep doing so until that flip lands.
+
+## Commands
+
+| Command                        | stdout                                  | Exit codes                                               |
+| ------------------------------ | --------------------------------------- | -------------------------------------------------------- |
+| `env`                          | resolved config as shell-eval'able vars | `0` = ran; non-zero = tool broken                        |
+| `litestream-config`            | the generated `litestream.yml`          | `0` = ran; non-zero = tool broken                        |
+| `restore <claude\|opencode>`   | —                                       | `0` = ran; non-zero = tool broken                        |
+| `sync-once <claude\|opencode>` | —                                       | `0` = ran; non-zero = tool broken                        |
+| `model-auth-ready`             | —                                       | `0` = ready, `10` = not ready; other = tool broken       |
+| `self-stop`                    | —                                       | `0` = stopped, `20` = keep the task; other = tool broken |
+
+`self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle
+exit, reading `CLUSTER`, `SERVICE` and (optionally) `EVIDENT_SELFSTOP_ROLE_ARN` from the
+environment. It exits `0` **only** on a confirmed `desiredCount` of 0.
+
+Everything the operator needs to read goes to **stderr**, prefixed `[auth-persistence]`,
+so a caller can safely capture stdout. The prefix now covers more than credentials — the
+ECS scale-to-zero messages carry it too. It is deliberately **not** renamed to something
+broader: `src/logger.ts` marks it verbatim because operator greps and CloudWatch filters
+match on that exact string, so widening what it labels is cheaper than breaking every
+saved query.
+
+### Why the contract is asymmetric
+
+`restore` and `sync-once` **never** report a domain outcome through the exit code. A
+missing remote object, a corrupt local file, a failed upload, an IAM denial — all of
+those are logged and still exit `0`. None of them should abort a boot: a runner with no
+credentials yet is a runner a human can still log into.
+
+The consequence is the point: **any non-zero status from `restore`/`sync-once` means the
+tool itself broke** — bad arguments (`2`), an uncaught throw (`1`), or a bundle that
+would not run at all. The shell needs no case analysis to know something is wrong, so
+its "credential persistence is DEGRADED" error lives in exactly one helper.
+
+The **two predicates** — `model-auth-ready` and `self-stop` — each need to distinguish
+"the answer is no" from "the tool is broken", so each answers no with its own dedicated
+code rather than a bare `1`: `10` for "no model auth", `20` for "not safe to stop, keep
+the task". That lets the caller treat _any other_ non-zero as the fail-safe answer.
+
+The fail-safe direction is the same for both, and deliberate. `model-auth-ready` holds a
+container alive for an operator, so a broken bundle must never convince a healthy runner
+that it has no auth. `self-stop` decides whether a task may go away, so a broken bundle
+must never kill one: `entrypoint.sh` reads any non-zero exactly like `20` and keeps the
+task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 — lets it stop.
+
+`src/shell-contract.json` is the machine-checked source of truth for the command list, and
+`shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
+`packages/runner-image/entrypoint.sh` and the MicroVM's
+`infrastructure/evident-microvm/docker/hooks` (#608), discovered by grep so a third one
+cannot go unchecked. Each shell must: call only subcommands the CLI implements (and, for
+`entrypoint.sh`, call all of them); route every call through one `run_synchroniser`; report
+a broken tool; and take each answer code the commands it calls can return **silently and by
+value**. That last obligation is derived from the subcommands the shell actually calls, so
+a shell that never self-stops is not made to know about `20`, and it is checked by
+**running** the real helper against a stubbed CLI rather than by pattern-matching it — the
+two shells spell the same intent differently, and a text check strict enough to catch a
+real violation also rejects correct rewrites.
+
+## Configuration
+
+All configuration is environment variables, resolved once in `src/config.ts`.
+
+| Variable                               | Meaning                                                                                      |
+| -------------------------------------- | -------------------------------------------------------------------------------------------- |
+| `HOME`                                 | **Required.** Every credential and database path is derived from it; unset is a fatal error. |
+| `LITESTREAM_BUCKET`                    | Object-store bucket.                                                                         |
+| `LITESTREAM_PREFIX`                    | Key prefix within the bucket.                                                                |
+| `AWS_REGION`                           | Region for the S3 client. Unset leaves it to the SDK's own resolution.                       |
+| `CREDS_SYNC_INTERVAL`                  | Seconds between sync ticks, reported by `env` for the caller's loop. Defaults to `60`.       |
+| `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` | Presence alone counts as configured model auth (see `model-auth-ready`).                     |
+
+**Persistence is enabled only when both `LITESTREAM_BUCKET` and `LITESTREAM_PREFIX` are
+set.** With either missing, `restore` and `sync-once` become no-ops and `env` warns
+loudly that credentials will be lost when the task is replaced — half a location can
+never produce a malformed key.
+
+The paths and keys that follow from that:
+
+| Store       | Local path                                | Object key                          |
+| ----------- | ----------------------------------------- | ----------------------------------- |
+| Claude      | `$HOME/.claude/.credentials.json`         | `<prefix>/claude/credentials.json`  |
+| OpenCode    | `$HOME/.local/share/opencode/auth.json`   | `<prefix>/opencode/auth.json`       |
+| OpenCode DB | `$HOME/.local/share/opencode/opencode.db` | `<prefix>/opencode.db` (litestream) |
+
+The session database is replicated by litestream itself; this package only generates its
+config. `env` emits `CREDS_BUCKET`, `CLAUDE_CREDS`, `OPENCODE_DB_PATH` and
+`CREDS_SYNC_INTERVAL`, each single-quoted so a value containing shell metacharacters is
+inert when `eval`'d.
+
+## Behaviour worth knowing
+
+- **A non-empty local file that is not valid JSON is protected**, never overwritten and
+  never uploaded: it may be unreadable to us and still be the operator's only copy. An
+  _empty_ one holds nothing to protect, so it does not block a restore.
+- **Valid JSON, not mere existence, is the readiness test** — for both stores, and it is
+  re-read on every call, because OpenCode provider auth can be bootstrapped long after
+  boot.
+- **Restores are atomic**: the object is written to an exclusively-created temporary
+  file, `chmod`ed before the secret lands in it, validated, then renamed into place. The
+  temporary file is removed on every failure path. Directories end up `700`, credential
+  files `600`.
+- **Nothing throws out of `restore`/`sync`.** `sync-once` runs on an unsupervised timer,
+  where an unguarded failure would silently stop persisting credentials for the rest of
+  the task's life. Failures degrade to a warning and are retried on the next tick.
+- **`sync-once` uploads only on change.** Each store's last uploaded hash (a sha256, not
+  the credentials) is kept in a state file next to it, so every invocation is a
+  self-contained process.
+- **Not-found is distinguished from failed.** A missing object is reported as "nothing to
+  restore"; an access or network error is reported as a failure, because that tells the
+  operator their persisted credentials may still be recoverable.
+
+## The `ObjectStore` port
+
+`restore` and `sync` talk to a two-method interface — `get(key)` returning `null` when
+the object is absent, and `put(key, body)`. S3 vocabulary (`@aws-sdk/client-s3`, the
+typed `NoSuchKey`/`NotFound` errors) is confined to the single adapter in
+`src/s3-object-store.ts`, so the storage backend can be swapped without touching the
+restore/sync logic.
+
+The S3 adapter is the only one that exists. It uses no static credentials — auth is
+whatever the ambient AWS credential chain resolves (on the Evident runner, the ECS task
+role).
+
+## Development
+
+```bash
+pnpm install
+pnpm build      # tsup -> dist/cli.js, a single self-contained ESM bundle
+pnpm test       # vitest
+pnpm typecheck
+pnpm lint
+```
+
+From the repository root, prefix with `pnpm --filter @evident-ai/runner-synchroniser`.
+
+The bundle must stay a **single file**: the runner image copies exactly one artifact out
+of its builder stage, so code splitting would ship a `cli.js` importing chunks that are
+not there. `tsup.config.ts` disables splitting and bundles the AWS SDK, and the
+Dockerfile asserts the output is one file.
+
+The restore/sync/CLI tests need neither AWS nor a real filesystem — rather than shimming
+a fake `aws` binary onto `PATH`, they inject two ports: `InMemoryObjectStore` (in
+`src/object-store.ts`) and `FakeFileOps` (in `src/test-support.ts`). The `FileOps` port
+exists specifically for failure injection: the suite runs as root, where a real
+`chmod 000` is a no-op, so "a `chmod` that fails must warn and not abort" and "a hash
+failure must skip one tick, not kill the loop" are otherwise not reproducible. The real
+`nodeFileOps` implementation is tested against a temporary directory.