@desplega.ai/agent-env 0.0.0-stage → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +343 -2
  3. package/assets/icon.svg +5 -0
  4. package/bin/agent-env.mjs +210 -0
  5. package/docs/agent-swarm.md +146 -0
  6. package/docs/releases.md +207 -0
  7. package/docs/screenshots/dashboard-desktop.png +0 -0
  8. package/docs/screenshots/dashboard-mobile.png +0 -0
  9. package/docs/snapshots-and-worktrees.md +221 -0
  10. package/examples/agent-swarm/README.md +5 -0
  11. package/examples/agent-swarm/agent-env.json +11 -0
  12. package/examples/agent-swarm/compose.agent-env.yaml +101 -0
  13. package/examples/agent-swarm/verify.mjs +55 -0
  14. package/examples/compose/.dockerignore +3 -0
  15. package/examples/compose/Dockerfile +4 -0
  16. package/examples/compose/README.md +109 -0
  17. package/examples/compose/compose.yaml +36 -0
  18. package/examples/compose/server.mjs +6 -0
  19. package/examples/hello/agent-env.json +20 -0
  20. package/examples/hello/index.html +12 -0
  21. package/package.json +49 -4
  22. package/src/cloud.mjs +314 -0
  23. package/src/config.mjs +133 -0
  24. package/src/daemon.mjs +484 -0
  25. package/src/environments.mjs +207 -0
  26. package/src/goodies.mjs +71 -0
  27. package/src/init.mjs +68 -0
  28. package/src/locks.mjs +56 -0
  29. package/src/maintenance.mjs +74 -0
  30. package/src/paths.mjs +5 -0
  31. package/src/providers/CLOUD_SANDBOX.md +240 -0
  32. package/src/providers/README.md +62 -0
  33. package/src/providers/compose.mjs +460 -0
  34. package/src/providers/docker.mjs +207 -0
  35. package/src/providers/index.mjs +13 -0
  36. package/src/providers/sandbox.mjs +170 -0
  37. package/src/proxy.mjs +149 -0
  38. package/src/runtime.mjs +83 -0
  39. package/src/snapshots.mjs +394 -0
  40. package/src/source-paths.mjs +15 -0
  41. package/src/system.mjs +39 -0
  42. package/src/tls.mjs +57 -0
  43. package/src/ui-client.js +51 -0
  44. package/src/ui.html +12 -0
  45. package/src/ui.mjs +32 -0
  46. package/src/watch.mjs +20 -0
  47. package/src/worktrees.mjs +154 -0
@@ -0,0 +1,146 @@
1
+ # Agent Swarm with agent-env
2
+
3
+ Run the real API and dashboard from an Agent Swarm checkout.
4
+ Copy two configuration files, provide a private test credential, and run `agent-env up`.
5
+ The Compose setup job installs Linux dependencies before the API and dashboard start.
6
+
7
+ ## Requirements
8
+
9
+ - Node.js 22 or newer on the host.
10
+ - Docker Engine and Docker Compose with JSON configuration, `config --variables`, and `up --wait`.
11
+ - An Agent Swarm checkout with `bun@1.4.4` in `package.json`.
12
+ - Docker Hub and package registry access for the first installation.
13
+ - Free host ports 4313 and 4574, and approximately 7 GiB for the first installation.
14
+
15
+ The host needs no Bun or portless installation.
16
+ The example uses `oven/bun:1.4.4` without building application images.
17
+
18
+ ## Start from an Agent Swarm checkout
19
+
20
+ Run these commands from the checkout.
21
+ Set `AGENT_ENV_REPO` to your agent-env checkout.
22
+ For a global npm installation, use `export AGENT_ENV_REPO="$(npm root -g)/@desplega.ai/agent-env"` instead.
23
+ Do not replace existing configuration files.
24
+ If either destination exists, select another filename and pass `--config` or `--compose-file` explicitly.
25
+
26
+ ```sh
27
+ export AGENT_ENV_REPO=/path/to/agent-env
28
+ cp "$AGENT_ENV_REPO/examples/agent-swarm/agent-env.json" ./agent-env.json
29
+ cp "$AGENT_ENV_REPO/examples/agent-swarm/compose.agent-env.yaml" ./compose.agent-env.yaml
30
+
31
+ umask 077
32
+ export AGENT_ENV_HOME="$(mktemp -d /tmp/agent-env-swarm-XXXXXX)"
33
+ export AGENT_SWARM_INSTANCE=swarm-example
34
+ node --input-type=module <<'JS'
35
+ import { writeFileSync } from 'node:fs';
36
+ import { randomBytes } from 'node:crypto';
37
+ writeFileSync(`${process.env.AGENT_ENV_HOME}/test.env`,
38
+ `AGENT_SWARM_API_KEY=${randomBytes(32).toString('hex')}\n` +
39
+ `AGENT_SWARM_INSTANCE=${process.env.AGENT_SWARM_INSTANCE}\n`, { mode: 0o600 });
40
+ JS
41
+
42
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" up --name "$AGENT_SWARM_INSTANCE" \
43
+ --env-file "$AGENT_ENV_HOME/test.env"
44
+ ```
45
+
46
+ The installed CLI also accepts `agent-env up` in place of the Node invocation.
47
+ `startupTimeoutSeconds: 900` allows the setup job to finish its first installation.
48
+ `--timeout 900` provides the equivalent CLI override.
49
+
50
+ The source bind points directly at the current checkout.
51
+ The dashboard sees source edits through Vite.
52
+ Restart the API with `down` and `up` after API source edits.
53
+ Docker stores Linux dependencies and Bun downloads in separate named volumes for each instance.
54
+ The setup job uses the checkout's frozen lockfile and hoisted linker configuration.
55
+ It skips lifecycle scripts.
56
+
57
+ ## Verify the real application
58
+
59
+ Keep the same `AGENT_ENV_HOME` in this shell.
60
+
61
+ ```sh
62
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" use "$AGENT_SWARM_INSTANCE"
63
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" list
64
+ node --env-file="$AGENT_ENV_HOME/test.env" \
65
+ "$AGENT_ENV_REPO/examples/agent-swarm/verify.mjs"
66
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" logs "$AGENT_SWARM_INSTANCE" --service api
67
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" logs "$AGENT_SWARM_INSTANCE" --service setup
68
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" exec "$AGENT_SWARM_INSTANCE" --service api -- \
69
+ bun --no-env-file -e "fetch('http://127.0.0.1:3013/health').then(r => r.json()).then(console.log)"
70
+
71
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" down "$AGENT_SWARM_INSTANCE"
72
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" up "$AGENT_SWARM_INSTANCE"
73
+ node --env-file="$AGENT_ENV_HOME/test.env" \
74
+ "$AGENT_ENV_REPO/examples/agent-swarm/verify.mjs" --read-only
75
+ ```
76
+
77
+ The first check writes a value through `PUT /api/kv/_/agent-env-example/persistence`.
78
+ It reads the value through the direct API and the dashboard proxy.
79
+ The second check only reads the retained value after restart.
80
+ Both checks verify authentication, health, dashboard HTML, and the transformed dashboard entry module.
81
+ The check uses HTTP and does not prove browser rendering or interaction.
82
+
83
+ | Service | Default address |
84
+ | --- | --- |
85
+ | API | `http://swarm-example.localhost:4313` |
86
+ | Dashboard | `http://swarm-example.localhost:4574` |
87
+ | Selected API | `http://localhost:4313` |
88
+ | Selected dashboard | `http://localhost:4574` |
89
+
90
+ Enter the API address and generated test key in the dashboard's connection form.
91
+ The key remains in the private environment file until you use it.
92
+ The example does not embed the key in browser assets.
93
+ If `.localhost` resolution fails, use `localhost` after `agent-env use`.
94
+ For curl, use `--resolve swarm-example.localhost:4313:127.0.0.1`.
95
+
96
+ ## Configuration and isolation
97
+
98
+ | Setting | Purpose |
99
+ | --- | --- |
100
+ | `AGENT_SWARM_API_KEY` | Required test credential, generated outside the checkout. |
101
+ | `AGENT_SWARM_INSTANCE` | CORS instance name. Must match `--name`. Default: `swarm-example`. |
102
+ | `AGENT_ENV_HOME` | Separate daemon, private configuration, and saved instance state. |
103
+ | `DATABASE_PATH=/data/agent-swarm.sqlite` | SQLite data in the instance's named volume. |
104
+ | `AGENT_FS_LOCAL_DIR=/data/files` | Local storage in the same isolated data volume. |
105
+ | `CAPABILITIES=core,task,kv,skills,repo` | Excludes the scheduler capability. |
106
+
107
+ No inference, Slack, tracker, GitHub, or agent-fs credentials are required.
108
+ The API receives only explicit Compose environment values.
109
+ All Bun commands use `--no-env-file`.
110
+ The dashboard starts the checkout's actual Vite configuration with `envDir: false`.
111
+ These settings prevent Bun and Vite from loading existing private `.env` files.
112
+ The explicit `--env-file` selects the generated credential for Compose interpolation.
113
+
114
+ The file disables heartbeat, runtime pools, automatic titles, integrations, OAuth sweeps, and telemetry.
115
+ It starts no lead or worker containers and does not execute schedules or create tasks.
116
+ Normal API startup seeds schedule definitions, including some marked enabled.
117
+ The scheduler remains disabled because the example excludes the `scheduling` capability.
118
+ The API still runs its normal internal maintenance services.
119
+ Its automatic public model-catalog refresh can contact `models.dev`.
120
+ The Compose network permits outbound traffic, so this example is not an internet sandbox.
121
+ Docker omitted published ports on an internal-only network during verification.
122
+
123
+ Agent-env isolates container names, the network, all dependency volumes, and the data volume.
124
+ It publishes Docker backend ports only on random loopback ports.
125
+ The configured localhost routes use ports 4313 and 4574.
126
+ The data volume retains SQLite, encryption material, and the page-session secret after `down` and `up`.
127
+ The API mounts source read-only.
128
+ Setup and Vite use a writable source bind, with dependency directories covered by Linux volumes.
129
+ Docker can create empty `node_modules` mountpoint directories in a fresh checkout.
130
+ The setup job does not change the frozen lockfile or package manifests.
131
+
132
+ ## Cleanup
133
+
134
+ ```sh
135
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" rm "$AGENT_SWARM_INSTANCE" --purge
136
+ node "$AGENT_ENV_REPO/bin/agent-env.mjs" daemon stop
137
+ rm -r "$AGENT_ENV_HOME"
138
+ # Remove only the two configuration files copied for this run.
139
+ rm ./agent-env.json ./compose.agent-env.yaml
140
+ df -h /
141
+ ```
142
+
143
+ Purge removes the instance's containers, network, dependency caches, and data volume.
144
+ The cleanup does not remove shared caches or other Docker resources.
145
+ No custom images need removal.
146
+ If this test alone downloaded the Bun image, remove it with `docker image rm oven/bun:1.4.4`.
@@ -0,0 +1,207 @@
1
+ # CI and releases
2
+
3
+ The repository is `desplega-ai/agent-env`. Its default branch is `master`.
4
+ The npm package is `@desplega.ai/agent-env`.
5
+
6
+ ## Ordinary CI
7
+
8
+ `.github/workflows/ci.yml` runs for every pull request and every push to `master`.
9
+ Public forks need no secrets. Ordinary CI never creates paid cloud resources.
10
+ CI uses public CLI E2E tests. It does not run unit tests.
11
+
12
+ The package job creates one tarball with `npm pack --ignore-scripts`.
13
+ `scripts/pack.mjs` audits its file list, package identity, version, and executable entry.
14
+ The audit rejects files outside the published directories, credential filenames, and unexpected artifacts.
15
+ Every test job downloads that same tarball and installs it into a temporary prefix.
16
+ No job installs the package globally.
17
+
18
+ The `CI` job requires all these jobs to succeed:
19
+
20
+ | Job | Runtime | Real dependencies |
21
+ | --- | --- | --- |
22
+ | Core | Linux and macOS, Node 22 and latest LTS | Git, public CLI |
23
+ | Docker | Linux, latest Node LTS | Docker, HTTP, TCP, TLS, sandbox bridge |
24
+ | Compose | Linux, latest Node LTS | Docker Compose |
25
+ | SSH | Linux, latest Node LTS | Dedicated localhost sshd, Docker, Compose, rsync |
26
+ | S3 | Linux, latest Node LTS | Docker Compose, AWS CLI, disposable SeaweedFS |
27
+
28
+ The SSH fixture generates new client and host keys for every job.
29
+ The client pins the generated host key and requires strict host verification.
30
+ The fixture starts its own sshd with password authentication disabled.
31
+ PAM handles account checks, which permits key authentication for the runner's password-locked account.
32
+ Its command wrapper isolates the remote workspace.
33
+ The SSH process keeps the real CI user's home.
34
+ It does not change the system SSH configuration or authorized keys.
35
+
36
+ SeaweedFS uses version `4.52` and a pinned image digest.
37
+ Its S3 port binds only to localhost.
38
+ Each job generates credentials and creates a disposable bucket.
39
+ The tests remove their snapshot prefixes. Cleanup removes the container and its storage.
40
+
41
+ ## Runner contract
42
+
43
+ The test owner maintains `test/run.mjs` and its suites.
44
+ `npm test` runs core, Docker, and Compose E2E.
45
+ `npm run test:core` runs the subset that requires no Docker.
46
+ CI calls the following commands through `scripts/ci-e2e.sh`:
47
+
48
+ ```sh
49
+ AGENT_ENV_CLI=/absolute/prefix/node_modules/.bin/agent-env node test/run.mjs core
50
+ AGENT_ENV_CLI=/absolute/prefix/node_modules/.bin/agent-env node test/run.mjs docker
51
+ AGENT_ENV_CLI=/absolute/prefix/node_modules/.bin/agent-env node test/run.mjs compose
52
+ AGENT_ENV_CLI=/absolute/prefix/node_modules/.bin/agent-env node test/run.mjs ssh
53
+ AGENT_ENV_CLI=/absolute/prefix/node_modules/.bin/agent-env node test/run.mjs s3
54
+ AGENT_ENV_CLI=/absolute/prefix/node_modules/.bin/agent-env node test/run.mjs cloud
55
+ ```
56
+
57
+ All CLI invocations must use `AGENT_ENV_CLI` when CI supplies it.
58
+ Tests must not import implementation modules or substitute Docker or SSH mocks.
59
+ Tests must create their directories beneath `TMPDIR` and remove their own resources.
60
+ The cloud suite must preserve state if resource cleanup fails.
61
+
62
+ The wrapper supplies these variables:
63
+
64
+ | Suite | Variables |
65
+ | --- | --- |
66
+ | All | `AGENT_ENV_CLI`, `AGENT_ENV_HOME`, `TMPDIR`, isolated npm and XDG directories |
67
+ | Docker | `AGENT_ENV_TEST_SANDBOX=1` |
68
+ | SSH | `AGENT_ENV_TEST_SSH=agent-env-ci`, `AGENT_ENV_TEST_SSH_CONFIG` |
69
+ | S3 | `AGENT_ENV_TEST_S3=s3://agent-env-ci/snapshots`, `AGENT_ENV_TEST_S3_ENDPOINT`, temporary AWS credentials |
70
+ | Cloud | `AGENT_ENV_TEST_CLOUD=1`, `HCLOUD_TOKEN` from the protected environment |
71
+
72
+ The wrapper removes unrelated SSH and S3 settings from the inherited environment.
73
+ Each suite must remove instances, stop daemons, and remove its temporary files.
74
+ The wrapper also attempts cleanup after failure or termination.
75
+ An `always()` workflow step provides another cleanup attempt.
76
+ Cleanup retains recovery files when resource removal fails.
77
+ Cleanup requires the test directory's ownership marker and checks recorded fixture ownership.
78
+ It refuses resources with a different owner and reports failure.
79
+
80
+ ## Automatic release
81
+
82
+ Merge an increasing version change in `package.json` to `master`.
83
+ The workflow compares the version against `github.event.before`, covering the entire push.
84
+ It does not compare only the final commit's parent.
85
+ Pull request CI also checks version increases against `pull_request.base.sha`.
86
+ A downgrade fails before merge.
87
+
88
+ The release job requires successful CI from the same workflow and exact commit.
89
+ It publishes the same archive that every E2E job tested.
90
+ It checks these conditions before publishing:
91
+
92
+ 1. The package version contains valid SemVer and increases from the version before the push.
93
+ 2. The archive name and version match `package.json` exactly.
94
+ 3. A new version exceeds every version already published to npm.
95
+ 4. Any existing version contains the exact same archive integrity.
96
+ 5. Any existing Git tag points to the tested commit.
97
+ 6. Any existing GitHub asset contains the exact same bytes.
98
+
99
+ npm releases cannot include SemVer build metadata because npm does not preserve it.
100
+
101
+ After npm confirms the exact version and integrity, the job creates `v<VERSION>` and the GitHub release.
102
+ The version check bypasses registry caches and retries propagation for up to twelve attempts, ten seconds apart.
103
+ If publication fails after npm accepted the archive, the retry verifies its integrity before continuing.
104
+ The GitHub release attaches only the tested `.tgz` file.
105
+ Prereleases use npm's `next` tag and GitHub's prerelease flag.
106
+ Stable releases use npm's `latest` tag.
107
+ Retries preserve existing tags and assets. They never overwrite conflicting releases.
108
+ An already published version must match the archive before the workflow resumes.
109
+ Registry failures stop the release instead of treating the version as absent.
110
+
111
+ Concurrency queues master runs so a later push cannot discard an earlier version bump.
112
+ GitHub permits up to 100 queued runs per group.
113
+ GitHub documents `queue: max` in its [concurrency reference](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency).
114
+ Actionlint 1.7.12 does not recognize this field yet.
115
+ Run its other checks with `actionlint -ignore 'unexpected key "queue" for "concurrency" section'` until actionlint supports it.
116
+
117
+ ## One-time npm setup
118
+
119
+ The repository already has access to the organization secret `NPM_TOKEN`.
120
+ No local npm authentication or publication forms part of CI implementation.
121
+
122
+ OIDC is the default publishing method.
123
+ The release job grants `id-token: write` and requires npm 11.5.1 or later.
124
+ Use npm's [trusted publishing instructions](https://docs.npmjs.com/trusted-publishers/) to configure:
125
+
126
+ | Setting | Value |
127
+ | --- | --- |
128
+ | Provider | GitHub Actions |
129
+ | Organization | `desplega-ai` |
130
+ | Repository | `agent-env` |
131
+ | Workflow filename | `ci.yml` |
132
+ | Environment | Leave empty |
133
+ | Allowed action | Publish |
134
+
135
+ The initial package may need token authentication before npm exposes its publisher settings.
136
+ To publish an initial version without another version bump, dispatch `CI and release` on `master` with both inputs enabled:
137
+
138
+ ```sh
139
+ gh workflow run ci.yml --repo desplega-ai/agent-env --ref master \
140
+ -f release=true -f use_npm_token=true
141
+ ```
142
+
143
+ This dispatch still runs every ordinary CI job before it publishes.
144
+ It passes `secrets.NPM_TOKEN` as `NODE_AUTH_TOKEN` only to the release step.
145
+ The release script uses that token only when token publishing is explicitly selected.
146
+ After bootstrap, configure the trusted publisher and dispatch with `use_npm_token` disabled to test OIDC.
147
+ For manual dispatch, this input overrides the repository variable, including an explicit disabled value.
148
+ Remove the token fallback when npm OIDC succeeds.
149
+ If token publishing remains necessary, repository variable `NPM_USE_TOKEN=true` enables it for automatic bumps.
150
+ This repository currently sets that variable for its initial token-based publication.
151
+ The token must permit public publishing under `@desplega.ai`.
152
+
153
+ To test a version bump before merging:
154
+
155
+ ```sh
156
+ npm version patch --no-git-tag-version
157
+ # Or: npm version minor --no-git-tag-version
158
+ # Or: npm version 0.3.0 --no-git-tag-version
159
+ npm run check
160
+ npm test
161
+ git diff -- package.json
162
+ ```
163
+
164
+ Commit the version change through the normal pull request process.
165
+ Do not create or push a version tag manually.
166
+ The workflow creates the tag after tests and npm verification succeed.
167
+ For a retry without another bump, dispatch `ci.yml` with `release=true` on the same commit.
168
+ Use GitHub's rerun action when `master` has advanced beyond that commit.
169
+
170
+ ## Manual Hetzner E2E
171
+
172
+ `.github/workflows/hetzner-e2e.yml` runs only through manual dispatch on `master`.
173
+ It creates billed infrastructure only after approval for environment `hetzner-e2e`.
174
+ An administrator must create that protected environment before use.
175
+ Configure required reviewers, prevent approval by the requester, and restrict deployments to `master`.
176
+ Store `HCLOUD_TOKEN` as an environment secret with access to a dedicated test project.
177
+ This implementation does not change GitHub settings.
178
+
179
+ The job installs a pinned Hetzner CLI archive and checks its SHA-256 digest.
180
+ It tests an audited package through its installed executable with `AGENT_ENV_TEST_CLOUD=1`.
181
+ The suite must remove its server, primary IP, SSH key, volumes, and local state.
182
+ The wrapper and `always()` step attempt recovery with the recorded CLI state.
183
+ Inspect the dedicated Hetzner project after a runner loss or forced cancellation.
184
+ An unavailable runner cannot execute cleanup.
185
+
186
+ ```sh
187
+ gh workflow run hetzner-e2e.yml --repo desplega-ai/agent-env --ref master
188
+ ```
189
+
190
+ ## Local package verification
191
+
192
+ Docker and Compose must work for integration groups.
193
+ The SSH fixture requires Linux, OpenSSH server, rsync, and passwordless sudo.
194
+ The S3 fixture requires AWS CLI v2.
195
+
196
+ ```sh
197
+ work=$(mktemp -d /tmp/agent-env-package-check-XXXXXX)
198
+ node scripts/pack.mjs "$work"
199
+ tarball=$(find "$work" -name '*.tgz' -type f)
200
+ bash scripts/ci-e2e.sh "$tarball" core
201
+ # Run these integration groups on Linux with Docker available:
202
+ bash scripts/ci-e2e.sh "$tarball" docker
203
+ bash scripts/ci-e2e.sh "$tarball" compose
204
+ bash scripts/ci-e2e.sh "$tarball" ssh
205
+ bash scripts/ci-e2e.sh "$tarball" s3
206
+ rm -r "$work"
207
+ ```
@@ -0,0 +1,221 @@
1
+ # Snapshot and Git worktree helpers
2
+
3
+ These modules provide storage and source helpers for the parent lifecycle code.
4
+ The daemon integrates them with snapshot, restore, and fork commands.
5
+ S3 stores backup archives. It does not store live database files.
6
+
7
+ ## Snapshot API
8
+
9
+ ```js
10
+ import { snapshotInstance, restoreSnapshot } from '../src/snapshots.mjs';
11
+
12
+ const exported = await snapshotInstance(handle, location, provider, options);
13
+ // { location, manifest }
14
+
15
+ const restored = await restoreSnapshot(destinationSpec, location, provider, options);
16
+ // { location, manifest, volumes, volumeNames }
17
+ ```
18
+
19
+ `location` is a new local directory or `s3://bucket/nonempty-prefix`.
20
+ The local parent directory must exist.
21
+ An existing local directory or nonempty S3 prefix causes export to fail.
22
+ Restore refuses every existing destination volume, including empty volumes.
23
+
24
+ The provider must implement `docker(spec, argv, options)` with the Docker provider's command contract.
25
+ It must forward `options.stdio` to the child process for archive commands.
26
+ The helper redirects archive bytes through file descriptors.
27
+ It never converts archive bytes to strings.
28
+ The provider handles local execution, SSH arguments, and command quoting.
29
+
30
+ Each entry in `handle.services` must identify `container` or a `containers` array of container names or IDs.
31
+ The helper inspects those containers to discover actual volume names and mount targets.
32
+ It does not reconstruct source names from provider naming conventions.
33
+ This supports ordinary Docker handles and Compose handles with multiple containers per service.
34
+
35
+ All source containers must be stopped or newly created.
36
+ The helper checks their state before and after export.
37
+ Callers must serialize lifecycle operations for the full export or restore operation.
38
+ External Docker operations must also leave these containers and volumes unchanged.
39
+ The helper cannot prevent another Docker client from starting a container during an archive operation.
40
+
41
+ **Cloud integration:** the parent must wake the host without starting its containers.
42
+ Do not call a cloud resume operation that also starts services before snapshot export.
43
+ Pass the reachable Docker host through the provider's existing command helpers.
44
+ The helpers do not provision, wake, or shut down cloud hosts.
45
+
46
+ The helper accepts these options:
47
+
48
+ | Option | Operation | Meaning |
49
+ | --- | --- | --- |
50
+ | `image` | Both | Archive helper image. Default: `busybox:1.37`. |
51
+ | `endpointUrl` | S3 operations | AWS CLI endpoint URL for compatible services. |
52
+ | `awsCommand` | S3 operations | AWS executable path. Default: installed `aws`. |
53
+ | `omitUnsupportedMounts` | Export | Explicitly omit unsupported or shared mounts. Default: reject them. |
54
+ | `volumeNames` | Restore | Map manifest volume IDs, such as `v0`, to fresh Docker volume names. |
55
+ | `destinationHandle` | Restore | Identify existing destination service containers for a stopped-state check. |
56
+ | `composeProject` | Restore | Identify an existing destination Compose project for a stopped-state check. |
57
+
58
+ The Docker naming fallback checks existing destination containers through `provider.prefix(spec)` when available.
59
+ Compose callers must supply `composeProject` or `destinationHandle` when destination containers already exist.
60
+ Restore returns a unique `restoreId` for safe failure cleanup through `disposeRestoredVolumes`.
61
+ Restore creates volumes only. It does not create containers, copy source files, or start services.
62
+ The parent must attach the returned volumes before service creation.
63
+
64
+ Without `volumeNames`, restore matches each saved service and mount target against `destinationSpec.services[service].volumes`.
65
+ It calls `provider.volume(spec, service, key)` to obtain the destination name.
66
+ Every saved mount of one source volume must resolve to the same destination name.
67
+ The Compose provider resolves destination volume names and labels through its snapshot hooks.
68
+ The parent must also preserve shared mounts within the same instance through that map.
69
+
70
+ ```js
71
+ const saved = await snapshotInstance(stoppedHandle, '/backups/checkpoint-1', docker);
72
+ const destinationSpec = { ...stoppedHandle.spec, name: 'feature-copy' };
73
+ const restored = await restoreSnapshot(destinationSpec, '/backups/checkpoint-1', docker);
74
+
75
+ const composed = await restoreSnapshot(destinationSpec, '/backups/checkpoint-1', compose, {
76
+ composeProject: 'feature-copy',
77
+ volumeNames: { v0: 'feature-copy_database' },
78
+ });
79
+ ```
80
+
81
+ ## Ownership and archive rules
82
+
83
+ Managed volumes need a matching `agent-env.owner` label or the container's matching `com.docker.compose.project` label.
84
+ Only local Docker volumes without driver options qualify for automated export.
85
+ Volumes attached to any container outside the handle are shared, even when that container is stopped.
86
+ The helper rejects those volumes.
87
+ Providers must not assign managed ownership labels to external volumes.
88
+
89
+ Bind mounts inside `handle.workspace` or `spec.cwd` are omitted as source files.
90
+ Source files need a separate Git fork or synchronization operation.
91
+ Other bind mounts, tmpfs mounts, external volumes, and shared volumes require separate backup procedures.
92
+ `omitUnsupportedMounts: true` records each omission with its service, target, type, and reason.
93
+ The parent must display these omissions to the caller.
94
+
95
+ `manifest.json` has format `agent-env-snapshot`, version `1`.
96
+ Each volume record includes its ID, actual source name, service mounts, archive name, SHA-256, and archive byte count.
97
+ One source volume produces one archive, even when several source services mount it.
98
+ The manifest includes only source name, owner, and provider identifiers.
99
+ It does not serialize the specification, container inspection, environment values, or image commands.
100
+ Volume contents can contain application secrets. The manifest exclusion does not redact those files.
101
+
102
+ Archives use uncompressed tar files.
103
+ Validation streams file bodies and checks tar headers before extraction.
104
+ It supports regular files, directories, internal symbolic links, hard links to earlier files, and GNU long paths.
105
+ It rejects traversal paths, absolute paths, duplicate paths, link traversal, external symbolic links, special files, and unsupported tar extensions.
106
+ Symbolic link cycles and chains longer than 64 links are also unsupported.
107
+ PAX metadata, sparse files, extended attributes, and ACL preservation are outside this format.
108
+ Tar numbers must use octal fields. Base-256 numeric fields are unsupported.
109
+ The default BusyBox image supplies the tested archive behavior.
110
+
111
+ Restore copies all archives to a private temporary directory.
112
+ It checks every SHA-256, size, and extraction path before creating any volume.
113
+ Staging requires additional local free space for the entire snapshot.
114
+ Restore labels its newly created volumes with a unique claim token.
115
+ Failure cleanup removes only volumes with that token and helper containers with their matching token.
116
+ If Docker prevents cleanup, the error includes `cleanupErrors` for the parent to report.
117
+ Restore never removes a preexisting volume or destination container.
118
+
119
+ ## S3 behavior
120
+
121
+ The helper uses installed AWS CLI configuration, credentials, region selection, and environment variables.
122
+ It does not accept or store AWS access keys in the manifest.
123
+ An optional `endpointUrl` selects an S3-compatible service.
124
+
125
+ Export checks the prefix with `s3api list-objects-v2`.
126
+ It uploads each archive with `s3api put-object --if-none-match '*'`.
127
+ It publishes `manifest.json` last with the same condition.
128
+ The storage service must support conditional creation.
129
+ Download uses `s3api get-object` with a local output file.
130
+
131
+ A failed S3 export can leave partial objects under its new prefix.
132
+ The helper retains them because another writer could own later object versions.
133
+ Choose a new prefix for a retry.
134
+ Inspect and remove partial objects through a separate storage operation.
135
+ The required permissions are object listing, object creation, and object reads.
136
+ This module does not list saved snapshots or delete cloud backups.
137
+
138
+ ## Git API
139
+
140
+ ```js
141
+ import { createWorktree, removeOwnedWorktree } from '../src/worktrees.mjs';
142
+
143
+ const fork = await createWorktree(sourceSpec, 'feature-copy', {
144
+ path: '/existing/parent/feature-copy',
145
+ branch: 'feature/database-copy',
146
+ ref: 'HEAD',
147
+ copyChanges: true,
148
+ });
149
+ // { cwd, ownership, omittedPaths }
150
+
151
+ await removeOwnedWorktree(fork);
152
+ // { removed, branchRetained }
153
+
154
+ // Explicitly discard files in this owned worktree:
155
+ await removeOwnedWorktree(fork, { dirty: 'discard' });
156
+ ```
157
+
158
+ `sourceSpec` can be a specification with `cwd` or a directory string.
159
+ Git identifies the repository root, common directory, refs, tracked changes, and untracked files.
160
+ The default path is a sibling of the source repository named `destinationName`.
161
+ The default branch is `destinationName`, and the default ref is `HEAD`.
162
+ The helper refuses existing branches, existing paths, and destinations within the source repository.
163
+ The destination parent directory must already exist.
164
+ For a source subdirectory, returned `cwd` preserves that relative location inside the new worktree.
165
+
166
+ The default fork applies the source's staged and unstaged changes as one working-tree patch.
167
+ It preserves binary bytes, additions, deletions, and executable modes.
168
+ It leaves the source index and source files unchanged.
169
+ A different base ref can cause patch conflicts. The helper then removes its new worktree and branch.
170
+ Set `copyChanges: false` for a clean checkout of the selected ref.
171
+
172
+ Untracked files come from `git ls-files --others --exclude-standard`.
173
+ Only regular files qualify for copying.
174
+ Credential names, private keys, dependency directories, and common build directories are excluded.
175
+ The same exclusions apply to tracked changes.
176
+ The helper reports excluded candidates in `omittedPaths`.
177
+ Git already excludes ignored files from the untracked candidates.
178
+ The helper does not report those ignored files individually.
179
+ Untracked symbolic links and special files are omitted.
180
+ The selected ref must not track recognized credential files.
181
+ Credential detection uses conservative filename rules. It cannot detect credentials embedded in arbitrary source files.
182
+ Committed build files remain part of the selected ref.
183
+ Submodule working directories and changes within submodules are not copied.
184
+
185
+ Ownership records reside in `<git-common-dir>/agent-env-worktrees/<token>.json`.
186
+ Persist the returned `ownership` object with the instance record.
187
+ Removal checks that record and Git's current repository, path, and branch.
188
+ It refuses dirty worktrees, including ignored files, unless the caller explicitly selects `dirty: 'discard'`.
189
+ Commit or preserve valuable files before removal.
190
+ Removal retains branches with new commits.
191
+ Failure cleanup reports a `cleanupError` if Git cannot remove the owned resources.
192
+ Callers must serialize Git operations while helpers create or remove a worktree.
193
+
194
+ The parent coordinates source and volume forks.
195
+ Create the worktree first, set the destination specification's `cwd`, and then create the destination volumes and services.
196
+ If the data fork fails, remove the returned worktree with the explicit discard option.
197
+ That new worktree normally contains the copied source changes and therefore counts as dirty.
198
+
199
+ ## Verification
200
+
201
+ ```sh
202
+ node --check src/snapshots.mjs
203
+ node --check src/worktrees.mjs
204
+ npm run check
205
+ node test/run.mjs features
206
+ AGENT_ENV_TEST_SSH=my-test-host node test/run.mjs ssh
207
+ AGENT_ENV_TEST_S3=s3://test-bucket/prefix node test/run.mjs s3
208
+ ```
209
+
210
+ Set `AGENT_ENV_CLI` to the absolute path of an installed package executable.
211
+ Docker tests require a running Docker host and can download public images.
212
+ Tests use owned temporary containers, volumes, directories, and Git repositories.
213
+ They check restoration, source isolation, Compose volumes, archive checksums, cleanup, conflicts, and credential exclusions through the public CLI.
214
+ S3 tests use a real S3 service. CI provides a disposable SeaweedFS fixture.
215
+ SSH tests use a real server. CI provides a disposable local SSH fixture.
216
+
217
+ Live integration passed against a temporary SeaweedFS S3 server. Verify each production storage service separately.
218
+ Test export and restore against a temporary bucket or a local S3-compatible server.
219
+ Check the service's conditional write behavior and remove the test objects afterward.
220
+ Live SSH snapshot and restore integration passed on a temporary Hetzner Docker host.
221
+ These checks require actual endpoint credentials or an SSH host that the parent supplies.
@@ -0,0 +1,5 @@
1
+ # Agent Swarm
2
+
3
+ Run the real API and dashboard from a local Agent Swarm checkout.
4
+ See [the run guide](../../docs/agent-swarm.md) for commands, isolation, and verification.
5
+ See [verification evidence](VERIFICATION.md) for the tested revision and observed results.
@@ -0,0 +1,11 @@
1
+ {
2
+ "provider": "compose",
3
+ "startupTimeoutSeconds": 900,
4
+ "compose": {
5
+ "files": ["compose.agent-env.yaml"],
6
+ "routes": {
7
+ "api": { "service": "api", "target": 3013, "port": 4313, "protocol": "http" },
8
+ "ui": { "service": "ui", "target": 5274, "port": 4574, "protocol": "http" }
9
+ }
10
+ }
11
+ }