@pirol/nido 0.2.0 → 0.3.0

package/README.md

@@ -1,50 +1,52 @@
 # nido
 
-CLI für Agenten-Sandboxes, wie im Konzept
-[Agenten-Sandboxes auf OpenShell](../docs/concepts/agent-sandboxes.md)
-beschrieben. Man gibt nido einen Ordner, nido lässt darin einen Agenten
-arbeiten und holt die Ergebnisse zurück. Jede Sandbox läuft in einer eigenen
-Kata-VM.
+Run coding agents and commands in isolated sandboxes. Give nido a folder, it lets an agent work
+on it in a sandbox and brings the results back.
 
-Stand: geschlossene Beta (#686–#688). nido spricht mit `pirol-agents-api`
-unter `https://nido.pirol.ai`. Die API prüft den Pirol-Schlüssel, trennt die
-Kunden und reicht die Aufrufe an das OpenShell-Gateway weiter. Die Doku für
-Kunden steht unter [pirol.ai/docs/sandboxes](https://pirol.ai/docs/sandboxes).
+Every sandbox is its own small virtual machine (Kata Containers) in Frankfurt, run by
+[Pirol](https://pirol.ai).
 
-## Einrichten
+> **Closed beta.** Free during the beta, for invited accounts only, up to 5 sandboxes per account.
+> Ask for access at [hello@pirol.ai](mailto:hello@pirol.ai).
 
-Voraussetzung: Node 20 oder neuer und ein Pirol-Konto, das für die Beta
-freigeschaltet ist (`BETA_ALLOWLIST` in
-[deploy-agents-api.sh](../infra/agent-sandbox/deploy-agents-api.sh)).
+Full documentation: [pirol.ai/docs/sandboxes](https://pirol.ai/docs/sandboxes)
+
+## Install
+
+Requires Node 20 or newer.
 
 ```bash
 npm install -g @pirol/nido
 nido login
 ```
 
-`nido login` läuft wie `kyte login`: Der Browser öffnet pirol.ai, man
-bestätigt, und nido legt den Schlüssel in `~/.nido/config.json` ab (nur für
-den Nutzer lesbar). Ohne Browser hilft `nido login --manual`. In CI reicht
-die Umgebungsvariable `PIROL_API_KEY`. `NIDO_API_URL` zeigt nido auf eine
-andere API, etwa eine lokale.
+`nido login` opens your browser. You confirm with your Pirol account, and nido stores a key in
+`~/.nido/config.json` (readable only by you). Without a browser, use `nido login --manual`. In CI,
+set `PIROL_API_KEY` instead.
 
-Aus dem Repo statt von npm:
+## Run an agent on a folder
 
 ```bash
-cd nido && npm install && npm run build && npm link
+cd my-project
+nido init                   # writes nido.json
+nido secret set anthropic   # once; asks for the key without echo
+nido run
 ```
 
-## Ordner mit `nido.json`
+`nido run`:
 
-```bash
-mkdir survey && cd survey
-cp ~/Downloads/results.csv .
-nido init                       # writes nido.json for Claude Code
-nido secret set anthropic       # once, asks for the key without echo
-nido run
-```
+1. Creates the sandbox if it does not exist yet. The name comes from `name` or the folder name.
+   A new sandbox is ready in about 20 seconds.
+2. Uploads changed files. Unchanged files are not uploaded again; files you deleted locally are
+   deleted in the sandbox too.
+3. Runs the agent and shows each step on one line.
+4. Brings `output/` back. A file you changed locally is not overwritten; the sandbox version is
+   saved next to it, for example as `report.remote.md`.
+
+The sandbox keeps running afterwards. The next `nido run` reuses it, `nido run --fresh` starts
+over. `nido pull` fetches `output/` again.
 
-`nido.json`:
+### nido.json
 
 ```json
 {
@@ -57,40 +59,39 @@ nido run
 }
 ```
 
-Was `nido run` macht:
-
-1. Sandbox anlegen, falls es sie noch nicht gibt. Der Name kommt aus
-   `name` oder dem Ordnernamen. Ein neuer Start dauert etwa 20 Sekunden.
-2. Geänderte Dateien hochladen. nido merkt sich die Hashes in
-   `.nido/state.json`. Unveränderte Dateien gehen nicht noch einmal hoch,
-   lokal gelöschte verschwinden auch in der Sandbox.
-3. Den Agenten starten. Bei Claude zeigt nido jeden Schritt in einer Zeile.
-4. `output/` zurückholen. Neue Dateien landen im Ordner. Eine Datei, die
-   man lokal geändert hat, überschreibt nido nicht. Die Fassung aus der
-   Sandbox heißt dann zum Beispiel `report.remote.md`.
-
-Die Sandbox bleibt danach stehen. Der nächste `nido run` nutzt sie weiter,
-`nido run --fresh` fängt neu an.
-
-Statt eines Agenten geht auch ein Befehl, das braucht keinen Schlüssel:
+Instead of an agent, run any command. It needs no key:
 
 ```json
 { "command": "python analyze.py", "output": "output/" }
 ```
 
-Felder:
+| Field | Meaning |
+| --- | --- |
+| `agent` | `claude`; more will follow |
+| `task` | What the agent should do |
+| `command` | Instead of `agent`: a command as a string (runs in bash) or a list |
+| `name` | Sandbox name; defaults to the folder name |
+| `image` | Your own container image; defaults to an image with Claude Code, Codex and OpenCode |
+| `files.exclude` | Patterns like in `.gitignore`; `.git` and `.nido` are always excluded |
+| `secrets` | Names of stored secrets, never values |
+| `network` | Hosts the sandbox may reach on port 443; everything else is blocked |
+| `storage` | Storage volumes, mounted at `/data/<name>` |
+| `output` | Folder that comes back after the run; default `output/` |
 
-| Feld | Bedeutung |
-|---|---|
-| `agent` | `claude`, weitere folgen |
-| `task` | Auftrag an den Agenten |
-| `command` | statt `agent`: Befehl als Text (läuft in bash) oder Liste |
-| `name` | Name der Sandbox, sonst der Ordnername |
-| `image` | eigenes Image, sonst das OpenShell-Community-Image mit Claude Code, Codex und OpenCode |
-| `files.exclude` | Muster wie in `.gitignore`. `.git` und `.nido` bleiben immer draußen |
-| `secrets` | Namen von Secrets, nie die Werte |
-| `network` | Hosts, die die Sandbox auf Port 443 erreichen darf. Sonst ist das Netz zu |
-| `output` | Ordner, den nido zurückholt, Standard `output/` |
+## Storage
+
+Keep large data in storage, independently of sandboxes:
+
+```bash
+nido storage put ./data my-data   # upload straight to Cloud Storage in Frankfurt
+nido storage ls                   # volumes with file count and size
+nido storage ls my-data           # files in one volume
+nido storage rm my-data           # only once no sandbox uses it
+```
+
+Add `"storage": ["my-data"]` to `nido.json` (or `nido create shop --storage my-data`), and the
+sandbox reads and writes the data at `/data/my-data`, without copying. Volumes are attached when a
+sandbox is created; after changing `storage`, run `nido run --fresh`.
 
 ## Secrets
 
@@ -102,14 +103,10 @@ nido secret ls
 nido secret rm github
 ```
 
-Der Wert geht per HTTPS an die API und von dort über eine
-Umgebungsvariable an das Gateway, nie als Argument. Lesen kann ihn danach
-niemand mehr, auch der Kunde nicht.
-In der Sandbox steht nur ein Platzhalter. Der Proxy von OpenShell setzt den
-echten Wert erst beim Aufruf ein, und nur für die Hosts des Dienstes, etwa
-`api.anthropic.com`. Andere Namen kennt nido noch nicht.
+A secret is stored once and can never be read again, not even by you. Inside the sandbox the agent
+only sees a placeholder; the real value is added on the way to the provider's API.
 
-## Weitere Befehle
+## More commands
 
 ```bash
 nido create shop --secret anthropic --network pypi.org
@@ -120,26 +117,27 @@ nido cp shop:/sandbox/work/output ./out
 nido logs shop
 nido stop shop && nido start shop
 nido rm shop
+nido whoami
+nido logout
 ```
 
-## Entwicklung
+Paths inside a sandbox must be below `/sandbox`.
 
-```bash
-npm test           # unit tests
-npm run build      # dist/
-```
+## Limits in the beta
 
-Die Regeln für Hoch- und Herunterladen stecken in `src/sync.ts` und sind
-reine Funktionen. Die Aufrufe an die API liegen in `src/api.ts`, die Anmeldung
-in `src/login.ts`. Mit OpenShell spricht nur noch
-[agents-api](../agents-api).
+- Up to 5 sandboxes per account, and 10 storage volumes with 100 GB in total.
+- No interactive shell yet; use `nido exec`.
+- The agent runs in the foreground. If the connection drops, `nido pull` fetches what is already
+  in `output/`.
+- No backups or templates of a sandbox yet. Keep anything important in `output/` or in storage.
 
-Veröffentlichen: `npm publish` im Ordner `nido` (baut vorher neu).
+## Development
 
-## Grenzen
+```bash
+cd nido && npm install && npm run build && npm link
+npm test
+```
 
-- Höchstens 5 Sandboxes pro Konto in der Beta.
-- `nido shell` fehlt noch. Stattdessen `nido exec`.
-- Der Agent läuft im Vordergrund. Bricht die Verbindung ab, holt
-  `nido pull` das, was schon in `output/` liegt.
-- Keine dauerhaften Arbeitsbereiche und kein Backup (#695).
+`NIDO_API_URL` points nido at another API, for example a local agents-api. Upload and
+download rules live in `src/sync.ts` as pure functions, API calls in `src/api.ts`, login in
+`src/login.ts`. Publish with `npm publish` in this folder; it builds and tests first.

package/dist/api.js

@@ -1,6 +1,8 @@
 // Client for pirol-agents-api. Files travel as tar.gz streams, command
 // output comes back as NDJSON while the command runs.
 import { spawn } from "node:child_process";
+import { createReadStream } from "node:fs";
+import { request } from "node:https";
 import { Readable } from "node:stream";
 import { loadConfig } from "./config.js";
 export class ApiError extends Error {
@@ -152,4 +154,37 @@ export class Api {
     removeSecret(name) {
         return this.json("DELETE", `/v1/secrets/${encodeURIComponent(name)}`);
     }
+    storageList() {
+        return this.json("GET", "/v1/storage");
+    }
+    storageFiles(name) {
+        return this.json("GET", `/v1/storage/${encodeURIComponent(name)}`);
+    }
+    async storageCreate(name) {
+        return (await this.json("PUT", `/v1/storage/${encodeURIComponent(name)}`)).result;
+    }
+    /** One upload URL per file; each takes exactly the given number of bytes. */
+    async storageUploads(name, files) {
+        return (await this.json("POST", `/v1/storage/${encodeURIComponent(name)}/uploads`, { files })).uploads;
+    }
+    storageRemove(name) {
+        return this.json("DELETE", `/v1/storage/${encodeURIComponent(name)}`);
+    }
+}
+/** Sends a local file to an upload URL from storageUploads (straight to Cloud Storage). */
+export function putFile(url, file, size) {
+    return new Promise((resolve, reject) => {
+        const req = request(url, { method: "PUT", headers: { "content-length": size } }, (res) => {
+            let body = "";
+            res.on("data", (d) => (body += d.toString("utf8")));
+            res.on("end", () => res.statusCode === 200 || res.statusCode === 201
+                ? resolve()
+                : reject(new Error(`upload failed: HTTP ${res.statusCode} ${body.slice(0, 200)}`)));
+        });
+        req.on("error", reject);
+        if (size === 0)
+            req.end();
+        else
+            createReadStream(file).on("error", reject).pipe(req);
+    });
 }

package/dist/cli.js

@@ -2,6 +2,7 @@
 import { readFileSync, realpathSync } from "node:fs";
 import { fileURLToPath } from "node:url";
 import { copy, createSandbox, exec, init, list, login, logout, pull, runFolder, secretList, secretRemove, secretSet, simple, whoami, } from "./commands.js";
+import { storageLs, storagePut, storageRm } from "./storage.js";
 const HELP = `nido — run agents in isolated sandboxes (closed beta)
 
 Start
@@ -14,13 +15,18 @@ Folder workflow
   nido pull                    bring output/ back again
 
 Sandboxes
-  nido create <name> [--image <ref>] [--secret <name>]… [--network <host>]…
+  nido create <name> [--image <ref>] [--secret <name>]… [--network <host>]… [--storage <name>]…
   nido ls                      list your sandboxes
   nido exec <name> -- <cmd>…   run a command
   nido cp <src> <dst>          copy files; write the sandbox side as <name>:<path>
   nido logs <name>             sandbox logs
   nido stop|start|rm <name>
 
+Storage (lives on without sandboxes, mounted at /data/<name>)
+  nido storage put <folder|file> <name>   upload, straight to Cloud Storage
+  nido storage ls [name]
+  nido storage rm <name>
+
 Secrets (stored once, never readable again)
   nido secret set <name> [--from-gh]   anthropic, openai, github
   nido secret ls
@@ -69,6 +75,7 @@ async function main(argv) {
                 image: flags.get("image")?.[0],
                 secrets: flags.get("secret") ?? [],
                 network: flags.get("network") ?? [],
+                storage: flags.get("storage") ?? [],
             });
             return 0;
         case "ls":
@@ -93,6 +100,8 @@ async function main(argv) {
             return 0;
         case "secret":
             return secret(args, flags);
+        case "storage":
+            return storage(args);
         default:
             throw new Error(`Unknown command "${cmd}". See: nido help`);
     }
@@ -109,7 +118,19 @@ async function secret(args, flags) {
         throw new Error("Usage: nido secret set <name> | nido secret ls | nido secret rm <name>");
     return 0;
 }
-const VALUE_FLAGS = new Set(["image", "secret", "network"]);
+async function storage(args) {
+    const [sub, a, b] = args;
+    if (sub === "put" && a && b)
+        await storagePut(a, b);
+    else if (sub === "ls")
+        await storageLs(a);
+    else if (sub === "rm" && a)
+        await storageRm(a);
+    else
+        throw new Error("Usage: nido storage put <folder|file> <name> | nido storage ls [name] | nido storage rm <name>");
+    return 0;
+}
+const VALUE_FLAGS = new Set(["image", "secret", "network", "storage"]);
 export function parse(argv) {
     const flags = new Map();
     const args = [];

package/dist/commands.js

@@ -164,7 +164,13 @@ async function ensureSandbox(api, m, fresh) {
     }
     if (!info) {
         say(`Creating sandbox ${m.name}…`);
-        const created = await api.create({ name: m.name, image: m.imageSet ? m.image : undefined, secrets: m.secrets, network: m.network });
+        const created = await api.create({
+            name: m.name,
+            image: m.imageSet ? m.image : undefined,
+            secrets: m.secrets,
+            network: m.network,
+            storage: m.storage,
+        });
         say(`Sandbox ${m.name} ready in ${(created.ready_ms / 1000).toFixed(1)} s`);
         return true;
     }

package/dist/manifest.js

@@ -25,7 +25,7 @@ export function parseManifest(raw, folderName) {
     if (!raw || typeof raw !== "object" || Array.isArray(raw))
         throw new Error(`${MANIFEST} must be an object`);
     const m = raw;
-    const known = ["name", "agent", "task", "command", "image", "files", "secrets", "network", "output"];
+    const known = ["name", "agent", "task", "command", "image", "files", "secrets", "network", "storage", "output"];
     const unknown = Object.keys(m).filter((key) => !known.includes(key) && !key.startsWith("$"));
     if (unknown.length)
         throw new Error(`${MANIFEST}: unknown field ${unknown.map((k) => `"${k}"`).join(", ")}`);
@@ -65,6 +65,11 @@ export function parseManifest(raw, folderName) {
         exclude,
         secrets: stringList(m, "secrets"),
         network: stringList(m, "network"),
+        storage: stringList(m, "storage").map((name) => {
+            if (!isVolumeName(name))
+                throw new Error(`${MANIFEST}: "${name}" in "storage" is not a valid name`);
+            return name;
+        }),
         output,
     };
 }
@@ -78,6 +83,10 @@ export function sandboxName(input) {
         .replace(/-+$/, "");
     return name || "nest";
 }
+/** Storage names, like sandbox names: lowercase letters, digits and dashes, at most 40. */
+export function isVolumeName(name) {
+    return /^[a-z0-9]([a-z0-9-]{0,38}[a-z0-9])?$/.test(name);
+}
 function optionalString(m, key) {
     const value = m[key];
     if (value === undefined)

package/dist/storage.js

@@ -0,0 +1,115 @@
+// nido storage: volumes in Cloud Storage that live independently of
+// sandboxes. Files go straight to Cloud Storage; the API only hands out an
+// upload URL per file.
+import { existsSync, statSync } from "node:fs";
+import { basename, dirname, join, resolve } from "node:path";
+import { Api, putFile } from "./api.js";
+import { excludeMatcher, listFiles } from "./files.js";
+import { isVolumeName } from "./manifest.js";
+const say = (text) => void process.stderr.write(text + "\n");
+const BATCH = 500;
+const PARALLEL = 8;
+export function checkVolumeName(name) {
+    if (!isVolumeName(name))
+        throw new Error(`"${name}" is not a valid storage name: lowercase letters, digits and dashes, at most 40 characters`);
+    return name;
+}
+/** Local files to upload: a folder's contents, or one file. Paths use "/". */
+export function localFiles(source) {
+    const full = resolve(source);
+    if (!existsSync(full))
+        throw new Error(`${source} does not exist`);
+    if (statSync(full).isFile())
+        return { root: dirname(full), files: [{ path: basename(full), size: statSync(full).size }] };
+    const paths = listFiles(full, excludeMatcher([]));
+    return { root: full, files: paths.map((path) => ({ path, size: statSync(join(full, path)).size })) };
+}
+/** nido storage put <folder|file> <name> */
+export async function storagePut(source, name) {
+    checkVolumeName(name);
+    const { root, files } = localFiles(source);
+    if (!files.length)
+        throw new Error(`${source} has no files`);
+    const api = Api.fromConfig();
+    if ((await api.storageCreate(name)) === "created")
+        say(`Created storage ${name}`);
+    const total = files.reduce((sum, f) => sum + f.size, 0);
+    say(`Uploading ${files.length} file${files.length === 1 ? "" : "s"} (${size(total)}) to ${name}…`);
+    let done = 0;
+    let sent = 0;
+    const started = Date.now();
+    const progress = () => {
+        if (process.stderr.isTTY)
+            process.stderr.write(`\r  ${done}/${files.length} files, ${size(sent)} of ${size(total)}   `);
+    };
+    for (let i = 0; i < files.length; i += BATCH) {
+        const batch = files.slice(i, i + BATCH);
+        const queue = await api.storageUploads(name, batch);
+        const bySize = new Map(batch.map((f) => [f.path, f.size]));
+        const worker = async () => {
+            for (let item = queue.shift(); item; item = queue.shift()) {
+                const bytes = bySize.get(item.path);
+                await uploadWithRetry(api, name, item, join(root, item.path), bytes);
+                done += 1;
+                sent += bytes;
+                progress();
+            }
+        };
+        await Promise.all(Array.from({ length: PARALLEL }, worker));
+    }
+    if (process.stderr.isTTY)
+        process.stderr.write("\n");
+    const seconds = (Date.now() - started) / 1000;
+    say(`Done in ${seconds.toFixed(1)} s. Sandboxes see it at /data/${name} with "storage": ["${name}"] in nido.json`);
+}
+async function uploadWithRetry(api, name, item, file, bytes) {
+    let url = item.url;
+    for (let attempt = 1;; attempt++) {
+        try {
+            return await putFile(url, file, bytes);
+        }
+        catch (err) {
+            if (attempt === 3)
+                throw new Error(`${item.path}: ${err.message}`);
+            // A failed session cannot take the whole file again; ask for a new one.
+            url = (await api.storageUploads(name, [{ path: item.path, size: bytes }]))[0].url;
+        }
+    }
+}
+/** nido storage ls [name] */
+export async function storageLs(name) {
+    const api = Api.fromConfig();
+    if (name) {
+        const volume = await api.storageFiles(checkVolumeName(name));
+        for (const f of volume.files)
+            process.stdout.write(`${size(f.size).padStart(10)}  ${f.path}\n`);
+        if (volume.total_files > volume.files.length)
+            say(`… ${volume.total_files - volume.files.length} more files`);
+        say(`${volume.total_files} files, ${size(volume.bytes)}`);
+        return;
+    }
+    const { volumes, limit_bytes } = await api.storageList();
+    if (!volumes.length)
+        return say("No storage yet. Upload a folder with: nido storage put ./data my-data");
+    const width = Math.max(4, ...volumes.map((v) => v.name.length));
+    process.stdout.write(`${"NAME".padEnd(width)}  ${"FILES".padStart(7)}  ${"SIZE".padStart(10)}\n`);
+    for (const v of volumes)
+        process.stdout.write(`${v.name.padEnd(width)}  ${String(v.files).padStart(7)}  ${size(v.bytes).padStart(10)}\n`);
+    const used = volumes.reduce((sum, v) => sum + v.bytes, 0);
+    say(`${size(used)} of ${size(limit_bytes)} used`);
+}
+/** nido storage rm <name> */
+export async function storageRm(name) {
+    await Api.fromConfig().storageRemove(checkVolumeName(name));
+    say(`Removed storage ${name}`);
+}
+export function size(bytes) {
+    const units = ["B", "KB", "MB", "GB", "TB"];
+    let value = bytes;
+    let unit = 0;
+    while (value >= 1024 && unit < units.length - 1) {
+        value /= 1024;
+        unit++;
+    }
+    return `${unit === 0 ? value : value.toFixed(1)} ${units[unit]}`;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pirol/nido",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Run agents in isolated sandboxes. Give it a folder, get results back.",
   "type": "module",
   "bin": {