@crvouga/dev-fleet 0.2.0 → 0.3.0

package/cliproxy/README.md

@@ -146,7 +146,7 @@ devfleet cliproxy inference responses EXACT_CODEX_MODEL_ID
 devfleet cliproxy inference messages EXACT_CLAUDE_MODEL_ID
 ```
 
-Codex uses the upstream device-code flow. Remote callback flows may require the documented dashboard callback flow or temporary SSH loopback forwarding; callback ports are not exposed by default. This pinned version has no generic Gemini CLI login flag; use its supported dashboard/API-key integration. Provider rules still govern permissible account use.
+`devfleet fleet-accounts --add|--relogin|--remove` wraps these and propagates the result to every node; a login writes into a staging directory and only a complete credential is moved in. Codex uses the upstream device-code flow. Remote callback flows may require the documented dashboard callback flow or temporary SSH loopback forwarding; callback ports are not exposed by default. This pinned version has no generic Gemini CLI login flag; use its supported dashboard/API-key integration. Provider rules still govern permissible account use.
 
 **Account routing.** Credential files are the only list of accounts. `ops declared` (run by every fleet-sync) gives each Claude, Codex and Antigravity credential the model prefix `<type>-<email>` through the proxy's management API (`scripts/accounts.py`; a credential file is never edited directly). A request for `<type>-<email>/<model>` is served only by that account, and `/v1/models` lists those ids, which is how every fleet node discovers the accounts to offer in T3. Nothing in Git names an account.
 
@@ -167,7 +167,7 @@ python3 cliproxy/clients/bootstrap.py --validate
 
 Enter the inference key at the hidden prompt. It is available in Vault `secret/personal/prd:CLIPROXY_FLEET_CLIENT_API_KEY`; dashboard login uses `CLIPROXY_FLEET_MANAGEMENT_KEY`. Secrets never appear in command arguments. Keys are saved mode 0600 and loaded only into child environments. Give runners separate revocable inference keys where needed, never provider OAuth credentials or the admin key.
 
-Codex uses custom-provider CLI overrides and Responses HTTP/SSE. Claude uses its supported base URL/auth-token variables and an explicit Claude model. OpenCode uses its config-file override, with separate Responses and Anthropic SDK providers. Once Claude is authenticated, rerun with `--claude-model MODEL_ID`; otherwise the Claude launcher fails with an explicit setup requirement. Native client config files are preserved. Use plain native clients during a proxy outage.
+Codex uses custom-provider CLI overrides and Responses HTTP/SSE. Claude uses its supported base URL/auth-token variables and an explicit Claude model. OpenCode uses its config-file override, with separate Responses and Anthropic SDK providers. Once Claude is authenticated, rerun with `--claude-model MODEL_ID`; otherwise the Claude launcher fails with an explicit setup requirement. Native client config files are preserved. Fleet nodes never fall back to native clients: provider accounts exist only on the proxy (see [docs/accounts.md](../docs/accounts.md)), and during a proxy outage every relayed launcher retries for about a minute, then reports "proxy unreachable".
 
 Run bootstrap from a repository checkout containing `clients/launcher.py`; detached downloaded bootstrap files are unsupported. Bootstrap is idempotent and refuses unrelated launcher collisions. `--uninstall` removes launcher links while retaining private settings for deliberate cleanup. On Windows run inside WSL; native PowerShell launchers are not provided. Validate from each actual machine and target repository because OpenCode merges project settings.
 
@@ -187,7 +187,7 @@ Vault login is interactive with hidden input; its token is restricted locally an
 
 Encrypted cloud checkpoints run every 15 minutes (legacy systemd adds up to 30 seconds jitter). A checkpoint never stops, pauses or restarts the stack: agents use the proxy around the clock, and a stopped connector fails every request. The capture copies the live state and keeps the copy only if no state file changed while it was taken (retrying a few times); a stack that never holds still fails that backup. A failed capture, restart, upload or retention operation records failure and never advances the success marker. Retention keeps the latest 96 snapshots plus 7 daily, 5 weekly and 12 monthly. Weekly integrity checks read 10% of stored data; `check-full` reads all data. Monitoring fails when the last successful checkpoint is older than 45 minutes. Failures are visible in `cliproxy status` (and `launchctl print gui/$UID/LABEL` or systemd); off-machine checks also run through the CLI proxy health workflow described below.
 
-R2 checkpoints are the durable recovery source for config, provider OAuth/account state and the dedicated Tailscale identity. Local state is a working cache with asynchronous durability: normally up to approximately 15 minutes plus upload duration can be lost, and an outage can extend that indefinitely. Run a successful manual checkpoint after important login/config changes or before moving machines. Provider refresh-token invalidation can still require a fresh provider login after recovering an older checkpoint. This is a single-writer deployment, with operator fencing of the previous node; there is no distributed lease or active-active support.
+R2 checkpoints are the durable recovery source for config, provider OAuth/account state and the dedicated Tailscale identity. Local state is a working cache with asynchronous durability: normally up to approximately 15 minutes plus upload duration can be lost, and an outage can extend that indefinitely. Run a successful manual checkpoint after important login/config changes or before moving machines. Provider refresh-token invalidation can still require a fresh provider login after recovering an older checkpoint. Each checkpoint records a digest of every refresh token, so `restore-verify RESTORED --newest NEWEST` names exactly the accounts that need one. This is a single-writer deployment, with operator fencing of the previous node; there is no distributed lease or active-active support.
 
 Checkpoints include only allowlisted runtime state, endpoint intent, repository identity and source revision metadata. GitHub supplies code; Vault supplies client/admin keys, the encryption password and current cloud credentials. Vault/tunnel tokens, local secret files, logs, prior config copies, status files and executable source are excluded. Config and OAuth files themselves contain secrets and remain encrypted in R2. Local capture/restore directories are private plaintext caches, deleted after normal completion. Abandoned generated `checkpoint-*` and `restore-*` directories under `recovery/` expire after 24 hours. Cleanup runs under the operation lock before each backup operation, so active local captures/restores cannot be collected. Unknown/manual rollback directories are preserved. Restic runs without a persistent local metadata cache.
 

package/cliproxy/scripts/accounts.py

@@ -36,3 +36,219 @@ def pending(auths):
         if wanted and data.get('prefix') != wanted:
             result.append((path.name, wanted))
     return result
+
+
+# ── Account state ────────────────────────────────────────────────────────────
+#
+# One state per credential, from its file (never a token, only whether one is
+# there and when the access token expires) and from the proxy's management
+# API, which knows about refreshes, cooldowns and disabled credentials.
+
+STATES = ('usable', 'cooling-down', 'needs-login', 'refresh-failing', 'removed')
+
+# A refresh the provider refused for good: only a new sign-in helps.
+REJECTED = re.compile(r'invalid_grant|invalid_token|unauthorized|revoked|refresh token (?:is )?(?:invalid|expired|rejected|reused)', re.I)
+
+
+def _instant(value):
+    """Seconds since the epoch from an ISO 8601 string or a number, or None."""
+    import datetime
+    if isinstance(value, (int, float)) and not isinstance(value, bool) and value > 0:
+        return float(value) / (1000 if value > 1e11 else 1)
+    if isinstance(value, str) and value and not value.startswith('0001-'):
+        try:
+            parsed = datetime.datetime.fromisoformat(value.replace('Z', '+00:00'))
+        except ValueError:
+            return None
+        if parsed.tzinfo is None:
+            parsed = parsed.replace(tzinfo=datetime.timezone.utc)
+        return parsed.timestamp()
+    return None
+
+
+def iso(seconds):
+    import datetime
+    return datetime.datetime.fromtimestamp(seconds, datetime.timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
+
+
+def classify(credential, entry, now):
+    """(state, until): `credential` is the auth file, `entry` what the
+    management API says about it (None when it does not list it). `until` is
+    when a cooldown clears, as epoch seconds, or None."""
+    entry = entry or {}
+    if entry.get('disabled') is True or credential.get('disabled') is True:
+        return 'removed', None
+    message = str(entry.get('status_message') or entry.get('last_error') or '')
+    expires = _instant(credential.get('expired') or credential.get('expiry'))
+    expired = expires is not None and expires <= now
+    has_refresh = bool(credential.get('refresh_token'))
+    if REJECTED.search(message) or (expired and not has_refresh):
+        return 'needs-login', None
+    if entry.get('status') == 'error' and 'refresh' in message.lower():
+        return ('needs-login' if expired else 'refresh-failing'), None
+    quota = entry.get('quota') if isinstance(entry.get('quota'), dict) else {}
+    until = max(
+        [t for t in (_instant(entry.get('next_retry_after')),
+                     _instant(quota.get('next_recover_at'))) if t is not None and t > now]
+        or [0]
+    )
+    if until or entry.get('unavailable') is True or quota.get('exceeded') is True:
+        return 'cooling-down', (until or None)
+    return 'usable', None
+
+
+def credentials(auths):
+    """[(file name, credential)] for every readable credential, in a stable
+    order: by provider, then sign-in email, then file name. Each provider's
+    accounts are numbered in this order, the same order every node lists
+    them in (src/agents/accounts.ts), so an index means one account
+    everywhere."""
+    found = []
+    for path in pathlib.Path(auths).glob('*.json'):
+        try:
+            data = json.loads(path.read_text())
+        except (OSError, ValueError):
+            continue
+        if isinstance(data, dict) and isinstance(data.get('type'), str):
+            found.append((path.name, data))
+    email = lambda data: data.get('email') if isinstance(data.get('email'), str) else ''
+    return sorted(found, key=lambda item: (item[1]['type'], email(item[1]), item[0]))
+
+
+def roster(auths, entries=(), now=None):
+    """Every provider account: provider, opaque index (1, 2, … per provider),
+    label, state and, while cooling down, when it clears. `entries` is the
+    management API's list of auth files. Never a token."""
+    import time
+    now = time.time() if now is None else now
+    by_name = {e.get('name'): e for e in entries if isinstance(e, dict)}
+    counters = {}
+    result = []
+    for name, credential in credentials(auths):
+        provider = credential['type']
+        counters[provider] = counters.get(provider, 0) + 1
+        state, until = classify(credential, by_name.get(name), now)
+        email = credential.get('email')
+        result.append({
+            'provider': provider,
+            'index': counters[provider],
+            'label': email if isinstance(email, str) else provider + '-' + str(counters[provider]),
+            'prefix': prefix(provider, email),
+            'state': state,
+            'until': iso(until) if until else None,
+            'file': name,
+        })
+    return result
+
+
+def version(accounts):
+    """The roster version: a digest of the non-secret metadata of the accounts
+    every node offers (their routing prefixes). Every node computes the same
+    digest from the proxy's model list (src/accounts/roster.ts)."""
+    import hashlib
+    prefixes = sorted({a['prefix'] for a in accounts if a.get('prefix') and a.get('state') != 'removed'})
+    return hashlib.sha256('\n'.join(prefixes).encode()).hexdigest()[:16]
+
+
+def public(accounts):
+    """The roster as reported: no file name."""
+    return [{k: v for k, v in a.items() if k != 'file'} for a in accounts]
+
+
+def find(accounts, provider, index):
+    """The one account `provider` `index` names, or None."""
+    for account in accounts:
+        if account['provider'] == provider and account['index'] == index:
+            return account
+    return None
+
+
+# ── Sign-in, re-login and removal ────────────────────────────────────────────
+#
+# A login writes into a staging directory, never into the auth directory. Only
+# a complete credential (readable JSON with a type) is moved in, with one
+# rename, so an interrupted login leaves no half-written auth file.
+
+def staging(auths):
+    return pathlib.Path(auths).parent / 'auths-staging'
+
+
+def begin(auths):
+    """A fresh, empty staging directory; leftovers of an interrupted login are
+    discarded."""
+    import shutil
+    stage = staging(auths)
+    shutil.rmtree(stage, ignore_errors=True)
+    stage.mkdir(mode=0o700, parents=True)
+    return stage
+
+
+def adopt(stage, auths, replace=None):
+    """Moves every complete credential from `stage` into `auths` and returns
+    their file names. `replace` (a file name) is the credential a re-login
+    signs in again: it is replaced by the new one in the same rename. The
+    staging directory is removed whatever happens."""
+    import os
+    import shutil
+    stage, auths = pathlib.Path(stage), pathlib.Path(auths)
+    adopted = []
+    try:
+        complete = []
+        for path in sorted(stage.glob('*.json')):
+            try:
+                data = json.loads(path.read_text())
+            except (OSError, ValueError):
+                continue
+            if isinstance(data, dict) and isinstance(data.get('type'), str):
+                complete.append(path)
+        if replace and len(complete) != 1:
+            raise ValueError('A re-login must produce exactly one credential')
+        for path in complete:
+            target = auths / (replace or path.name)
+            os.chmod(path, 0o600)
+            os.replace(path, target)
+            adopted.append(target.name)
+        if replace and adopted and adopted[0] != replace:
+            (auths / replace).unlink(missing_ok=True)
+        return adopted
+    finally:
+        shutil.rmtree(stage, ignore_errors=True)
+
+
+def remove(auths, provider, index):
+    """Deletes the credential of one account; returns its file name."""
+    account = find(roster(auths), provider, index)
+    if account is None:
+        raise ValueError('No such account')
+    (pathlib.Path(auths) / account['file']).unlink()
+    return account['file']
+
+
+# ── Refresh tokens across checkpoints ────────────────────────────────────────
+#
+# Providers rotate refresh tokens. Each checkpoint records a short digest of
+# every credential's refresh token (never the token). After a restore, a
+# credential whose digest differs from the newest checkpoint's carries a
+# token the provider has already replaced: that account needs a new login.
+
+def token_digests(auths):
+    import hashlib
+    digests = {}
+    for name, credential in credentials(auths):
+        token = credential.get('refresh_token')
+        if isinstance(token, str) and token:
+            digests[name] = hashlib.sha256(token.encode()).hexdigest()[:16]
+    return digests
+
+
+def stale_after_restore(auths, newest):
+    """[(provider, index)] of the restored accounts whose refresh token is
+    older than the one `newest` (the token digests of the newest checkpoint)
+    records."""
+    current = token_digests(auths)
+    stale = []
+    for account in roster(auths):
+        name = account['file']
+        if name in newest and name in current and current[name] != newest[name]:
+            stale.append((account['provider'], account['index']))
+    return stale

package/cliproxy/scripts/checkpoint.py

@@ -127,11 +127,39 @@ def copy_state(root, target):
             shutil.copyfile(source, destination)
     shutil.copyfile(root / "INSTALLATION.json", target / "INSTALLATION.json")
     (target / "CHECKPOINT.json").write_text(
-        json.dumps({"version": 1, "files": hashes(target)}) + "\n"
+        json.dumps(
+            {
+                "version": 1,
+                "files": hashes(target),
+                "tokens": _accounts().token_digests(target / "state/runtime/auths"),
+            }
+        )
+        + "\n"
     )
     validate(target)
 
 
+def _accounts():
+    """accounts.py beside this file, however this module was loaded."""
+    import importlib.util
+
+    spec = importlib.util.spec_from_file_location(
+        "accounts", pathlib.Path(__file__).resolve().parent / "accounts.py"
+    )
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def stale_accounts(restored, newest):
+    """[(provider, index)] of the accounts in the restored checkpoint
+    `restored` whose refresh token the provider has since replaced, judged
+    against the manifest of the newest checkpoint `newest`. Restoring the
+    newest checkpoint itself finds none."""
+    tokens = json.loads((newest / "CHECKPOINT.json").read_text()).get("tokens") or {}
+    return _accounts().stale_after_restore(restored / "state/runtime/auths", tokens)
+
+
 def capture(root, target, attempts=6, sleep=time.sleep):
     """Copies the state of the running stack without ever stopping it: agents
     use the proxy around the clock, and a stopped connector fails every one of

package/cliproxy/scripts/cli.py

@@ -21,7 +21,7 @@ root = cliproxy_host.default_root().resolve()
 args = sys.argv[1:]
 if not args or args[0] in ["help", "--help", "-h"]:
     print(
-        "Usage: devfleet cliproxy COMMAND\nCommands: install [--revision REV|--rollback|--dry-run|--prepare-only], audit, public-endpoint [enable|start|disable|verify], status, start, stop, restart, health, models, logs, validate, apply FILE, add-account PROVIDER, inference PROTOCOL MODEL, model-health [--profile liveness|readiness|light|nightly|manual] [--suite core|cursor|all] [--target live|mock], model-suite (same flags; defaults to nightly), backup, gc, snapshots, check, check-full, restore SNAPSHOT EMPTY_DIRECTORY, restore-cache --previous-node-stopped, vault-login, recover, recover-node NEW_DIRECTORY [--vault-token-file FILE], verify-recovery, portable-login, activate, drain"
+        "Usage: devfleet cliproxy COMMAND\nCommands: install [--revision REV|--rollback|--dry-run|--prepare-only], audit, public-endpoint [enable|start|disable|verify], status, start, stop, restart, health, models, logs, validate, apply FILE, add-account PROVIDER, relogin PROVIDER INDEX, remove-account PROVIDER INDEX, accounts [--json], client-keys, issue-key|rotate-key|retire-key|revoke-key NODE, inference PROTOCOL MODEL, model-health [--profile liveness|readiness|light|nightly|manual] [--suite core|cursor|all] [--target live|mock], model-suite (same flags; defaults to nightly), backup, gc, snapshots, check, check-full, restore SNAPSHOT EMPTY_DIRECTORY, restore-cache --previous-node-stopped, vault-login, recover, recover-node NEW_DIRECTORY [--vault-token-file FILE], verify-recovery, portable-login, activate, drain"
     )
     sys.exit(0)
 command, *rest = args

package/cliproxy/scripts/clientkeys.py

@@ -0,0 +1,99 @@
+"""One proxy client key per node, rotated with overlap and revoked with the node.
+
+The key ring is a private JSON document in the proxy's secrets
+(`state/secrets/client-keys.json`): per node, the current key and, during a
+rotation, the previous one. The proxy accepts every key in the ring
+(`api_keys`), so a rotation adds the new key, every node moves to it, and
+only then is the old key retired: no request fails in between. Revoking a
+node removes its keys and leaves every other node working. Reports name keys
+only by a short fingerprint, never by value.
+"""
+import hashlib
+import json
+import os
+import pathlib
+import re
+import secrets
+
+NODE = re.compile(r'^[a-z][a-z0-9-]{0,62}$')
+
+
+def fingerprint(key):
+    return hashlib.sha256(key.encode()).hexdigest()[:12]
+
+
+def load(path):
+    try:
+        ring = json.loads(pathlib.Path(path).read_text())
+    except (OSError, ValueError):
+        return {'version': 1, 'nodes': {}}
+    if not isinstance(ring, dict) or ring.get('version') != 1:
+        raise ValueError('Unsupported client key ring')
+    return ring
+
+
+def save(path, ring):
+    path = pathlib.Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    staged = path.with_suffix('.new')
+    staged.write_text(json.dumps(ring, indent=2, sort_keys=True) + '\n')
+    os.chmod(staged, 0o600)
+    os.replace(staged, path)
+
+
+def _node(ring, node):
+    if not NODE.match(node):
+        raise ValueError('Invalid node id')
+    return ring['nodes'].setdefault(node, {'current': None, 'previous': None})
+
+
+def issue(ring, node, generate=lambda: secrets.token_urlsafe(32)):
+    """Gives `node` a key when it has none; returns its current key."""
+    entry = _node(ring, node)
+    if not entry['current']:
+        entry['current'] = generate()
+    return entry['current']
+
+
+def rotate(ring, node, generate=lambda: secrets.token_urlsafe(32)):
+    """Starts a rotation: a new current key, the old one kept as previous
+    until `retire`. Both are accepted meanwhile."""
+    entry = _node(ring, node)
+    if entry['previous']:
+        raise ValueError('A rotation of this node is already in progress; retire it first')
+    entry['previous'], entry['current'] = entry['current'], generate()
+    return entry['current']
+
+
+def retire(ring, node):
+    """Ends a rotation once the node uses its new key."""
+    _node(ring, node)['previous'] = None
+
+
+def revoke(ring, node):
+    """Removes every key of `node`."""
+    ring['nodes'].pop(node, None)
+
+
+def api_keys(ring, shared=()):
+    """Every key the proxy accepts: the ring's, plus `shared` keys still in
+    use (the fleet-wide key, until each node has its own)."""
+    keys = list(shared)
+    for node in sorted(ring['nodes']):
+        entry = ring['nodes'][node]
+        for key in (entry['current'], entry['previous']):
+            if key and key not in keys:
+                keys.append(key)
+    return keys
+
+
+def report(ring):
+    """Per node: the current key's fingerprint and whether a rotation is
+    still waiting to be retired. No key value."""
+    return {
+        node: {
+            'current': fingerprint(entry['current']) if entry['current'] else None,
+            'rotating': bool(entry['previous']),
+        }
+        for node, entry in sorted(ring['nodes'].items())
+    }

package/cliproxy/scripts/lease.py

@@ -0,0 +1,80 @@
+"""A fencing lease on the proxy state: only its holder is the writer.
+
+Providers rotate refresh tokens, so two proxies started from the same
+checkpoint spend each other's tokens. The lease is one small JSON document in
+a store both would see (the checkpoint bucket in production, a directory in
+tests): the holder, a fencing epoch that only ever grows, and when it
+expires. A proxy that cannot take the lease refuses to start as writer.
+"""
+import json
+import os
+import pathlib
+import time
+
+TTL = 120
+
+
+class Held(Exception):
+    """Another proxy holds the lease."""
+
+    def __init__(self, holder, expires):
+        super().__init__('The proxy state lease is held by another proxy')
+        self.holder = holder
+        self.expires = expires
+
+
+class DirectoryStore:
+    """A lease document in a directory, replaced with one rename. `put`
+    writes only when the document is still the one `get` returned."""
+
+    def __init__(self, directory):
+        self.path = pathlib.Path(directory) / 'proxy-lease.json'
+
+    def get(self):
+        try:
+            return json.loads(self.path.read_text())
+        except (OSError, ValueError):
+            return None
+
+    def put(self, document, expected):
+        import fcntl
+        self.path.parent.mkdir(parents=True, exist_ok=True)
+        with open(self.path.with_suffix('.lock'), 'w') as lock:
+            fcntl.flock(lock, fcntl.LOCK_EX)
+            if self.get() != expected:
+                return False
+            staged = self.path.with_suffix('.new')
+            staged.write_text(json.dumps(document) + '\n')
+            os.replace(staged, self.path)
+            return True
+
+
+def acquire(store, holder, now=None, ttl=TTL):
+    """Takes or renews the lease for `holder`; returns its fencing epoch.
+    Raises Held while another holder's lease has not expired. Taking over an
+    expired lease increases the epoch, so the old holder's writes are fenced."""
+    now = time.time() if now is None else now
+    current = store.get()
+    if current and current.get('holder') != holder and current.get('expires', 0) > now:
+        raise Held(current.get('holder'), current.get('expires'))
+    epoch = (current or {}).get('epoch', 0)
+    if not current or current.get('holder') != holder:
+        epoch += 1
+    document = {'holder': holder, 'epoch': epoch, 'expires': now + ttl}
+    if not store.put(document, current):
+        raise Held(None, None)
+    return epoch
+
+
+def valid(store, holder, epoch, now=None):
+    """Whether `holder` still holds the lease at `epoch`: checked before a
+    token refresh is written."""
+    now = time.time() if now is None else now
+    current = store.get() or {}
+    return current.get('holder') == holder and current.get('epoch') == epoch and current.get('expires', 0) > now
+
+
+def release(store, holder):
+    current = store.get()
+    if current and current.get('holder') == holder:
+        store.put({**current, 'expires': 0}, current)