google-colab-cli 0.6.0__tar.gz → 0.7.3__tar.gz

{google_colab_cli-0.6.0 → google_colab_cli-0.7.3}/AGENTS.md

@@ -6,10 +6,9 @@
 - **Client**: `ColabClient` handles API interactions (assignment, unassignment).
 - **Auth**: `auth.py` exposes a single `get_credentials(config_path, provider)` facade that dispatches on the `AuthProvider` enum. Two providers are supported, selected via the global `--auth=oauth2|adc` flag (default `oauth2`):
   - `oauth2`: public `google-auth-oauthlib` `InstalledAppFlow`, token cached at `~/.config/colab-cli/token.json`. Reads the client OAuth config from `-c/--client-oauth-config` (default `~/.colab-cli-oauth-config.json`), falling back to the **bundled** `src/colab_cli/oauth_config.json` resource (re-added in PR #41 / `9f44fe2`, 2026-05-29 — the earlier "removed in `20eb88e`" note was stale/incorrect; the file exists and `auth.py:_get_google_auth_credentials` loads it via `importlib.resources`). As of 2026-06-11 the flow is a **remote copy-paste flow**, not a localhost server: `_run_remote_flow` sets `redirect_uri=https://sdk.cloud.google.com/applicationdefaultauthcode.html` + `token_usage=remote`, prints the URL, and reads the pasted code via `input()`. NEVER revert to OOB (`urn:ietf:wg:oauth:2.0:oob`) — Google blocked it in 2022 ("OOB flow has been blocked"); the `sdk.cloud.google.com` redirect is registered only to the bundled cloud-SDK client (`764086051850-...`), so any other client id gets `redirect_uri_mismatch`. Server-side acceptance/rejection of these variants is verifiable GET-only by building the authorization URL and inspecting whether Google reaches sign-in vs. an OAuth error page (no resources allocated).
-  - `adc`: Google Application Default Credentials via `google.auth.default()`. The CLI passes `scopes=PUBLIC_SCOPES` (which includes `colaboratory`) and re-applies via `creds.with_scopes()` for credential types that support it. **User credentials minted by `gcloud auth application-default login` ignore the `scopes=` kwarg AND raise `NotImplementedError` on `with_scopes`**: ADC users must explicitly re-authenticate with `gcloud auth application-default login --scopes=openid,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/colaboratory`. `userinfo.email` is required by the session backend at `colab.research.google.com` (assign/unassign/sessions/keep-alive return 401 without it); `colaboratory` is retained for forward compatibility and other Colab features (keep-alive no longer uses `colab.pa.googleapis.com` — see the Keep-alive note below); `openid` and `cloud-platform` are mandated by `gcloud` itself, which rejects scope lists that omit `cloud-platform` with `Invalid value for [--scopes]`. Service-account / GCE / GKE / impersonated creds get the right scopes transparently via `with_scopes`.
+  - `adc`: Google Application Default Credentials via `google.auth.default()`. The CLI passes `scopes=PUBLIC_SCOPES` (which includes `colaboratory`) and re-applies via `creds.with_scopes()` for credential types that support it. **User credentials minted by `gcloud auth application-default login` ignore the `scopes=` kwarg AND raise `NotImplementedError` on `with_scopes`**: ADC users must explicitly re-authenticate with `gcloud auth application-default login --scopes=openid,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/colaboratory`. `userinfo.email` is required by the session backend at `colab.research.google.com` (assign/unassign/sessions return 401 without it); `colaboratory` is retained for forward compatibility and other Colab features; `openid` and `cloud-platform` are mandated by `gcloud` itself, which rejects scope lists that omit `cloud-platform` with `Invalid value for [--scopes]`. Service-account / GCE / GKE / impersonated creds get the right scopes transparently via `with_scopes`.
 - **Backend Hosts**: Two distinct backends with different requirements:
-  - `colab.research.google.com` (session backend / `tun/m/...`): accepts the `userinfo.email` scope. Handles assign, unassign, the contents API, **and keep-alive** (see below).
-  - **Keep-alive (2026-06-15, issue #14)**: keep-alive is a **Tunnel Frontend (TFE) HTTP ping** — `GET https://colab.research.google.com/tun/m/<endpoint>/keep-alive/` with header `X-Colab-Tunnel: Google`, authenticated by the user's own Gaia bearer token (same host/credential as `assign`). TFE records `LastActiveTime` before forwarding, refreshing the idle timer. The VM usually doesn't answer on this path, so the request commonly **read-times-out even on success** — `client.keep_alive_assignment` therefore catches `requests.exceptions.ReadTimeout` and treats it as success, while genuine HTTP errors (e.g. 404 for a deleted assignment) propagate. This mirrors the official `colab-vscode` extension's `sendKeepAlive` (`src/colab/client.ts`). **DO NOT revert to the `colab.pa.googleapis.com` `RuntimeService/KeepAliveAssignment` RPC**: that RPC requires the caller to be a `serviceusage` consumer of Colab's internal project `1014160490159`, which no ordinary user account is, so it returned HTTP 403 `USER_PROJECT_DENIED` for every external user (issue #14) and silently idle-pruned their sessions within minutes. The browser only made that RPC work by riding the user's `google.com` cookie through an internal cookie-proxy (`colab.clients6.google.com`), which a headless bearer-token CLI cannot use. Verified live 2026-06-15 with a third-party account (the RPC 403'd; the tunnel ping succeeded and kept the VM alive).
+  - `colab.research.google.com` (session backend / `tun/m/...`): accepts the `userinfo.email` scope. Handles assign, unassign, CCU info, and the contents API.
 - **Runtime**: `ColabRuntime` wraps `jupyter-kernel-client` for execution.
 - **State**:
   - `StateStore` persists session metadata in `~/.config/colab-cli/sessions.json`.
@@ -42,6 +41,21 @@
 - **Files**: `ls`, `rm`, `upload`, `download`, `edit`.
 - **Automation**: `auth`, `drivemount`, `install`, `log`, `pay`, `version`, `update`.
 
+## Release Tagging Workflow
+The package version is derived from the git tag via `hatch-vcs` (see `pyproject.toml`), so a release is just (1) a `CHANGELOG.md` entry and (2) a `vX.Y.Z` tag on the merged commit. Follow this workflow ONLY when the user explicitly requests a release (e.g. "cut a release", "tag v0.7.0"). NEVER propose or perform a release proactively.
+
+1. **Verify a clean, current `main`**: `git fetch origin && git checkout main && git pull --ff-only origin main`. Confirm `git status` is clean. If `main` has diverged or there are unmerged feature branches the user expects in the release, stop and ask.
+2. **Identify unreleased commits**: Find the last release tag with `git describe --tags --abbrev=0`, then list commits since then with `git log <last-tag>..HEAD --oneline`. These are what the new changelog section must cover. Cross-reference each commit's PR number (the `(#NN)` suffix in the merge commit subject) — every entry in the changelog should cite at least one PR.
+3. **Infer the next version (semver)**: Categorize each commit using the Keep-a-Changelog buckets already present in `CHANGELOG.md` (`Added`, `Changed`, `Fixed`, `Removed`, `Deprecated`, `Security`). Pick the bump:
+   - **Major** (`X.0.0`): any breaking change to the CLI surface, flag semantics, on-disk state schema (`sessions.json` / `settings.json`), or persisted token format.
+   - **Minor** (`0.X.0`): at least one `Added` entry (new subcommand, new flag, new capability) with no breaking changes.
+   - **Patch** (`0.0.X`): only `Fixed`, `Changed`, `Removed` (non-breaking cleanup), `Deprecated`, or `Security` entries.
+   Confirm the inferred version with the user before proceeding — never tag without that confirmation, even when the bump seems obvious.
+4. **Draft and commit the CHANGELOG entry on a branch**: Create a release branch (e.g. `release-v0.7.0`), then prepend a new `## [X.Y.Z] - YYYY-MM-DD` section above the previous release. Group entries by category in the order already established in `CHANGELOG.md` (`Changed`, `Added`, `Fixed`, `Removed`). Each bullet should be one short paragraph naming the user-visible behavior change (not the implementation detail) and ending with the PR reference `(#NN)`. Append a new link reference at the bottom: `[X.Y.Z]: https://github.com/googlecolab/google-colab-cli/compare/v<PREV>...vX.Y.Z`. Commit with message `docs: add CHANGELOG.md for vX.Y.Z release`, push the branch, and open a PR with `gh pr create`.
+5. **Wait for the changelog PR to merge**: Do NOT tag the pre-merge commit on your branch. The tag must land on the squash-merge commit that appears on `main` (this is what `hatch-vcs` will see and what users will `git checkout`). After the user reports the PR is merged (or you observe it via `gh pr view <num> --json state,mergeCommit`), `git fetch origin && git checkout main && git pull --ff-only origin main`.
+6. **Tag the merged changelog commit and push the tag**: Verify `git log -1 --oneline` is the merged changelog commit (subject matches `docs: add CHANGELOG.md for vX.Y.Z release (#NN)`). Create an annotated tag whose message is the changelog section body: `git tag -a vX.Y.Z -m "vX.Y.Z" -m "<changelog section body>"`. Push with `git push origin vX.Y.Z`. Do NOT create a GitHub Release entry via `gh release create` — the tag alone is sufficient because `hatch-vcs` reads it directly, and the canonical release notes already live in `CHANGELOG.md`.
+7. **Verify**: Run `git describe --tags --exact-match HEAD` (should print `vX.Y.Z`) and `uv run colab version` (the version string should match). Confirm with `gh api repos/googlecolab/google-colab-cli/tags --jq '.[0].name'` that the tag is visible on the remote.
+
 ## Implementation Principles
 1.  **Direct Execution**: Code for `auth`, `drivemount`, etc., should be injected and executed on the VM kernel.
 2.  **Contents API**: Use the Jupyter Contents API for file management as seen in the browser traces.
@@ -49,23 +63,20 @@
 4.  **No netrc**: Avoid `netrc` for token persistence in this project.
 5.  **Mocking Interactivity**: When testing commands that branch on `stdin.isatty()`, use the `is_stdin_tty` helper in `execution.py` and mock it via `mocker.patch("colab_cli.commands.execution.is_stdin_tty", return_value=...)`. This ensures tests don't hang in CI/agent environments.
 6.  **State Isolation**: Always patch the `colab_cli.common.state` singleton in tests to control session persistence and client behavior. Refer to `tests/conftest.py` for the standard global fixture.
-7.  **Fire-and-Forget Architecture**: The Colab CLI is a "fire-and-forget" tool. Avoid using background threads for long-running tasks within the main command flows. For persistent needs such as keep-alive, utilize detached background daemon processes (with PID tracking in the session state).
+7.  **Fire-and-Forget Architecture**: The Colab CLI is a "fire-and-forget" tool. Avoid using background threads or persistent daemon processes within command flows. Commands authenticate, perform their task, and exit cleanly.
 8.  **Verify the Local Install**: A globally-installed `colab` may exist on `PATH` (e.g. at `~/.local/bin/colab`) and can shadow the project's editable install when `uv run` is invoked from outside the repo. ALWAYS run shell commands with the repo as the working directory (e.g. via the `workdir` parameter, never `cd && cmd`) so `uv run colab ...` resolves to `.venv/bin/colab`. Confirm with `which colab` and `uv run which colab` if a CLI test produces unexpected results (e.g. flag-not-recognized errors for flags you just added). **Shebang invocations always resolve via `$PATH`**, so a script like `#!/usr/bin/env -S colab run ...` will pick up the stale global tool even when the editable install is current — when testing shebang-based behavior after a code change, always run `uv tool install --reinstall --force --from . colab` first, then verify with `colab version` (the version string includes the git short SHA). Encoded 2026-05-12 after the SystemExit-suppression fix appeared not to work in `examples/hello_colab.py` because the shebang resolved to a uv-tool install pinned to the prior commit.
 9.  **Isolate the Regression First**: When a user reports an error in code you just touched, do NOT assume your change caused it. First, reproduce the failure on `main` (or the branch point) to determine whether the bug is pre-existing. Only after confirming the regression is yours should you start debugging the new code. Encoded after spending a turn debugging "ADC broke `colab new`" only to discover `colab new --gpu A100` was already failing on `main` due to an A100-quota-vs-default issue unrelated to ADC.
 10. **Live Probes Allocate Real Resources**: Probing the Colab API to debug an issue creates real, billable assignments — every successful POST `/tun/m/assign` reserves a VM. Prefer GET-only (read) probes whenever possible. For any state-mutating call, (a) record every endpoint you create as you go, and (b) clean up via `client.unassign(endpoint)` (or `colab stop`) before declaring the investigation done. Then verify with `colab sessions` that nothing was orphaned.
 11. **Push Freshness**: The remote may have advanced during a session (other contributors land commits while you work). ALWAYS `git fetch <remote>` immediately before pushing or merging. If `git log main..<remote>/main` is non-empty, reset local `main` to the remote, rebase feature branches onto it, retest, then push. NEVER force-push `main` to recover from divergence.
 12. **Amend Safety**: Before `git commit --amend`, explicitly verify all three preconditions: (a) the user requested amend OR a pre-commit hook auto-modified files for an otherwise-successful commit, (b) HEAD was created by you in this conversation (`git log -1 --format='%an %ae'`), (c) the commit has not been pushed to a remote. If any precondition fails, create a new commit instead. NEVER amend a failed/rejected commit — fix the issue and create a new one.
-13. **Run Integration Tests Yourself**: Re-read AGENTS.md "Agent Execution Limitations" before claiming you can't run a test. The CANNOT-run list is exclusively interactive commands (`colab auth`, `colab drivemount`, `colab repl`, `colab console`). Tests built on `colab new` / `colab stop` / `colab log` are non-interactive and you MUST run them yourself before declaring a fix complete. Encoded after running through a full implement-and-document cycle for a fix that the integration test would have falsified in 30 seconds — the cure was `uv run bash integration/repro_keep_alive/test.sh`.
+13. **Run Integration Tests Yourself**: Re-read AGENTS.md "Agent Execution Limitations" before claiming you can't run a test. The CANNOT-run list is exclusively interactive commands (`colab auth`, `colab drivemount`, `colab repl`, `colab console`). Tests built on `colab new` / `colab stop` / `colab log` are non-interactive and you MUST run them yourself before declaring a fix complete. Encoded after running through a full implement-and-document cycle for a fix that the integration test would have falsified in 30 seconds — the cure was `uv run bash integration/repro_run_command/test.sh`.
 14. **Heed Research Caveats**: When a research subagent surfaces a caveat ("the proto allows it but the policy may reject"), treat it as a TODO to verify, not as a footnote. The validation pattern: shell out to the actual service with the proposed inputs and confirm the response matches expectations BEFORE writing the code that depends on it. For policy-gated paths, run a one-shot probe before committing to a design.
 15. **Two distinct auths — never confuse them**: This codebase has two unrelated authentication concerns: (a) **CLI-to-Colab-control-plane**: how `Client` authenticates HTTP requests to `colab.research.google.com` and `colab.pa.googleapis.com`. Driven by the global `--auth={oauth2,adc}` flag and `auth.py:get_credentials`. The OAuth2 path is bootstrapped automatically by `google_auth_oauthlib`'s `InstalledAppFlow` on first invocation when `~/.config/colab-cli/token.json` doesn't exist; **no separate command is needed** — any `--auth=oauth2` invocation (e.g. `colab --auth=oauth2 sessions`) triggers the browser consent flow. (b) **VM-side credentials**: how the `colab auth` subcommand injects user GCP credentials *into* the running Colab kernel so notebook code (e.g. `gcloud`, BigQuery client) can make authenticated calls from inside the VM. This uses the `USE_AUTH_EPHEM='0'` gcloud-fallback path executed on the kernel via `ColabRuntime`. **NEVER tell a user to "run `colab auth`" as a prerequisite for fixing CLI-side auth issues** — they're orthogonal layers and the suggestion misleads.
-16. **Detached daemons inherit nothing useful**: When spawning a detached child process via `subprocess.Popen` (e.g. `spawn_keep_alive`), the child does NOT inherit the parent's parsed Typer flags. It re-parses argv from scratch, so any global flag the parent saw via `--auth=adc` or `--config /tmp/foo.json` MUST be re-emitted as part of the child's command line. Forgetting this causes silent fallback to Typer defaults: in 2026-04-30 this manifested as the keep-alive daemon (a) using OAUTH2 instead of the parent's ADC, and (b) reading from `~/.config/colab-cli/sessions.json` instead of the parent's `--config` path. Always propagate every relevant global flag in `cmd = [sys.executable, "-m", ...]` BEFORE the subcommand name (Typer requires global flags before the subcommand).
-17. **Persist-before-spawn for daemons that read shared state**: When a detached child reads from a state file the parent owns (e.g. `state.store.get(session_name)`), the parent MUST persist BEFORE spawning. Otherwise the child can race ahead of the parent's `add()` call and observe an empty store. Symptom in 2026-04-30: keep-alive daemon exits immediately with `keep_alive_stopped reason=session_not_found iters=1 duration=0.0s`. Fix: call `state.store.add(s)` once before `spawn_keep_alive`, and again after (to capture the PID).
-18. **[SUPERSEDED 2026-06-15 — keep-alive no longer calls `colab.pa.googleapis.com`; see issue #14 / the Keep-alive note above] `colab.pa.googleapis.com` rejects ADC user creds without `X-Goog-User-Project`**: When calling `colab.pa.googleapis.com` from ADC user credentials minted by `gcloud auth application-default login`, the bearer token carries the user's gcloud quota project. The colab API key sent alongside it belongs to a different project (Colab, `1014160490159`). The backend enforces project-match between the two and returns HTTP 400 `CONSUMER_INVALID` ("The API Key and the authentication credential are from different projects."). The old fix was to send `X-Goog-User-Project: 1014160490159` to pin the consumer project — but that in turn required `serviceusage.serviceUsageConsumer` on project `1014160490159`, which ordinary users lack, yielding HTTP 403 `USER_PROJECT_DENIED` (the issue #14 root cause). Both failure modes are now moot because keep-alive uses the TFE tunnel ping on `colab.research.google.com` instead. Retained here as institutional knowledge: if any future code path must call `colab.pa.googleapis.com` with a bearer token, it will face this same project-entitlement wall for non-internal accounts.
-19. **Pydantic validation requires a schema**: `_issue_request` accepts an optional `schema=` and historically called `TypeAdapter(schema).validate_python(...)` unconditionally. When `schema=None` (a caller that doesn't care about the response body — e.g. fire-and-forget RPCs like `KeepAliveAssignment` that return `[]`), this raises `pydantic.ValidationError: Input should be None`. Always guard with `if schema is None: return` after the empty-body short-circuit.
-20. **Suggest the branch-diff review command after committing**: The user reviews changes with `git diff main..<branch-name>` (full cumulative diff against `main`, not just the latest commit). After landing one or more commits on a feature branch, ALWAYS suggest the exact command — e.g. "Review with `git diff main..sort-help-commands`" — instead of `git show <sha>` (which only shows a single commit and misses context when a branch has multiple commits). Encoded 2026-05-05 after suggesting `git show 9d9c7da` for a branch the user wanted to review holistically.
-21. **Verify research-tool claims with primary sources**: Research tools and AI assistants can be confidently wrong, especially about edge cases or features outside their training corpus. When such a tool says "X is not used / not parsed / doesn't exist", treat it as a hypothesis to verify, not a fact. Always cross-check against the primary source (the actual code or config) — and when a tool names files, check whether the indirection chain it describes actually exists. The cost of believing the tool when it's wrong is shipping a non-functional feature; the cost of double-checking is small. Encoded 2026-05-05 after a `colab url` first-cut shipped the wrong URL format because of unverified output.
-22. **Clean up orphaned assignments before finishing live tests**: After running a live integration test, `colab sessions` may show server-side assignments that the local `colab stop` couldn't see (e.g. assignments leaked from earlier in the session, or from crashed prior runs). Always run `colab sessions` as the final cleanup step, and for any `[?]`-marked orphan, run `python -c "from colab_cli.common import state; state.client.unassign('<endpoint>')"` (the CLI doesn't expose a direct unassign-by-endpoint command). Re-verify with `colab sessions` returning "No active sessions". Encoded 2026-05-05 after the first `colab url` live test left an orphan from a prior conversation turn that would have idled-out and billed compute units.
-23. **Forward unknown args through Typer with `context_settings`**: Typer/Click consumes any token starting with `-`/`--` as a flag of the parent command unless told otherwise. For a subcommand like `colab run script.py --some-script-flag` (where `--some-script-flag` belongs to the user's script, not to `colab`), declare it with `app.command(name="run", context_settings={"allow_extra_args": True, "ignore_unknown_options": True})` and accept the positional with `Annotated[Optional[List[str]], typer.Argument(...)] = None`. Also use `repr()` (not f-string interpolation) when embedding those forwarded strings into kernel-side Python source — `repr()` produces a safe round-trippable literal regardless of the user's shell-passed quotes, backslashes, or non-ASCII bytes. Encoded 2026-05-12 while implementing `colab run`.
+16. **Pydantic validation requires a schema**: `_issue_request` accepts an optional `schema=` and historically called `TypeAdapter(schema).validate_python(...)` unconditionally. When `schema=None` (a caller that doesn't care about the response body — e.g. unassign or fire-and-forget endpoints that return `[]`), this raises `pydantic.ValidationError: Input should be None`. Always guard with `if schema is None: return` after the empty-body short-circuit.
+17. **Suggest the branch-diff review command after committing**: The user reviews changes with `git diff main..<branch-name>` (full cumulative diff against `main`, not just the latest commit). After landing one or more commits on a feature branch, ALWAYS suggest the exact command — e.g. "Review with `git diff main..sort-help-commands`" — instead of `git show <sha>` (which only shows a single commit and misses context when a branch has multiple commits). Encoded 2026-05-05 after suggesting `git show 9d9c7da` for a branch the user wanted to review holistically.
+18. **Verify research-tool claims with primary sources**: Research tools and AI assistants can be confidently wrong, especially about edge cases or features outside their training corpus. When such a tool says "X is not used / not parsed / doesn't exist", treat it as a hypothesis to verify, not a fact. Always cross-check against the primary source (the actual code or config) — and when a tool names files, check whether the indirection chain it describes actually exists. The cost of believing the tool when it's wrong is shipping a non-functional feature; the cost of double-checking is small. Encoded 2026-05-05 after a `colab url` first-cut shipped the wrong URL format because of unverified output.
+19. **Clean up orphaned assignments before finishing live tests**: After running a live integration test, `colab sessions` may show server-side assignments that the local `colab stop` couldn't see (e.g. assignments leaked from earlier in the session, or from crashed prior runs). Always run `colab sessions` as the final cleanup step, and for any `[?]`-marked orphan, run `python -c "from colab_cli.common import state; state.client.unassign('<endpoint>')"` (the CLI doesn't expose a direct unassign-by-endpoint command). Re-verify with `colab sessions` returning "No active sessions". Encoded 2026-05-05 after the first `colab url` live test left an orphan from a prior conversation turn that would have idled-out and billed compute units.
+20. **Forward unknown args through Typer with `context_settings`**: Typer/Click consumes any token starting with `-`/`--` as a flag of the parent command unless told otherwise. For a subcommand like `colab run script.py --some-script-flag` (where `--some-script-flag` belongs to the user's script, not to `colab`), declare it with `app.command(name="run", context_settings={"allow_extra_args": True, "ignore_unknown_options": True})` and accept the positional with `Annotated[Optional[List[str]], typer.Argument(...)] = None`. Also use `repr()` (not f-string interpolation) when embedding those forwarded strings into kernel-side Python source — `repr()` produces a safe round-trippable literal regardless of the user's shell-passed quotes, backslashes, or non-ASCII bytes. Encoded 2026-05-12 while implementing `colab run`.
 
 ## Agent Execution Limitations (What I Can vs Cannot Run)
 As an AI agent operating via non-interactive shell tools (`run_shell_command`), there are strict limits on what I can test autonomously without human intervention:

{google_colab_cli-0.6.0 → google_colab_cli-0.7.3}/CHANGELOG.md

@@ -8,6 +8,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 The package version is derived from the git tag via `hatch-vcs`; each release
 below corresponds to a tag of the same name.
 
+## [0.7.1] - 2026-09-14
+
+### Fixed
+
+- **build:** Pin core metadata to version 2.4 for both the wheel and the sdist.
+  hatchling 1.32.0 bumped its default to 2.5, which our publishing pipeline
+  rejects, so 0.7.0 was built but never reached PyPI. `colab ssh` (added in
+  0.7.0) was therefore unavailable to anyone installing from PyPI.
+
+## [0.7.0] - 2026-09-03
+
+### Changed
+
+- **deps:** Bump cryptography from 49.0.0 to 50.0.0 (#98), and pyasn1 from 0.6.3 to 0.6.4 (#87).
+- **lockfile:** Upgrade lockfile dependencies via `uv lock --upgrade`. (#67)
+- **skill:** Move the CLI help skill details into a dedicated `SKILL.md`. (#55)
+- **docs:** Update `AGENTS.md` with instructions for the release tagging workflow. (#66)
+
+### Added
+
+- **ssh:** Add `colab ssh` command to provide secure SSH-over-WebSocket direct access to the Colab runtime VM. (#88)
+- **execution:** Add `--env KEY=VALUE` flag to `colab run` and `colab exec` commands to allow injecting custom environment variables into the running kernel. (#65)
+- **session:** Add `--high-mem` flag to machine shape selection when creating a new session to support high-RAM resources. (#105)
+
+### Fixed
+
+- **session:** Surface a friendly error message on 412 GPU/TPU allocation failure instead of printing a raw traceback. (#112)
+- **runtime:** Support both `ColabKernelClient` and standard `KernelClient` in the runtime layer to improve compatibility across environments. (#95)
+
 ## [0.6.0] - 2026-06-16
 
 ### Changed
@@ -47,4 +76,5 @@ below corresponds to a tag of the same name.
   `colaboratory`-scope / `pa.googleapis.com` pre-flight remediation messaging,
   superseded by the TFE keep-alive ping. (#61)
 
+[0.7.0]: https://github.com/googlecolab/google-colab-cli/compare/v0.6.0...v0.7.0
 [0.6.0]: https://github.com/googlecolab/google-colab-cli/compare/v0.5.11...v0.6.0

{google_colab_cli-0.6.0 → google_colab_cli-0.7.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-colab-cli
-Version: 0.6.0
+Version: 0.7.3
 Summary: CLI for interacting with Colab.
 Project-URL: Homepage, https://github.com/googlecolab/google-colab-cli
 Project-URL: Repository, https://github.com/googlecolab/google-colab-cli
@@ -55,7 +55,7 @@ Designed to support seamless developer productivity, headless automation, and AI
 * **Instant VM Provisioning:** Spin up CPU, GPU (T4, L4, G4, H100, A100), or TPU (v5e1, v6e1) runtimes in seconds.
 * **Robust Code Execution:** Run local Python scripts, Jupyter Notebooks (`.ipynb`), or piped `stdin` code; launch interactive REPLs or raw TTY console shells.
 * **Ephemeral Job Runner (`colab run`):** Provision a fresh VM, execute a local script with forwarded arguments, retrieve output files, and automatically tear down the runtime in a single command.
-* **Automatic Keep-Alive:** Built-in background daemon automatically prevents idle VM termination, keeping resource allocations active without requiring open browser tabs.
+* **Activity-Based Liveness:** Sessions stay alive as long as the kernel is active, with the Colab backend automatically maintaining liveness.
 * **Seamless Workspace Automation:** Mount Google Drive, authenticate Google Cloud Platform (GCP) credentials, and install dependencies with high-performance `uv` package management.
 * **State & Log Archival:** Inspect local session states or export interactive history logs to standard Jupyter Notebooks, Markdown, or structured JSONL.
 
@@ -104,20 +104,21 @@ Run `colab <command> --help` to view specific options, defaults, and detailed he
 ### Session Management
 | Command | Description |
 | --- | --- |
-| `colab new [-s NAME] [--gpu GPU] [--tpu TPU]` | Allocate a new CPU, GPU, or TPU VM runtime |
+| `colab new [-s NAME] [--gpu GPU] [--tpu TPU] [--high-mem]` | Allocate a new CPU, GPU, or TPU VM runtime (optionally high-RAM) |
 | `colab sessions` | List all active sessions currently active on the backend |
-| `colab status [-s NAME]` | Display hardware, status, and local metadata for active sessions |
+| `colab status [-s NAME]` | Display hardware, machine shape, status, and local metadata for active sessions |
 | `colab restart-kernel [-s NAME]` | Restart the active session's Jupyter kernel |
-| `colab stop [-s NAME]` | Terminate a session VM and tear down its keep-alive daemon |
+| `colab stop [-s NAME]` | Terminate a session VM and release its resources |
 | `colab url [-s NAME] [--open]` | Print or open a browser URL connecting to the active session |
 
 ### Execution
 | Command | Description |
 | --- | --- |
-| `colab run [--gpu GPU] [--tpu TPU] [--keep] SCRIPT [ARGS...]` | Run a local script on a fresh VM, forwarding arguments, then release it |
+| `colab run [--gpu GPU] [--tpu TPU] [--high-mem] [--keep] SCRIPT [ARGS...]` | Run a local script on a fresh VM, forwarding arguments, then release it |
 | `colab exec [-s NAME] [-f FILE] [--output-image PATH]` | Execute Python code from stdin, a local `.py` file, or a `.ipynb` notebook |
 | `colab repl [-s NAME] [--output-image PATH]` | Start an interactive Python REPL on the VM (exits cleanly on piped EOF) |
 | `colab console [-s NAME]` | Connect to a raw interactive TTY shell (tmux) on the remote VM |
+| `colab ssh [-s NAME] [--proxy-mode] [-i KEY] [--gpu GPU] [--tpu TPU] [--high-mem]` | Open an SSH shell to the runtime over WebSocket, or act as an OpenSSH `ProxyCommand` bridge for IDE remote-dev |
 
 ### File Operations
 | Command | Description |
@@ -135,6 +136,7 @@ Run `colab <command> --help` to view specific options, defaults, and detailed he
 | `colab drivemount [-s NAME] [PATH]` | Mount Google Drive on the VM (default: `/content/drive`) |
 | `colab install [-s NAME] [-r FILE \| PKG...]` | Install packages on the VM using `uv` (falls back to `pip`) |
 | `colab log [-s NAME] [-n N] [-o FILE]` | View or export session history (`.ipynb`, `.md`, `.txt`, `.jsonl`) |
+| `colab usage` | Show account compute-unit usage rate and balance |
 | `colab pay` | Open the Colab subscription page to manage compute units |
 | `colab version` | Print the installed version of the CLI |
 | `colab update [--install]` | Check for a newer release (and optionally upgrade the CLI in place) |
@@ -177,6 +179,7 @@ colab stop -s analysis
 
 ## Usage Notes
 
+* **Machine shape:** Use `--high-mem` with `colab new`, `colab run`, or `colab ssh` (when auto-creating a runtime) to request a high-RAM machine shape. Requires Colab Pro or Pro+ entitlement for supported accelerators (CPU, T4, A100, etc.). L4 and TPU runtimes ignore this flag because they only offer one shape. Machine shape is shown in `colab sessions` and `colab status`.
 * **TTY Requirements:** The interactive commands `repl` and `console` require a local TTY. When running inside automated scripts or pipelines, make sure to pipe stdin (e.g., `echo "print(1)" | colab repl`) to trigger non-interactive execution modes.
 * **Transparent Code Execution:** When calling `colab exec -f file.py`, the CLI reads the file locally and transmits its content to the remote kernel. You do not need to manually upload files before execution.
 * **Storage & State Paths:** Session tokens and metadata are stored at `~/.config/colab-cli/sessions.json`. Global CLI settings are located at `~/.config/colab-cli/settings.json`. These can be customized or isolated via the global `--config` flag.
@@ -210,7 +213,7 @@ Make the script executable (`chmod +x script.py`) and run it: `./script.py`. The
 
 For comprehensive architectural overviews and deep-dives into specific CLI sub-systems, refer to the detailed documentation:
 
-* [Session Management & Keep-Alive Architecture](docs/01_session_management.md)
+* [Session Management Architecture](docs/01_session_management.md)
 * [Interactive & Non-Interactive Execution Design](docs/02_execution_and_interactive.md)
 * [File Management & Jupyter Contents API](docs/03_file_management.md)
 * [Authentication Providers & VM Automation](docs/04_automation_and_utility.md)

{google_colab_cli-0.6.0 → google_colab_cli-0.7.3}/README.md

@@ -19,7 +19,7 @@ Designed to support seamless developer productivity, headless automation, and AI
 * **Instant VM Provisioning:** Spin up CPU, GPU (T4, L4, G4, H100, A100), or TPU (v5e1, v6e1) runtimes in seconds.
 * **Robust Code Execution:** Run local Python scripts, Jupyter Notebooks (`.ipynb`), or piped `stdin` code; launch interactive REPLs or raw TTY console shells.
 * **Ephemeral Job Runner (`colab run`):** Provision a fresh VM, execute a local script with forwarded arguments, retrieve output files, and automatically tear down the runtime in a single command.
-* **Automatic Keep-Alive:** Built-in background daemon automatically prevents idle VM termination, keeping resource allocations active without requiring open browser tabs.
+* **Activity-Based Liveness:** Sessions stay alive as long as the kernel is active, with the Colab backend automatically maintaining liveness.
 * **Seamless Workspace Automation:** Mount Google Drive, authenticate Google Cloud Platform (GCP) credentials, and install dependencies with high-performance `uv` package management.
 * **State & Log Archival:** Inspect local session states or export interactive history logs to standard Jupyter Notebooks, Markdown, or structured JSONL.
 
@@ -68,20 +68,21 @@ Run `colab <command> --help` to view specific options, defaults, and detailed he
 ### Session Management
 | Command | Description |
 | --- | --- |
-| `colab new [-s NAME] [--gpu GPU] [--tpu TPU]` | Allocate a new CPU, GPU, or TPU VM runtime |
+| `colab new [-s NAME] [--gpu GPU] [--tpu TPU] [--high-mem]` | Allocate a new CPU, GPU, or TPU VM runtime (optionally high-RAM) |
 | `colab sessions` | List all active sessions currently active on the backend |
-| `colab status [-s NAME]` | Display hardware, status, and local metadata for active sessions |
+| `colab status [-s NAME]` | Display hardware, machine shape, status, and local metadata for active sessions |
 | `colab restart-kernel [-s NAME]` | Restart the active session's Jupyter kernel |
-| `colab stop [-s NAME]` | Terminate a session VM and tear down its keep-alive daemon |
+| `colab stop [-s NAME]` | Terminate a session VM and release its resources |
 | `colab url [-s NAME] [--open]` | Print or open a browser URL connecting to the active session |
 
 ### Execution
 | Command | Description |
 | --- | --- |
-| `colab run [--gpu GPU] [--tpu TPU] [--keep] SCRIPT [ARGS...]` | Run a local script on a fresh VM, forwarding arguments, then release it |
+| `colab run [--gpu GPU] [--tpu TPU] [--high-mem] [--keep] SCRIPT [ARGS...]` | Run a local script on a fresh VM, forwarding arguments, then release it |
 | `colab exec [-s NAME] [-f FILE] [--output-image PATH]` | Execute Python code from stdin, a local `.py` file, or a `.ipynb` notebook |
 | `colab repl [-s NAME] [--output-image PATH]` | Start an interactive Python REPL on the VM (exits cleanly on piped EOF) |
 | `colab console [-s NAME]` | Connect to a raw interactive TTY shell (tmux) on the remote VM |
+| `colab ssh [-s NAME] [--proxy-mode] [-i KEY] [--gpu GPU] [--tpu TPU] [--high-mem]` | Open an SSH shell to the runtime over WebSocket, or act as an OpenSSH `ProxyCommand` bridge for IDE remote-dev |
 
 ### File Operations
 | Command | Description |
@@ -99,6 +100,7 @@ Run `colab <command> --help` to view specific options, defaults, and detailed he
 | `colab drivemount [-s NAME] [PATH]` | Mount Google Drive on the VM (default: `/content/drive`) |
 | `colab install [-s NAME] [-r FILE \| PKG...]` | Install packages on the VM using `uv` (falls back to `pip`) |
 | `colab log [-s NAME] [-n N] [-o FILE]` | View or export session history (`.ipynb`, `.md`, `.txt`, `.jsonl`) |
+| `colab usage` | Show account compute-unit usage rate and balance |
 | `colab pay` | Open the Colab subscription page to manage compute units |
 | `colab version` | Print the installed version of the CLI |
 | `colab update [--install]` | Check for a newer release (and optionally upgrade the CLI in place) |
@@ -141,6 +143,7 @@ colab stop -s analysis
 
 ## Usage Notes
 
+* **Machine shape:** Use `--high-mem` with `colab new`, `colab run`, or `colab ssh` (when auto-creating a runtime) to request a high-RAM machine shape. Requires Colab Pro or Pro+ entitlement for supported accelerators (CPU, T4, A100, etc.). L4 and TPU runtimes ignore this flag because they only offer one shape. Machine shape is shown in `colab sessions` and `colab status`.
 * **TTY Requirements:** The interactive commands `repl` and `console` require a local TTY. When running inside automated scripts or pipelines, make sure to pipe stdin (e.g., `echo "print(1)" | colab repl`) to trigger non-interactive execution modes.
 * **Transparent Code Execution:** When calling `colab exec -f file.py`, the CLI reads the file locally and transmits its content to the remote kernel. You do not need to manually upload files before execution.
 * **Storage & State Paths:** Session tokens and metadata are stored at `~/.config/colab-cli/sessions.json`. Global CLI settings are located at `~/.config/colab-cli/settings.json`. These can be customized or isolated via the global `--config` flag.
@@ -174,7 +177,7 @@ Make the script executable (`chmod +x script.py`) and run it: `./script.py`. The
 
 For comprehensive architectural overviews and deep-dives into specific CLI sub-systems, refer to the detailed documentation:
 
-* [Session Management & Keep-Alive Architecture](docs/01_session_management.md)
+* [Session Management Architecture](docs/01_session_management.md)
 * [Interactive & Non-Interactive Execution Design](docs/02_execution_and_interactive.md)
 * [File Management & Jupyter Contents API](docs/03_file_management.md)
 * [Authentication Providers & VM Automation](docs/04_automation_and_utility.md)

{google_colab_cli-0.6.0 → google_colab_cli-0.7.3}/cloudbuild.yaml

@@ -38,6 +38,28 @@ steps:
         echo "TAG_NAME=${TAG_NAME}"
         echo "git describe: $(git describe --tags --always)"
 
+  # 1b. Delete any artifacts a previous run left staged.
+  #     Staged artifacts are only cleaned up after a successful release; a
+  #     failed one leaves them in place so the release can be retried. Since
+  #     step 4 publishes with `publish_all: true`, a later release would pick
+  #     those up too and republish them alongside the new ones.
+  #     This has to run as part of the build, which holds the delete
+  #     permission that individual accounts do not.
+  #     `|| true` so a first-ever release (nothing to delete) isn't a failure.
+  - id: purge-stale-artifacts
+    name: gcr.io/google.com/cloudsdktool/cloud-sdk:slim
+    entrypoint: bash
+    args:
+      - -c
+      - |
+        set -e
+        gcloud artifacts versions delete ${TAG_NAME}  \
+          --package={_EG_PROJECT_NAME} \
+          --project={_AR_PROJECT} \
+          --repository=${_AR_REPOSITORY} \
+          --location=us \
+          --quiet || true
+
   # 2. Build sdist + wheel into dist/ using uv.
   #    Use the non-slim bookworm variant: hatch-vcs derives the version
   #    by shelling out to `git describe`, which requires a git binary
@@ -57,6 +79,8 @@ steps:
   #    google-artifactregistry-auth keyring plugin (uses ADC from the
   #    Cloud Build service account, which is registered as a builder in
   #    the project's project.txtpb).
+  #    Delete any existing artifact for this tag which can happen if the
+  #    build failed to publish in step 4.
   - id: publish-to-ar
     name: python:3.13-slim
     entrypoint: bash

{google_colab_cli-0.6.0 → google_colab_cli-0.7.3}/docs/01_session_management.md

@@ -1,5 +1,9 @@
 ---
 log:
+2026-09-25: Runtime proxy tokens are now refreshed before expiry. `SessionState` persists `token_expires_at`, and `State.get_session` re-fetches the token via `GET /tun/m/assignments` when it's within 5 minutes of expiring.
+2026-09-25: Removed the keep-alive background daemon and related client/CLI ping functions. VM liveness is automatically maintained by the Colab backend based on kernel activity.
+2026-08-10: Added `colab usage` for account-level compute-unit rate/balance via `GET /tun/m/ccu-info` on the session backend (same bearer token as `colab new`).
+2026-08-09: Added `--high-mem` to `colab new`, `colab run`, and `colab ssh` (auto-create). Assign requests now send `shape=hm` when high-RAM is requested; `colab sessions` and `colab status` display machine shape.
 2026-06-15: Switched the keep-alive daemon from the `colab.pa.googleapis.com` `RuntimeService/KeepAliveAssignment` RPC to a Tunnel Frontend HTTP ping (`GET /tun/m/<endpoint>/keep-alive/` with `X-Colab-Tunnel: Google`) on `colab.research.google.com`. The RPC required `serviceusage` consumer access to Colab's internal project `1014160490159`, which ordinary user accounts lack, so every external user hit HTTP 403 `USER_PROJECT_DENIED` and their CLI sessions were idle-pruned within minutes (issue #14). Reproduced live with a third-party account; verified the tunnel ping is accepted by the same bearer-token credential that already works for `assign`. A `ReadTimeout` on the ping is treated as success (TFE records activity before forwarding to the often-non-responding VM). Generalized the pre-flight remediation messaging away from the now-irrelevant `colaboratory`/`pa.googleapis.com` framing, and removed the dead grpc-web client-registry/API-key code.
 2026-06-10: Replaced the POSIX-only `fcntl.flock` file locking in `_LockedFileStore` with the cross-platform `filelock` library (reported broken on Windows). Reads use `ReadWriteLock.read_lock()` (shared) and writes use `write_lock()` (exclusive), preserving the original `LOCK_SH`/`LOCK_EX` semantics. The lock is constructed with `is_singleton=False` so two `StateStore` instances for the same path in one process don't collapse into a single reentrant lock (which would raise `RuntimeError` on multi-threaded write contention). Added shared-read, cross-process exclusion, and multi-thread/multi-process regression tests.
 ---
@@ -32,11 +36,20 @@ Defines the specific hardware model.
     - `V5E1`: TPU v5e (1 core, optimized for inference/efficient training).
     - `V6E1`: TPU v6e (1 core, high performance).
 
-### 3. CLI Mapping
+### 3. Machine shape (`shape`)
+Defines the RAM profile for runtimes that support a choice (CPU, T4, A100, etc.).
+- `STANDARD` (default): omit the `shape` query param on assign.
+- `HIGH_RAM`: send `shape=hm` on assign (requires Colab Pro/Pro+ entitlement).
+
+Accelerators with only one shape (L4, v5e1, v6e1) ignore `--high-mem`.
+
+### 4. CLI Mapping
 The CLI maps user flags to these backend parameters:
 - `colab new my-session` -> `variant=DEFAULT`, `accelerator=NONE`
-- `colab new my-session -gpu=L4` -> `variant=GPU`, `accelerator=L4`
-- `colab new my-session -tpu=v5e1` -> `variant=TPU`, `accelerator=V5E1`
+- `colab new my-session --gpu=L4` -> `variant=GPU`, `accelerator=L4`
+- `colab new my-session --tpu=v5e1` -> `variant=TPU`, `accelerator=V5E1`
+- `colab new my-session --high-mem` -> adds `shape=hm` (when supported)
+- `colab new my-session --gpu A100 --high-mem` -> `variant=GPU`, `accelerator=A100`, `shape=hm`
 
 ## Approach
 
@@ -51,8 +64,8 @@ The CLI maps user flags to these backend parameters:
     - Format: `{ "session_name": { "token": "...", "backend_url": "...", "hardware": "..." } }`
 
 ### 2. Session Status (`colab status`)
-- **API**: `/api/sessions` or querying the kernel for resource usage via a special "status" message.
-- **Metric Collection**: Execute a small snippet on the VM to get memory/CPU usage if the backend API doesn't provide it directly.
+- **Session lines**: Local metadata per tracked session — hardware, shape, variant, IDLE/BUSY (from CLI bookkeeping), and optional last-execution detail. Syncs with `GET /tun/m/assignments` to prune stale entries.
+- **Future**: Real-time VM resource usage (CPU/RAM/GPU) via a kernel diagnostic snippet is still TODO. For account-level compute-unit balance and usage rate, use `colab usage`.
 
 ### 3. Stop Session (`colab stop`)
 - **API**: `POST https://colab.sandbox.google.com/tun/m/unassign/<endpoint>` (based on `tpu-v5e1-unassign.har`).
@@ -68,23 +81,15 @@ The CLI maps user flags to these backend parameters:
 - **API**: `GET https://colab.research.google.com/tun/m/assignments` (based on `colab-agent` implementation).
 - **Function**: Lists all active VM assignments for the user. This is useful for synchronizing local state with actual backend sessions.
 
-### 5. Keep-Alive Protocol
-To prevent Colab VMs from being deleted due to idle timeouts (standard is ~90 minutes), the CLI implements a background keep-alive mechanism.
-- **Daemon Process**: Since the CLI is a fire-and-forget tool, `colab new` spawns a detached background process running a hidden `keep-alive` command.
-- **Tunnel ping**: Every 60 seconds, the daemon issues `GET https://colab.research.google.com/tun/m/<endpoint>/keep-alive/` with the header `X-Colab-Tunnel: Google`, authenticated with the user's own Gaia bearer token (the same credential and host used for `/tun/m/assign`). The Tunnel Frontend (TFE) records `LastActiveTime` before forwarding the request, which refreshes the idle timer. This matches the official `colab-vscode` extension's `sendKeepAlive`. TFE notes the activity on arrival and then forwards to the VM, which often does not answer on this path — so the request commonly read-times-out even though the keep-alive succeeded; a `ReadTimeout` is therefore treated as success, while genuine HTTP errors (e.g. 404 for a deleted assignment) propagate.
-  - **Why not the RuntimeService RPC**: The previous implementation called `google.internal.colab.v1.RuntimeService/KeepAliveAssignment` at `colab.pa.googleapis.com` with `X-Goog-User-Project: 1014160490159`. That path requires the caller to be a `serviceusage` consumer of Colab's internal project `1014160490159`, which no ordinary user account is — so it returned HTTP 403 `USER_PROJECT_DENIED` for every external user, causing CLI sessions to be idle-pruned within minutes (issue #14). Dropping the header instead produced HTTP 400 `CONSUMER_INVALID` (public API-key project ≠ bearer-token quota project). The browser only succeeds because it rides the user's `google.com` cookie through an internal cookie-proxy (`colab.clients6.google.com`), which a headless bearer-token client cannot use. The TFE tunnel ping needs no project entitlement and works for any account that can assign a VM.
-- **Pre-flight (`colab new`, OAuth2/ADC only)**: Immediately after a successful `assign`, the CLI invokes `keep_alive_assignment` once synchronously. If the response is 403 with a `SCOPE_NOT_PERMITTED` body, it unassigns the new VM (to avoid leaking a billable assignment) and prints a per-provider remediation message before exiting non-zero. Other errors are tolerated — the daemon will retry and surface them via the structured event log. (Because keep-alive now uses the same backend/credential as `assign`, a scope failure at this stage is rare — assignment would normally have failed first.)
-- **Structured logging**: The daemon emits `keep_alive_started` (with `pid`, `endpoint`), one `keep_alive_error` per failed iteration (with `status_code`, `error_type`, truncated `error`, `response_body`, `iteration`, `consecutive_4xx`), and `keep_alive_stopped` (with `reason`, `iterations`, `duration_seconds`, optional `last_error`, optional `expected_endpoint`/`actual_endpoint`). All three are rendered specially by `colab log` so users get diagnostic context without parsing JSONL by hand.
-- **Termination**:
-    - **Explicit**: `colab stop` terminates the daemon using its stored PID.
-    - **Implicit**: If a session is pruned (e.g., during `sync_sessions`), its daemon is also terminated.
-    - **Safety Fallback**: The daemon automatically terminates after 24 hours to prevent permanent zombie processes.
-    - **State Check**: The daemon periodically verifies that its session still exists in the local state store; if missing, it exits.
-    - **Repeated 4xx**: After two consecutive 4xx responses, the daemon exits with `reason=consecutive_4xx_errors`. With the TFE tunnel ping, a normal read-timeout is not counted as a 4xx (it is treated as success), so this branch is now reached only by genuine HTTP errors such as a 404 for a deleted/expired assignment.
+### 5. Session Liveness
+VM liveness is automatically maintained by the Colab backend based on active kernel executions and interactions. The CLI runs no keep-alive daemon. A session persists until explicitly terminated via `colab stop` or until the backend reclaims it due to inactivity.
+
+### 6. Runtime Proxy Token Refresh
+The runtime proxy token (`runtimeProxyInfo.token`) authenticates every request to the VM and expires after `tokenExpiresInSeconds` (1 hour). `SessionState.token_expires_at` stores the absolute expiry, computed when the token is received. Commands that talk to the VM load their session through `State.get_session(name)`, which, when the token is missing an expiry or is within 5 minutes of it, calls `GET /tun/m/assignments`, takes the fresh token/URL for the matching endpoint, and persists it. If the endpoint is no longer assigned, the session is pruned. `sync_sessions` also writes back the fresh tokens it already fetches. Long-lived connections (`repl`, `console`, `ssh`) only present the token on initial connect and aren't refreshed mid-session.
 
 ## TODO / Future Work
 - **Backend Sync**: Implement a way to reconcile the local `sessions.json` with the output of `colab sessions`.
-- **Resource Usage**: Add real-time resource usage (CPU/RAM/GPU) to the `status` output by executing a diagnostic snippet on the VM.
+- **VM Resource Usage**: Add real-time resource usage (CPU/RAM/GPU) to the `status` output by executing a diagnostic snippet on the VM (distinct from account-level CU info, which is available via `colab usage`).
 
 ## Implementation Details
 - **Authentication**: Uses `google-auth-oauthlib` to perform a local server OAuth flow.
@@ -96,7 +101,7 @@ To prevent Colab VMs from being deleted due to idle timeouts (standard is ~90 mi
 - Handle authentication headers (likely `Authorization: Bearer <token>` or cookies).
 
 ### State Persistence & File Locking
-- **Stores**: `StateStore` (`sessions.json`) and `SettingsStore` (`settings.json`) both derive from `_LockedFileStore`, which guards concurrent access across independent `colab` invocations and the detached keep-alive daemon.
+- **Stores**: `StateStore` (`sessions.json`) and `SettingsStore` (`settings.json`) both derive from `_LockedFileStore`, which guards concurrent access across independent `colab` invocations.
 - **Cross-platform locking**: Locking uses the [`filelock`](https://pypi.org/project/filelock/) library rather than `fcntl.flock`. `fcntl` is POSIX-only and is unavailable on Windows, so the original implementation crashed on import there. `filelock` provides the same advisory cross-process locking on Linux, macOS, and Windows.
 - **Shared vs. exclusive**: Each store owns a `filelock.ReadWriteLock` bound to a sidecar file (`<path>.lock`). Reads acquire `read_lock()` (shared — multiple concurrent readers allowed) and writes acquire `write_lock()` (exclusive). This preserves the `LOCK_SH`/`LOCK_EX` distinction of the previous `fcntl` implementation.
 - **`is_singleton=False`**: The `ReadWriteLock` is created with `is_singleton=False`. With `filelock`'s default (`True`), two `ReadWriteLock` objects for the same path *within a single process* are deduplicated into one reentrant lock; its reentrancy guard then raises `RuntimeError` when two threads each construct their own `StateStore` and contend for the write lock. Disabling the singleton registry makes each store's lock independent so they serialize via the underlying file lock instead.

{google_colab_cli-0.6.0 → google_colab_cli-0.7.3}/docs/04_automation_and_utility.md

@@ -1,5 +1,7 @@
 ---
 log:
+2026-09-25: Cleaned up references to keep-alive background daemon and pre-flight pings. VM liveness is automatically maintained by the Colab backend based on active kernel activity.
+2026-08-10: Added `colab usage` for account-level compute-unit rate/balance via `GET /tun/m/ccu-info` on the session backend (same bearer token as `colab new`).
 2026-06-11: Replaced the `oauth2` provider's `run_local_server()` (localhost redirect) with a remote copy-paste flow (`_run_remote_flow` in `auth.py`). The CLI now prints an authorization URL built with `redirect_uri=https://sdk.cloud.google.com/applicationdefaultauthcode.html` and `token_usage=remote`, then reads the pasted authorization code via `input()` and exchanges it with `flow.fetch_token(code=...)`. This is the same flow `gcloud auth application-default login` uses and works identically in local and remote/headless/container environments, removing the heuristic of whether to auto-open a browser. Confirmed server-side acceptance with a live GET-only check against the bundled cloud-SDK client (`764086051850-...`); the OOB redirect and a non-bundled client id were both verified to be rejected (`OOB flow has been blocked` / `redirect_uri_mismatch`). Unit tests in `tests/test_auth.py` assert no localhost server is started, the redirect URI + `token_usage=remote` are set, and the pasted code is exchanged.
 2026-06-01: Enabled `colab update --install` self-update on macOS in addition to Linux. Refactored platform check logic to keep the implementation DRY and updated both tests and documentation. Also, on these platforms, an additional message is shown recommending `colab update --install` to upgrade in place, positioned above the standard `pip`/`uv` installation command.
 2026-05-29: Added default OAuth2 client config (`oauth_config.json`) as a bundled package resource and restored fallback loading logic in `get_credentials()`. The CLI now falls back to using these default credentials when no explicit local config is found. Added `integration/repro_bundled_oauth` integration test.
@@ -12,7 +14,7 @@ log:
 2026-05-12: Added an optional `timeout=` parameter to `ColabRuntime.execute_code` that flows through to both the `execute()` and `execute_interactive()` branches. `colab auth` and `colab drivemount` now pass `timeout=600` (10 min) via a shared `INTERACTIVE_AUTOMATION_TIMEOUT_SEC` constant in `commands/automation.py`. Background: `jupyter_kernel_client` defaults to a 10s wall-clock timeout that is consumed even when the kernel is idle waiting on `input_request`. With the drivefs hook intercepting that request and prompting the user to OAuth in their browser, any user that takes >10s to click through (essentially everyone) hit `TimeoutError` and saw "drivemount failed" even though the mount had actually succeeded server-side. The fix is scoped narrowly to the two human-in-the-loop subcommands; non-interactive paths (`colab exec`, `colab run`, `colab install`, `colab repl --pipe`, `colab console --pipe`) keep the upstream default since they receive continuous iopub traffic that resets the practical inactivity ceiling.
 ---
 
-# Design: Automation and Utility (`auth`, `install`, `log`, `pay`, `version`, `update`, `whoami`)
+# Design: Automation and Utility (`auth`, `install`, `log`, `pay`, `usage`, `version`, `update`, `whoami`)
 
 ## Overview
 
@@ -57,18 +59,9 @@ allowing the core `Client` to remain authentication-agnostic — it only sees a
 ### Required Scopes
 
 The CLI talks to the Colab session backend at `colab.research.google.com`
-for assignment, unassignment, the contents API, **and keep-alive** (the TFE
-tunnel ping — see `01_session_management.md`). The `userinfo.email` scope is
+for assignment, unassignment, CCU info, and the contents API. The `userinfo.email` scope is
 sufficient for this host.
 
-> Historical note: keep-alive previously used the `RuntimeService`
-> (`KeepAliveAssignment`) at `colab.pa.googleapis.com`, which required the
-> `https://www.googleapis.com/auth/colaboratory` scope **and** the caller to
-> be a `serviceusage` consumer of Colab's internal project `1014160490159`.
-> The latter is impossible for ordinary user accounts, which made keep-alive
-> fail with HTTP 403 `USER_PROJECT_DENIED` for all external users (issue #14).
-> Keep-alive no longer touches `colab.pa.googleapis.com`.
-
 How each provider supplies the scope:
 
 -   **`oauth2`**: `PUBLIC_SCOPES` already includes `colaboratory`, so the
@@ -91,16 +84,12 @@ How each provider supplies the scope:
     ```
 
     `userinfo.email` is required for the session backend at
-    `colab.research.google.com` (otherwise assign/unassign/sessions/keep-alive
+    `colab.research.google.com` (otherwise assign/unassign/sessions
     return HTTP 401); `colaboratory` is retained for forward compatibility and
     other Colab features; `openid` and `cloud-platform` are mandated by
     `gcloud` itself (`gcloud auth application-default login` rejects scope
     lists that omit `cloud-platform` with `Invalid value for [--scopes]`).
 
-`colab new` performs a one-shot keep-alive pre-flight after `assign`
-succeeds so missing-scope failures surface immediately (with per-provider
-remediation guidance) rather than silently after ~1 minute via the daemon.
-
 ## Approach
 
 ### 1. Authentication (`colab auth`)
@@ -151,13 +140,22 @@ remediation guidance) rather than silently after ~1 minute via the daemon.
 -   **Conversion (Planned)**: Future expansion to convert history logs to
     `.ipynb` or `.html`.
 
-### 5. Subscription Management (`colab pay`)
+### 5. Compute-Unit Usage (`colab usage`)
+
+-   **Action**: Print account-level compute-unit (CU) usage rate and balance.
+-   **Data source**: `GET https://colab.research.google.com/tun/m/ccu-info`
+    (balance, hourly rate, assignment count). Same bearer token and session
+    backend as `colab new`.
+-   **Output**: `Current balance: … compute units`, `Usage rate: {rate}/hr`
+    (aggregate CU/hour across assigned VMs), `Active assignments: N`.
+
+### 6. Subscription Management (`colab pay`)
 
 -   **Action**: Open the Colab signup page in the user's browser.
 -   **Implementation**: Uses
     `webbrowser.open("https://colab.research.google.com/signup")`.
 
-### 6. Version Information (`colab version`)
+### 7. Version Information (`colab version`)
 
 -   **Action**: Show the current version of the Colab CLI.
 -   **Implementation**:
@@ -167,7 +165,7 @@ remediation guidance) rather than silently after ~1 minute via the daemon.
         Git commit hash using `git rev-parse --short HEAD`.
     -   Dynamic versioning is supported in the build system via `hatch-vcs`.
 
-### 7. Auto-Update (`colab update`)
+### 8. Auto-Update (`colab update`)
 
 -   **Action**: Check if a new version of the Colab CLI is available.
 -   **Auto-check**: The CLI automatically checks for updates once every 24 hours
@@ -218,7 +216,7 @@ remediation guidance) rather than silently after ~1 minute via the daemon.
     automation. If the upgrade command exits non-zero, `colab update --install`
     propagates the same exit code.
 
-### 8. Identity Inspection (`colab whoami`) [developer-only]
+### 9. Identity Inspection (`colab whoami`) [developer-only]
 
 -   **Action**: Resolve the active credentials, mint an access token, and
     print the email, audience, scopes, and expiry of that token.
@@ -260,7 +258,7 @@ remediation guidance) rather than silently after ~1 minute via the daemon.
       - openid
     ```
 
-### 9. README and AGENT (`colab README`, `colab AGENT`)
+### 10. README and AGENT (`colab README`, `colab AGENT`)
 
 -   **Action**: Print the bundled `README.md` or `AGENTS.md` file.
 -   **Implementation**:

{google_colab_cli-0.6.0 → google_colab_cli-0.7.3}/docs/05_run_command.md

@@ -1,5 +1,7 @@
 ---
 log:
+2026-09-25: Removed keep-alive daemon spawning from session allocation and teardown. `--keep` remains supported and skips unassignment on completion.
+2026-08-09: Added `--high-mem` flag (passthrough to session creation; sends `shape=hm` on assign when supported).
 2026-05-12: Initial design and implementation of `colab run <script.py> [args...]`. Combines `colab new` + `colab exec` + `colab stop` into a single fire-and-forget invocation so a Python file can use `#!/usr/bin/env -S colab run` as a shebang line and execute on a freshly-allocated Colab VM. Adds `--keep` (skip auto-stop), `--gpu` / `--tpu` (passthrough to session creation), `-s/--session` (name the ephemeral session), and propagates the script's exit status (non-zero on any uncaught exception in the kernel). The script's `sys.argv` is re-set inside the kernel to mirror native `python script.py arg1 arg2` semantics, and `__name__` is set to `"__main__"`.
 2026-05-12: Native CPython exit-code semantics for `sys.exit()` / `raise SystemExit(...)` from the script body. The Colab kernel reports a `SystemExit` as `output_type=='error'`, which under the previous logic would have (a) printed the IPython traceback (`An exception has occurred, use %tb...`) and (b) flagged the run as a failure regardless of the integer exit code. Now: `sys.exit()` / `sys.exit(0)` exit 0 silently; `sys.exit(N)` exits N; `sys.exit('msg')` exits 1 (matching CPython). The IPython "To exit: use 'exit', 'quit', or Ctrl-D." UserWarning is filtered via the prelude. Encoded after running `examples/gpu_hello.py` end-to-end and seeing the noisy `SystemExit: 0` traceback at the end of an otherwise-successful GPU run.
 2026-06-04: Bumped the default value of the `--timeout` flag from 10.0s to 30.0s so short-but-silent tasks aren't prematurely killed out of the box. Mirrors the same change for `colab exec`.
@@ -29,6 +31,7 @@ colab run [OPTIONS] SCRIPT [SCRIPT_ARGS]...
 | `-s`, `--session` | str | auto | Name the ephemeral session (helpful with `--keep`). Auto-generated as `run-<6 hex>` if omitted. |
 | `--gpu` | str | None | Same set as `colab new --gpu` (T4, L4, G4, H100, A100). |
 | `--tpu` | str | None | Same set as `colab new --tpu` (v5e1, v6e1). |
+| `--high-mem` | bool | False | Same as `colab new --high-mem` — request high-RAM when supported. |
 | `--keep` | bool | False | Do **not** stop the session after the script finishes. |
 | `--timeout` | float | 30.0 | Timeout in seconds for code execution to prevent hanging on silent tasks. |
 
@@ -47,7 +50,7 @@ print(torch.cuda.get_device_name(0))
 
 ## Behavior
 
-1. **Allocate**: Creates a fresh session (mirrors `colab new` end-to-end: `assign` → keep-alive pre-flight → spawn keep-alive daemon → persist `SessionState`). Session name defaults to `run-<6 hex>`.
+1. **Allocate**: Creates a fresh session (mirrors `colab new`: `assign` → persist `SessionState`). Session name defaults to `run-<6 hex>`.
 2. **Execute**: Reads the script file. Prepends a deterministic prelude that re-sets `sys.argv` and `__name__` so the script body sees the same execution context as `python script.py arg1 arg2`:
    ```python
    import sys
@@ -60,16 +63,13 @@ print(torch.cuda.get_device_name(0))
    - Sends `runtime.stop(shutdown_kernel=True)` (best-effort).
    - Calls `state.client.unassign(endpoint)` to free the billable VM.
    - Removes the session from `StateStore`.
-   - Kills the keep-alive daemon (`kill_process(s.keep_alive_pid)`).
    - Logs `session_terminated` with `reason="run_completed"` (or `"run_failed"`).
 
-If `--keep` is set, the session remains visible in `colab sessions` and `colab status` and can be reused with `colab exec -s <name>`, `colab repl -s <name>`, etc., until the user runs `colab stop` (or the keep-alive daemon hits its 24h cap).
+If `--keep` is set, the session remains visible in `colab sessions` and `colab status` and can be reused with `colab exec -s <name>`, `colab repl -s <name>`, etc., until the user runs `colab stop` (or the backend reclaims it due to inactivity).
 
 ## AGENTS.md Constraints Honoured
-- **Item 7 (no background threads)**: The keep-alive daemon is the existing detached process from `colab new`; this command introduces no new threads.
+- **Item 7 (fire-and-forget)**: No background threads or daemons are created; commands execute and exit cleanly.
 - **Item 10 (live probes allocate real resources)**: The teardown is in a `try/finally` so an exception during execution still releases the VM. Tests assert `unassign` is called even when the script errors.
-- **Item 16 (daemon flag propagation)**: Reuses `spawn_keep_alive(...)` which already propagates `--auth` and `--config`.
-- **Item 17 (persist-before-spawn)**: Uses the same persist-before-spawn pattern as `colab new`.
 
 ## Testing Strategy (TDD)