podbench 0.1.0b1__tar.gz

podbench-0.1.0b1/.claude/skills/ephemeral-containers/SKILL.md

@@ -0,0 +1,120 @@
+---
+name: ephemeral-containers
+description: What the Kubernetes API will and will not let an ephemeral container do — the constraints that shaped podbench's ladder, dev pods and seat identity. Read before changing spec.py, launcher.py, or anything that authors a container spec.
+---
+
+# Ephemeral containers
+
+`podbench attach` adds an **ephemeral** container; `podbench dev` authors an
+**ordinary** pod. Most of podbench's odder shapes exist because those two are not
+interchangeable, and the API server enforces the difference. Every rule below was hit
+for real against a cluster, not read in a doc.
+
+## They are permanent
+
+An ephemeral container cannot be removed, stopped or restarted. Consequences that drive
+real code:
+
+- **A name is burnt once used.** A container that exits reaches `Completed` and its name
+  is unusable for the pod's lifetime. Hence `kubectl.next_container_name` and the
+  `podbench-1`, `-2`, `-3` suffixes — and hence a failing rung must take a *fresh* name
+  rather than retry its own.
+- **Never give one a short-lived command.** `spec.AGENT_COMMAND` is the default for this
+  reason. A container handed `true` — or handed a verb the CLI does not dispatch — burns
+  its name immediately.
+- **The agent must not die.** An unhandled exception in start-up costs a name and, if the
+  ladder then walks every rung, three of them. `agent.ensure_all` records failures and
+  keeps idling instead of raising. Keep it that way.
+- **An OOM is unrecoverable.** A replacement comes up with a fresh rootfs, losing the
+  vscode-server, extensions and host keys. Nothing may live *only* in the writable layer
+  that cannot be rebuilt.
+
+## They may not use `subPath`
+
+```
+spec.ephemeralContainers[0].volumeMounts[0].subPath:
+  Forbidden: cannot be set for an Ephemeral Container
+```
+
+The API server refuses the whole request. There is no workaround: mounting the volume
+somewhere else does not help, because NSS reads `/etc/passwd` and nothing else.
+
+This is why the seat's identity has two mechanisms, split by container kind:
+
+| | mechanism |
+|---|---|
+| `attach` (ephemeral) | the agent registers its own passwd record, against the image's group-writable `/etc/passwd`, under `--seat-gid-root` |
+| `podbench dev` (ordinary pod) | the projected identity from the chart's `seatIdentity`, mounted with `subPath` — better, since nothing is written and no group 0 is needed |
+
+`spec.validate_ephemeral_volume_mounts` enforces it at the authoring layer so it cannot
+recur.
+
+## They may only mount volumes the pod already declares
+
+Pod volumes are immutable after creation, and an ephemeral container may not introduce
+one. So anything needing a volume — Patch mode's venv claim, the seat's home directory —
+only works where the deployment cooperated at creation time. That is the whole reason
+Patch mode asks for a helm change, and `patch --print-values` emits the ask.
+
+`launcher.resolve_mounts` refuses an undeclared volume with an explanation rather than
+letting the API server produce a confusing one.
+
+## They may not carry `resources`
+
+The field is rejected outright, and the container is confined by the pod's cgroup
+regardless. So on a live pod podbench **shares the workload's memory and
+ephemeral-storage limits and cannot reserve its own** — the biggest footgun in the
+product, and the reason Iterate mode authors a real pod (where the sidecar *can* have
+resources) instead of attaching.
+
+The mitigation is `kubectl patch pod --subresource resize`, offered behind `--resize`
+and only lightly proven (one cluster, one pod, never against a `LimitRange` or a
+controller that would fight it).
+
+## `capabilities.add` on a non-root uid is a silent no-op
+
+The kernel grants capabilities to a non-zero uid only through the ambient set, which no
+CRI populates. `capabilities.add: [SYS_PTRACE]` beside `runAsUser: 1000` is **admitted**
+and produces `CapEff: 0000000000000000`: a container that looks privileged and behaves
+unprivileged.
+
+The ladder therefore has exactly two capability rungs and no middle ground, and
+`spec.validate_security_context` raises rather than emitting the combination. Do not
+"fix" a failing degraded rung by adding the capability back.
+
+## Refusal arrives through two unrelated channels
+
+- **Synchronous** — Pod Security Admission, in `kubectl`'s stderr, matched on the stable
+  substring `must not include "SYS_PTRACE" in securityContext.capabilities.add`.
+- **Asynchronous** — the kubelet, *seconds later*, in the container's status:
+  `CreateContainerConfigError` / `container's runAsUser breaks non-root policy`. The API
+  call already exited 0.
+
+Only the first can be caught by wrapping the call, and only the second burns a name — so
+each needs its own arm of the walk. Pre-empt the second by reading the target pod's
+`securityContext.runAsNonRoot` before trying a root container.
+
+## Attribution: which processes belong to the target
+
+Under a shared PID namespace the debug container sees everything, including its own
+processes and any other podbench session's. The only attribution that stays correct is a
+substring match of the target's container id against `/proc/<pid>/cgroup`, injected as
+`PODBENCH_TARGET_CID` (`model.TARGET_CID_ENV`).
+
+The cgroup-difference fallback is a guess. `proc.scan_processes` returns
+`Attribution.CGROUP_FALLBACK` and a warning so callers can say so rather than present it
+as fact. PID 1 is *not* reliably the target — under `--target` it is often `/pause`.
+
+## Dev pods are clones, and clones carry junk
+
+`spec.dev_pod_spec` strips what the API server owns or what would make a controller adopt
+the clone. Two that were learned the hard way:
+
+- **`spec.ephemeralContainers`** — `Forbidden: cannot be set on create`. Cloning any pod
+  that had ever been attached to failed until this was popped.
+- **Controller labels** (`pod-template-hash` and friends) — keeping them gives a
+  `replicas: 1` ReplicaSet two matching pods and one gets reaped.
+
+`kubectl debug --copy-to` is not an alternative: it strips **all** labels, so the clone
+never joins the Service, which is the demo. That is why the launcher authors the spec
+itself — and why it can also add resources, volumes and a real sidecar.

podbench-0.1.0b1/.claude/skills/ssh-over-exec/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: ssh-over-exec
+description: The invariants of podbench's ssh-through-kubectl-exec transport, all of which fail silently or misleadingly when broken. Read before touching sshcfg.py, agent.py, or anything that generates a ProxyCommand or an sshd config.
+---
+
+# ssh over `kubectl exec`
+
+The transport is a complete OpenSSH connection whose only carrier is `kubectl exec`:
+`sshd -i` runs inside the debug container as the client's `ProxyCommand`, so there is no
+listening socket, no port-forward and no pod IP. Auth to the cluster is the kubeconfig;
+auth to the container is a key.
+
+**Every rule here has a silent or misleading failure mode.** That is why the client
+config, the server config and the ProxyCommand are generated from one place and never
+hand-written, and why `tests/test_sshcfg.py` is full of exact-string assertions. The
+regression to fear is a well-meaning tidy-up, not a crash.
+
+## `-e` is mandatory, and not for the reason it looks
+
+`sshd -i -e`. The flag is **not** about log tidiness — it is about keeping **fd 2 open**.
+
+Closing or redirecting stderr in a `kubectl exec`'d process tears down the whole CRI exec
+stream and silently truncates stdio with `rc=0`:
+
+```
+( echo one; sleep 4; echo two ) | kubectl exec -i pod -c c -- sh -c 'exec cat'
+one
+two                     # both arrive
+
+( echo one; sleep 4; echo two ) | kubectl exec -i pod -c c -- sh -c 'exec 2>/dev/null; exec cat'
+one                     # "two" SILENTLY LOST, rc=0
+```
+
+Against sshd it surfaces as a network error — `ssh_dispatch_run_fatal: … Broken pipe`,
+dying at `expecting SSH2_MSG_KEX_ECDH_REPLY` — which sends you debugging the wrong layer.
+
+| ProxyCommand | result |
+|---|---|
+| `sshd -i -e` | works |
+| `sshd -i -E /tmp/log` | works (`-E` implies `log_stderr=1`) |
+| `sshd -i` | **fails, reproducibly** |
+| `sh -c '/usr/sbin/sshd -i'` (no `exec`, shell holds fd 2) | works — **masks the bug** |
+| `sh -c 'exec /usr/sbin/sshd -i'` | fails |
+| `sh -c 'exec /usr/sbin/sshd -i -e 2>&1'` | fails |
+
+Two spikes disagreed about this (S2 ran without `-e` for a whole session), so the shipped
+form is `-e -o LogLevel=ERROR`: zero stderr bytes, fd 2 still open. `tests/e2e/test_s1_transport.py::test_transport_dies_without_dash_e`
+edits the config podbench *actually generates* and asserts the transport breaks — if it
+ever fails, settle the contradiction, do not delete the flag.
+
+## Never `-t`
+
+`kubectl exec -t` silently degrades to non-tty from a script, so it looks fine — and
+**hangs the ssh client forever** when a real TTY is present. There is no case where the
+transport wants a TTY.
+
+## Never redirect or merge stderr
+
+Any `2>&1`, `>/dev/null`, or wrapper that closes fd 2 breaks the transport per the above.
+`proxy_command` returns argv (no shell) so there is nowhere to introduce one.
+
+## `ControlPath` must be short
+
+`sizeof(sockaddr_un.sun_path)` is 108 bytes. Put the socket next to a kubeconfig or in a
+workspace and ssh refuses with `ControlPath too long`. Hence `/tmp/podbench-cm/%C` and
+`control_path_ok`.
+
+**Create the parent directory.** ssh will not, and the failure reads like a transport
+fault: `unix_listener: cannot bind to path …: No such file or directory`.
+`write_ssh_config` calls `ensure_control_dir` because writing the config is when the path
+is known.
+
+## `ServerAliveInterval` is mandatory
+
+A *stalled* transport — what an apiserver or konnectivity hiccup looks like — hangs ssh
+**forever** with no keepalive. With `ServerAliveInterval 15` / `ServerAliveCountMax 3` it
+fails in seconds. A hard kill or pod deletion is detected instantly either way.
+
+## sshd does not leak its own environment
+
+The launcher injects `PODBENCH_TARGET_CID` into the container spec and it reaches the
+container — but **not** an ssh session, and a non-interactive `ssh host 'pids'` sources no
+profile either. Without forwarding, the in-pod helpers fall back to *guessing* which
+processes belong to the target.
+
+`SetEnv` in the sshd config is the only route that survives. And:
+
+> **One `SetEnv` directive carrying every pair, never one per variable.** sshd resolves
+> each keyword **first-match-wins**, so a second `SetEnv` line is silently ignored. This
+> shipped once: only `PODBENCH_NODE_NAME` arrived and `PODBENCH_TARGET_CID` did not.
+
+Values containing whitespace are dropped — a space ends the pair early.
+
+## Nothing is on `PATH` in an ssh session
+
+`ssh host '<cmd>'` runs a non-interactive shell that sources nothing and does not inherit
+the image's `ENV PATH`. Every helper in `image/bin/` names `/app/.venv/bin/podbench` by
+absolute path for exactly this reason. If you add a helper, do the same.
+
+## sshd needs an NSS identity
+
+sshd resolves the login name through NSS **before** it looks at a key, so a uid with no
+`/etc/passwd` entry fails as `Permission denied (publickey)` — pointing at the key, which
+is not the problem. `ssh-keygen` cannot even mint a host key (`No user exists for uid
+1000`), and `-C` does not help: it calls `getpwuid()` regardless.
+
+See the `ephemeral-containers` skill for the two mechanisms that supply an identity and
+why they differ by container kind.
+
+## Host keys
+
+Fresh keys per attach would mean either `StrictHostKeyChecking no` — teaching users to
+disable host verification — or a warning on every new pod. podbench instead manages
+`known_hosts` programmatically, keyed on `HostKeyAlias podbench-<pod uid>`, and supports a
+stable key from a Secret. Do not "simplify" this to `no`.
+
+## Measured budgets
+
+Cold connect ~0.345 s; ~0.06 s over `ControlMaster` (≈6×). ~26 MB/s pod→client, ~13 MB/s
+client→pod. ~10 MB RSS per live session. 600 s idle survives with no keepalives; a
+stalled transport needs them.

podbench-0.1.0b1/.copier-answers.yml

@@ -0,0 +1,16 @@
+# Changes here will be overwritten by Copier
+_commit: 5.4.0
+_src_path: git@github.com:diamondlightsource/python-copier-template
+author_email: gilesknap@gmail.com
+author_name: Giles Knap
+description: debugging container for kubernetes pods
+distribution_name: podbench
+docker: true
+docs_type: sphinx
+git_platform: github.com
+github_org: gilesknap
+package_name: podbench
+pypi: true
+repo_name: podbench
+strict_typing: true
+type_checker: pyright

podbench-0.1.0b1/.devcontainer/devcontainer.json

@@ -0,0 +1,72 @@
+// For format details, see https://containers.dev/implementors/json_reference/
+{
+    "name": "Python 3 Developer Container",
+    "build": {
+        "dockerfile": "../Dockerfile",
+        "target": "developer"
+    },
+    "remoteEnv": {
+        // Allow X11 apps to run inside the container
+        "DISPLAY": "${localEnv:DISPLAY}",
+        // Put things that allow it in the persistent cache
+        "PRE_COMMIT_HOME": "/cache/pre-commit",
+        "UV_CACHE_DIR": "/cache/uv",
+        "UV_PYTHON_CACHE_DIR": "/cache/uv-python",
+        // Make a venv that is specific for this workspace path as the cache is shared
+        "UV_PROJECT_ENVIRONMENT": "/cache/venv-for${localWorkspaceFolder}",
+        // Do the equivalent of "activate" the venv so we don't have to "uv run" everything
+        "VIRTUAL_ENV": "/cache/venv-for${localWorkspaceFolder}",
+        "PATH": "/cache/venv-for${localWorkspaceFolder}/bin:${containerEnv:PATH}"
+    },
+    "customizations": {
+        "vscode": {
+            // Set *default* container specific settings.json values on container create.
+            "settings": {
+                // Use the container's python by default
+                "python.defaultInterpreterPath": "/cache/venv-for${localWorkspaceFolder}/bin/python",
+                // Don't activate the venv as it is already in the PATH
+                "python.terminal.activateEnvInCurrentTerminal": false,
+                "python.terminal.activateEnvironment": false,
+                // Workaround to prevent garbled python REPL in the terminal
+                // https://github.com/microsoft/vscode-python/issues/25505
+                "python.terminal.shellIntegration.enabled": false
+            },
+            // Add the IDs of extensions you want installed when the container is created.
+            "extensions": [
+                "ms-python.python",
+                "github.vscode-github-actions",
+                "tamasfe.even-better-toml",
+                "redhat.vscode-yaml",
+                "ryanluker.vscode-coverage-gutters",
+                "charliermarsh.ruff",
+                "ms-azuretools.vscode-docker"
+            ]
+        }
+    },
+    // Create the config folder for the bash-config feature and uv cache
+    "initializeCommand": "mkdir -p ${localEnv:HOME}/.config/terminal-config",
+    "runArgs": [
+        // Allow the container to access the host X11 display and EPICS CA
+        "--net=host",
+        // Make sure SELinux does not disable with access to host filesystems like tmp
+        "--security-opt=label=disable"
+    ],
+    "mounts": [
+        // Mount in the user terminal config folder so it can be edited
+        {
+            "source": "${localEnv:HOME}/.config/terminal-config",
+            "target": "/user-terminal-config",
+            "type": "bind"
+        },
+        // Keep a persistent cross container cache for uv, pre-commit, and the venvs
+        {
+            "source": "devcontainer-shared-cache",
+            "target": "/cache",
+            "type": "volume"
+        }
+    ],
+    // Mount the parent as /workspaces so we can pip install peers as editable
+    "workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces,type=bind",
+    // After the container is created, recreate the venv then make pre-commit first run faster
+    "postCreateCommand": "uv venv --clear && uv sync && pre-commit install --install-hooks"
+}

podbench-0.1.0b1/.dockerignore

@@ -0,0 +1,25 @@
+# .git is deliberately NOT ignored: the build stage runs setuptools_scm, which
+# derives the version from git metadata, so `uv sync` cannot build the project
+# without it (and _version.py is itself gitignored).
+
+# Build and test outputs - regenerated inside the image, and a stale .venv from
+# the host would be the wrong architecture.
+.venv/
+build/
+dist/
+docs/_build/
+.tox/
+*.egg-info/
+**/__pycache__/
+*.py[cod]
+
+# Caches and coverage
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+cov.xml
+
+# Editor and container tooling that never reaches the runtime image
+.devcontainer/
+.vscode/

podbench-0.1.0b1/.github/CONTRIBUTING.md

@@ -0,0 +1,27 @@
+# Contribute to the project
+
+Contributions and issues are most welcome! All issues and pull requests are
+handled through [GitHub](https://github.com/gilesknap/podbench/issues). Also, please check for any existing issues before
+filing a new one. If you have a great idea but it involves big changes, please
+file a ticket before making a pull request! We want to make sure you don't spend
+your time coding something that might not fit the scope of the project.
+
+## Issue or Discussion?
+
+Github also offers [discussions](https://github.com/gilesknap/podbench/discussions) as a place to ask questions and share ideas. If
+your issue is open ended and it is not obvious when it can be "closed", please
+raise it as a discussion instead.
+
+## Code Coverage
+
+While 100% code coverage does not make a library bug-free, it significantly
+reduces the number of easily caught bugs! Please make sure coverage remains the
+same or is improved by a pull request!
+
+## Developer Information
+
+It is recommended that developers use a [vscode devcontainer](https://code.visualstudio.com/docs/devcontainers/containers). This repository contains configuration to set up a containerized development environment that suits its own needs.
+
+This project was created using the [Diamond Light Source Copier Template](https://github.com/DiamondLightSource/python-copier-template) for Python projects.
+
+For more information on common tasks like setting up a developer environment, running the tests, and setting a pre-commit hook, see the template's [How-to guides](https://diamondlightsource.github.io/python-copier-template/5.4.0/how-to.html).

podbench-0.1.0b1/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,21 @@
+---
+name: Bug Report
+about: The template to use for reporting bugs and usability issues
+title: " "
+labels: 'bug'
+assignees: ''
+
+---
+
+Describe the bug, including a clear and concise description of the expected behaviour, the actual behavior and the context in which you encountered it (ideally include details of your environment).
+
+## Steps To Reproduce
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+
+## Acceptance Criteria
+- Specific criteria that will be used to judge if the issue is fixed

podbench-0.1.0b1/.github/ISSUE_TEMPLATE/issue.md

@@ -0,0 +1,13 @@
+---
+name: Issue
+about: The standard template to use for feature requests, design discussions and tasks
+title: " "
+labels: ''
+assignees: ''
+
+---
+
+A brief description of the issue, including specific stakeholders and the business case where appropriate
+
+## Acceptance Criteria
+- Specific criteria that will be used to judge if the issue is fixed

podbench-0.1.0b1/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md

@@ -0,0 +1,8 @@
+Fixes #ISSUE
+
+### Instructions to reviewer on how to test:
+1. Do thing x
+2. Confirm thing y happens
+
+### Checks for reviewer
+- [ ] Would the PR title make sense to a user on a set of release notes

podbench-0.1.0b1/.github/pages/index.html

@@ -0,0 +1,11 @@
+<!DOCTYPE html>
+<html>
+
+<head>
+    <title>Redirecting to main branch</title>
+    <meta charset="utf-8">
+    <meta http-equiv="refresh" content="0; url=./main/index.html">
+    <link rel="canonical" href="main/index.html">
+</head>
+
+</html>

podbench-0.1.0b1/.github/pages/make_switcher.py

@@ -0,0 +1,96 @@
+"""Make switcher.json to allow docs to switch between different versions."""
+
+import json
+import logging
+from argparse import ArgumentParser
+from pathlib import Path
+from subprocess import CalledProcessError, check_output
+
+
+def report_output(stdout: bytes, label: str) -> list[str]:
+    """Print and return something received frm stdout."""
+    ret = stdout.decode().strip().split("\n")
+    print(f"{label}: {ret}")
+    return ret
+
+
+def get_branch_contents(ref: str) -> list[str]:
+    """Get the list of directories in a branch."""
+    stdout = check_output(["git", "ls-tree", "-d", "--name-only", ref])
+    return report_output(stdout, "Branch contents")
+
+
+def get_sorted_tags_list() -> list[str]:
+    """Get a list of sorted tags in descending order from the repository."""
+    stdout = check_output(["git", "tag", "-l", "--sort=-v:refname"])
+    return report_output(stdout, "Tags list")
+
+
+def get_versions(ref: str, add: str | None) -> list[str]:
+    """Generate the file containing the list of all GitHub Pages builds."""
+    # Get the directories (i.e. builds) from the GitHub Pages branch
+    try:
+        builds = set(get_branch_contents(ref))
+    except CalledProcessError:
+        builds = set()
+        logging.warning(f"Cannot get {ref} contents")
+
+    # Add and remove from the list of builds
+    if add:
+        builds.add(add)
+
+    # Get a sorted list of tags
+    tags = get_sorted_tags_list()
+
+    # Make the sorted versions list from main branches and tags
+    versions: list[str] = []
+    for version in ["master", "main"] + tags:
+        if version in builds:
+            versions.append(version)
+            builds.remove(version)
+
+    # Add in anything that is left to the bottom
+    versions += sorted(builds)
+    print(f"Sorted versions: {versions}")
+    return versions
+
+
+def write_json(path: Path, repository: str, versions: list[str]):
+    """Write the JSON switcher to path."""
+    org, repo_name = repository.split("/")
+    struct = [
+        {"version": version, "url": f"https://{org}.github.io/{repo_name}/{version}/"}
+        for version in versions
+    ]
+    text = json.dumps(struct, indent=2)
+    print(f"JSON switcher:\n{text}")
+    path.write_text(text, encoding="utf-8")
+
+
+def main(args=None):
+    """Parse args and write switcher."""
+    parser = ArgumentParser(
+        description="Make a versions.json file from gh-pages directories"
+    )
+    parser.add_argument(
+        "--add",
+        help="Add this directory to the list of existing directories",
+    )
+    parser.add_argument(
+        "repository",
+        help="The GitHub org and repository name: ORG/REPO",
+    )
+    parser.add_argument(
+        "output",
+        type=Path,
+        help="Path of write switcher.json to",
+    )
+    args = parser.parse_args(args)
+
+    # Write the versions file
+    versions = get_versions("origin/gh-pages", args.add)
+    write_json(args.output, args.repository, versions)
+
+
+if __name__ == "__main__":
+    main()