sgpu 0.6.0__tar.gz

sgpu-0.6.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 alex6095
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sgpu-0.6.0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.4
+Name: sgpu
+Version: 0.6.0
+Summary: Simple GPU monitor for the SGVR H200 lab (MLXP) - kubectl-native dashboards, in-pod TUI, per-user usage stats
+Author: alex6095
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/alex6095/sgpu
+Project-URL: Repository, https://github.com/alex6095/sgpu
+Keywords: gpu,kubernetes,monitoring,nvml,nvitop,h200
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# SGPU
+
+**SGVR GPU** / **S**imple **GPU** monitor for the lab's MLXP H200 nodes —
+check the GPUs before you launch a pod.
+
+- Every GPU process is attributed to its **pod and owner** (not just a PID).
+- **In-pod TUI** (`kubectl exec -it`) — smooth like nvitop, scroll/sort/filter.
+- **Usage stats 24/7**: per-owner GPU-hours, awards, GitHub-style activity
+  calendar, idle-allocation warnings (KST).
+- Shared **storage (pv-01/pv-02) usage** at a glance.
+- Monitor pod is read-only, always-on, and requests **no GPU**.
+
+## Install
+
+```bash
+uvx sgpu            # run without installing (uv)
+pipx install sgpu   # or pipx
+pip install sgpu    # or plain pip (WSL/Ubuntu: add --user --break-system-packages)
+```
+
+Needs `kubectl` configured for the MLXP namespace
+([kubectl setup](#kubectl-setup-linuxwsl)).
+
+## Use
+
+```text
+sgpu               interactive TUI      sgpu stats [days]  usage report + awards
+sgpu once          one-shot dashboard   sgpu apps          processes + owners
+sgpu watch [sec]   dumb-terminal loop   sgpu nvitop        raw nvitop
+sgpu pods|smi|gpustat|json|health|version|--help
+```
+
+TUI keys: `j/k` scroll · `Tab` pane · `s` sort · `o` owner filter · `p` pause · `q` quit.
+Options: `-n` namespace, `--pod`, `-r` refresh, `--no-color`. Env: `SGPU_NAMESPACE`, `SGPU_POD`.
+
+### Zero-install (kubectl only)
+
+```bash
+kubectl exec -it -n p-sgvr-node-02 sangmin-gpu-monitor -- python3 /opt/gpu-monitor/tui.py
+kubectl exec -n p-sgvr-node-02 sangmin-gpu-monitor -- curl -fsS http://127.0.0.1:8080/table
+```
+
+Endpoints on `:8080`: `/table /apps /json /stats /pods /smi /topo /gpustat
+/health /version /stats/files /stats/raw?date=` (`?color=1&cols=N&ascii=1`).
+
+## Stats
+
+Sampled every 15 s around the clock into raw JSONL (full fidelity — future
+tools can recompute anything), gzipped + rolled up daily, stored on the
+shared volume at `pv-01/sangmin/sgpu`. Retention 365 d, capped at 2 GB.
+`sgpu stats 30` shows the leaderboard, awards, daily activity calendar and
+KST hour heatmap. Raw export: `/stats/raw?date=YYYYMMDD`.
+
+> The monitor pod must stay running for stats to accumulate — it is designed
+> to (tini init, `restartPolicy: Always`, no GPU held).
+
+## Deploy / operate
+
+Image push works **from anywhere** via the registry's public endpoint
+(`sgvr-registry.kr.ncr.ntruss.com`); the cluster pulls via the private
+endpoint (`vnxb4cz3.kr.private-ncr.ntruss.com`, in-cluster only — preferred
+per the MLXP guide). API key: NCP console → Access Management.
+
+```bash
+docker login sgvr-registry.kr.ncr.ntruss.com
+docker build -f docker/Dockerfile.gpu-monitor -t sgvr-registry.kr.ncr.ntruss.com/sangmin/gpu-monitor:TAG .
+docker push sgvr-registry.kr.ncr.ntruss.com/sangmin/gpu-monitor:TAG
+
+kubectl delete pod sangmin-gpu-monitor -n p-sgvr-node-02 --ignore-not-found
+kubectl apply -f k8s/gpu-monitor.yaml   # pods are immutable: delete + apply
+kubectl wait --for=condition=Ready pod/sangmin-gpu-monitor -n p-sgvr-node-02 --timeout=180s
+```
+
+Optional (enables the pod-allocation view + idle stats; kubelet syncs it in
+within a minute, no restart):
+
+```bash
+kubectl -n p-sgvr-node-02 create secret generic sgpu-kubeconfig --from-file=config=$HOME/.kube/config
+```
+
+> Anyone with exec access to the monitor pod can read that token — fine
+> inside a trusting lab namespace; use your least-privileged kubeconfig.
+
+## kubectl setup (Linux/WSL)
+
+```bash
+mkdir -p ~/.local/bin ~/.kube
+V=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
+curl -fsSL -o ~/.local/bin/kubectl "https://dl.k8s.io/release/${V}/bin/linux/amd64/kubectl" && chmod +x ~/.local/bin/kubectl
+cp /path/to/sgvr-node-02-kubeconfig.yaml ~/.kube/config && chmod 600 ~/.kube/config
+kubectl get pods -n p-sgvr-node-02   # connectivity test
+```
+
+## Development
+
+```bash
+SGPU_MOCK=1 python3 tools/gpu-monitor/server.py   # full pipeline, no GPU needed
+SGPU_MOCK=1 python3 tools/gpu-monitor/tui.py
+python3 -m unittest discover -s tests
+```
+
+How it works: `sgpu` (thin Python client) → `kubectl exec` → monitor pod
+(privileged, hostPID) where server.py renders everything; process→pod
+attribution reads `/proc/<pid>/environ` (`HOSTNAME` = pod name), owner =
+pod-name prefix. Known limits: pods overriding `spec.hostname` and MPS show
+as `?`.
+
+Troubleshooting: broken terminal after a dropped TUI → `reset` · frozen TUI
+→ rerun `sgpu` · garbled bars → Windows Terminal or `--no-color`.

sgpu-0.6.0/README.md

@@ -0,0 +1,109 @@
+# SGPU
+
+**SGVR GPU** / **S**imple **GPU** monitor for the lab's MLXP H200 nodes —
+check the GPUs before you launch a pod.
+
+- Every GPU process is attributed to its **pod and owner** (not just a PID).
+- **In-pod TUI** (`kubectl exec -it`) — smooth like nvitop, scroll/sort/filter.
+- **Usage stats 24/7**: per-owner GPU-hours, awards, GitHub-style activity
+  calendar, idle-allocation warnings (KST).
+- Shared **storage (pv-01/pv-02) usage** at a glance.
+- Monitor pod is read-only, always-on, and requests **no GPU**.
+
+## Install
+
+```bash
+uvx sgpu            # run without installing (uv)
+pipx install sgpu   # or pipx
+pip install sgpu    # or plain pip (WSL/Ubuntu: add --user --break-system-packages)
+```
+
+Needs `kubectl` configured for the MLXP namespace
+([kubectl setup](#kubectl-setup-linuxwsl)).
+
+## Use
+
+```text
+sgpu               interactive TUI      sgpu stats [days]  usage report + awards
+sgpu once          one-shot dashboard   sgpu apps          processes + owners
+sgpu watch [sec]   dumb-terminal loop   sgpu nvitop        raw nvitop
+sgpu pods|smi|gpustat|json|health|version|--help
+```
+
+TUI keys: `j/k` scroll · `Tab` pane · `s` sort · `o` owner filter · `p` pause · `q` quit.
+Options: `-n` namespace, `--pod`, `-r` refresh, `--no-color`. Env: `SGPU_NAMESPACE`, `SGPU_POD`.
+
+### Zero-install (kubectl only)
+
+```bash
+kubectl exec -it -n p-sgvr-node-02 sangmin-gpu-monitor -- python3 /opt/gpu-monitor/tui.py
+kubectl exec -n p-sgvr-node-02 sangmin-gpu-monitor -- curl -fsS http://127.0.0.1:8080/table
+```
+
+Endpoints on `:8080`: `/table /apps /json /stats /pods /smi /topo /gpustat
+/health /version /stats/files /stats/raw?date=` (`?color=1&cols=N&ascii=1`).
+
+## Stats
+
+Sampled every 15 s around the clock into raw JSONL (full fidelity — future
+tools can recompute anything), gzipped + rolled up daily, stored on the
+shared volume at `pv-01/sangmin/sgpu`. Retention 365 d, capped at 2 GB.
+`sgpu stats 30` shows the leaderboard, awards, daily activity calendar and
+KST hour heatmap. Raw export: `/stats/raw?date=YYYYMMDD`.
+
+> The monitor pod must stay running for stats to accumulate — it is designed
+> to (tini init, `restartPolicy: Always`, no GPU held).
+
+## Deploy / operate
+
+Image push works **from anywhere** via the registry's public endpoint
+(`sgvr-registry.kr.ncr.ntruss.com`); the cluster pulls via the private
+endpoint (`vnxb4cz3.kr.private-ncr.ntruss.com`, in-cluster only — preferred
+per the MLXP guide). API key: NCP console → Access Management.
+
+```bash
+docker login sgvr-registry.kr.ncr.ntruss.com
+docker build -f docker/Dockerfile.gpu-monitor -t sgvr-registry.kr.ncr.ntruss.com/sangmin/gpu-monitor:TAG .
+docker push sgvr-registry.kr.ncr.ntruss.com/sangmin/gpu-monitor:TAG
+
+kubectl delete pod sangmin-gpu-monitor -n p-sgvr-node-02 --ignore-not-found
+kubectl apply -f k8s/gpu-monitor.yaml   # pods are immutable: delete + apply
+kubectl wait --for=condition=Ready pod/sangmin-gpu-monitor -n p-sgvr-node-02 --timeout=180s
+```
+
+Optional (enables the pod-allocation view + idle stats; kubelet syncs it in
+within a minute, no restart):
+
+```bash
+kubectl -n p-sgvr-node-02 create secret generic sgpu-kubeconfig --from-file=config=$HOME/.kube/config
+```
+
+> Anyone with exec access to the monitor pod can read that token — fine
+> inside a trusting lab namespace; use your least-privileged kubeconfig.
+
+## kubectl setup (Linux/WSL)
+
+```bash
+mkdir -p ~/.local/bin ~/.kube
+V=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
+curl -fsSL -o ~/.local/bin/kubectl "https://dl.k8s.io/release/${V}/bin/linux/amd64/kubectl" && chmod +x ~/.local/bin/kubectl
+cp /path/to/sgvr-node-02-kubeconfig.yaml ~/.kube/config && chmod 600 ~/.kube/config
+kubectl get pods -n p-sgvr-node-02   # connectivity test
+```
+
+## Development
+
+```bash
+SGPU_MOCK=1 python3 tools/gpu-monitor/server.py   # full pipeline, no GPU needed
+SGPU_MOCK=1 python3 tools/gpu-monitor/tui.py
+python3 -m unittest discover -s tests
+```
+
+How it works: `sgpu` (thin Python client) → `kubectl exec` → monitor pod
+(privileged, hostPID) where server.py renders everything; process→pod
+attribution reads `/proc/<pid>/environ` (`HOSTNAME` = pod name), owner =
+pod-name prefix. Known limits: pods overriding `spec.hostname` and MPS show
+as `?`.
+
+Troubleshooting: broken terminal after a dropped TUI → `reset` · frozen TUI
+→ rerun `sgpu` · garbled bars → Windows Terminal or `--no-color`.

sgpu-0.6.0/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sgpu"
+version = "0.6.0"
+description = "Simple GPU monitor for the SGVR H200 lab (MLXP) - kubectl-native dashboards, in-pod TUI, per-user usage stats"
+readme = "README.md"
+requires-python = ">=3.8"
+license = "MIT"
+authors = [{ name = "alex6095" }]
+keywords = ["gpu", "kubernetes", "monitoring", "nvml", "nvitop", "h200"]
+classifiers = [
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Topic :: System :: Monitoring",
+]
+
+[project.scripts]
+sgpu = "sgpu.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/alex6095/sgpu"
+Repository = "https://github.com/alex6095/sgpu"
+
+[tool.setuptools]
+packages = ["sgpu"]

sgpu-0.6.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sgpu-0.6.0/sgpu/__init__.py

@@ -0,0 +1,3 @@
+"""sgpu — simple GPU monitor client for the SGVR lab MLXP cluster."""
+
+__version__ = "0.6.0"

sgpu-0.6.0/sgpu/__main__.py

@@ -0,0 +1,4 @@
+from sgpu.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

sgpu-0.6.0/sgpu/cli.py

@@ -0,0 +1,176 @@
+"""sgpu command-line client.
+
+A thin cross-platform wrapper: all rendering happens inside the monitor pod,
+this client only shells out to kubectl (fetch server-rendered text, or hand
+the terminal to the in-pod TUI via `kubectl exec -it`).
+"""
+
+import argparse
+import os
+import shutil
+import subprocess
+import sys
+import time
+
+from sgpu import __version__
+
+DEFAULT_NAMESPACE = os.environ.get("SGPU_NAMESPACE", "p-sgvr-node-02")
+DEFAULT_POD = os.environ.get("SGPU_POD", "sangmin-gpu-monitor")
+DEPLOY_URL = ("https://raw.githubusercontent.com/alex6095/sgpu/main/"
+              "k8s/gpu-monitor.yaml")
+
+COMMANDS = ("top", "once", "watch", "apps", "stats", "nvitop", "pods",
+            "smi", "gpustat", "json", "health", "version")
+
+EPILOG = """commands:
+  (none) | top     interactive TUI (scroll, sort, owner filter)
+  once             one-shot dashboard
+  watch [sec]      simple refresh loop (for dumb terminals)
+  apps             GPU process table with pod owners
+  stats [days]     per-owner usage report (default 7 days)
+  nvitop           raw nvitop TUI
+  pods | smi | gpustat | json | health | version
+"""
+
+
+def _enable_ansi():
+    if os.name == "nt":
+        os.system("")  # nudges conhost/Windows Terminal into VT mode
+    try:
+        sys.stdout.reconfigure(encoding="utf-8")
+        sys.stderr.reconfigure(encoding="utf-8")
+    except (AttributeError, OSError):
+        pass
+
+
+def _hint_on_failure(args, stderr_text):
+    print("sgpu: cannot reach monitor pod '%s' in namespace '%s'"
+          % (args.pod, args.namespace), file=sys.stderr)
+    tail = [line for line in stderr_text.strip().splitlines()][-3:]
+    for line in tail:
+        print(line, file=sys.stderr)
+    lowered = stderr_text.lower()
+    if "notfound" in lowered or "not found" in lowered:
+        print("hint: the monitor pod is not running. Deploy it with:",
+              file=sys.stderr)
+        print("  kubectl apply -f " + DEPLOY_URL, file=sys.stderr)
+    elif "unauthorized" in lowered or "forbidden" in lowered \
+            or "credentials" in lowered:
+        print("hint: check your access: kubectl auth can-i get pods -n %s"
+              % args.namespace, file=sys.stderr)
+
+
+def _fetch(args, path):
+    result = subprocess.run(
+        ["kubectl", "exec", "-n", args.namespace, args.pod, "--",
+         "curl", "-fsS", "http://127.0.0.1:8080" + path],
+        capture_output=True, text=True, encoding="utf-8", errors="replace")
+    if result.returncode != 0:
+        _hint_on_failure(args, result.stderr or result.stdout)
+        return None
+    return result.stdout
+
+
+def _color_param(args):
+    use = (not args.no_color) and sys.stdout.isatty()
+    return "color=1" if use else "color=0"
+
+
+def _cols_param():
+    cols = shutil.get_terminal_size(fallback=(120, 40)).columns
+    return "cols=%d" % max(40, cols)
+
+
+def _print(text):
+    if text is not None:
+        sys.stdout.write(text)
+        if not text.endswith("\n"):
+            sys.stdout.write("\n")
+
+
+def _interactive(args, pod_command):
+    if not (sys.stdout.isatty() and sys.stdin.isatty()):
+        _print(_fetch(args, "/table?color=0&" + _cols_param()))
+        return 0
+    return subprocess.call(
+        ["kubectl", "exec", "-it", "-n", args.namespace, args.pod, "--"]
+        + pod_command)
+
+
+def _watch(args):
+    seconds = args.number if args.number else args.refresh
+    esc = "\x1b"
+    sys.stdout.write(esc + "[?1049h" + esc + "[?25l")
+    try:
+        while True:
+            frame = _fetch(args, "/table?%s&%s"
+                           % (_color_param(args), _cols_param()))
+            if frame is None:
+                frame = "sgpu: fetch failed; retrying..."
+            sys.stdout.write(esc + "[H" + frame + "\n" + esc + "[0J")
+            sys.stdout.flush()
+            time.sleep(max(1, seconds))
+    except KeyboardInterrupt:
+        return 0
+    finally:
+        sys.stdout.write(esc + "[?25h" + esc + "[?1049l")
+        sys.stdout.flush()
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(
+        prog="sgpu",
+        description="Simple GPU monitor for the SGVR lab MLXP cluster.",
+        epilog=EPILOG,
+        formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("command", nargs="?", default="top",
+                        choices=COMMANDS, metavar="command")
+    parser.add_argument("number", nargs="?", type=int, default=0,
+                        metavar="N",
+                        help="days for stats, seconds for watch")
+    parser.add_argument("-n", "--namespace", default=DEFAULT_NAMESPACE)
+    parser.add_argument("--pod", default=DEFAULT_POD)
+    parser.add_argument("-r", "--refresh", type=int, default=2,
+                        help="TUI/watch refresh interval (default 2)")
+    parser.add_argument("--no-color", action="store_true")
+    parser.add_argument("-V", "--version", action="version",
+                        version="sgpu " + __version__)
+    args = parser.parse_args(argv)
+
+    if shutil.which("kubectl") is None:
+        print("sgpu: kubectl not found on PATH", file=sys.stderr)
+        print("hint: install kubectl and configure the MLXP kubeconfig "
+              "(see the sgpu README)", file=sys.stderr)
+        return 127
+
+    _enable_ansi()
+    query = "%s&%s" % (_color_param(args), _cols_param())
+
+    if args.command == "top":
+        return _interactive(args, ["python3", "/opt/gpu-monitor/tui.py",
+                                   str(args.refresh)])
+    if args.command == "nvitop":
+        return _interactive(args, ["nvitop"])
+    if args.command == "watch":
+        return _watch(args)
+
+    paths = {
+        "once": "/table?" + query,
+        "apps": "/apps?" + query,
+        "stats": "/stats?days=%d&%s" % (args.number or 7, query),
+        "pods": "/pods",
+        "smi": "/smi",
+        "gpustat": "/gpustat",
+        "json": "/json",
+        "health": "/health",
+        "version": "/version",
+    }
+    text = _fetch(args, paths[args.command])
+    if text is None:
+        return 1
+    _print(text)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

sgpu-0.6.0/sgpu.egg-info/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.4
+Name: sgpu
+Version: 0.6.0
+Summary: Simple GPU monitor for the SGVR H200 lab (MLXP) - kubectl-native dashboards, in-pod TUI, per-user usage stats
+Author: alex6095
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/alex6095/sgpu
+Project-URL: Repository, https://github.com/alex6095/sgpu
+Keywords: gpu,kubernetes,monitoring,nvml,nvitop,h200
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# SGPU
+
+**SGVR GPU** / **S**imple **GPU** monitor for the lab's MLXP H200 nodes —
+check the GPUs before you launch a pod.
+
+- Every GPU process is attributed to its **pod and owner** (not just a PID).
+- **In-pod TUI** (`kubectl exec -it`) — smooth like nvitop, scroll/sort/filter.
+- **Usage stats 24/7**: per-owner GPU-hours, awards, GitHub-style activity
+  calendar, idle-allocation warnings (KST).
+- Shared **storage (pv-01/pv-02) usage** at a glance.
+- Monitor pod is read-only, always-on, and requests **no GPU**.
+
+## Install
+
+```bash
+uvx sgpu            # run without installing (uv)
+pipx install sgpu   # or pipx
+pip install sgpu    # or plain pip (WSL/Ubuntu: add --user --break-system-packages)
+```
+
+Needs `kubectl` configured for the MLXP namespace
+([kubectl setup](#kubectl-setup-linuxwsl)).
+
+## Use
+
+```text
+sgpu               interactive TUI      sgpu stats [days]  usage report + awards
+sgpu once          one-shot dashboard   sgpu apps          processes + owners
+sgpu watch [sec]   dumb-terminal loop   sgpu nvitop        raw nvitop
+sgpu pods|smi|gpustat|json|health|version|--help
+```
+
+TUI keys: `j/k` scroll · `Tab` pane · `s` sort · `o` owner filter · `p` pause · `q` quit.
+Options: `-n` namespace, `--pod`, `-r` refresh, `--no-color`. Env: `SGPU_NAMESPACE`, `SGPU_POD`.
+
+### Zero-install (kubectl only)
+
+```bash
+kubectl exec -it -n p-sgvr-node-02 sangmin-gpu-monitor -- python3 /opt/gpu-monitor/tui.py
+kubectl exec -n p-sgvr-node-02 sangmin-gpu-monitor -- curl -fsS http://127.0.0.1:8080/table
+```
+
+Endpoints on `:8080`: `/table /apps /json /stats /pods /smi /topo /gpustat
+/health /version /stats/files /stats/raw?date=` (`?color=1&cols=N&ascii=1`).
+
+## Stats
+
+Sampled every 15 s around the clock into raw JSONL (full fidelity — future
+tools can recompute anything), gzipped + rolled up daily, stored on the
+shared volume at `pv-01/sangmin/sgpu`. Retention 365 d, capped at 2 GB.
+`sgpu stats 30` shows the leaderboard, awards, daily activity calendar and
+KST hour heatmap. Raw export: `/stats/raw?date=YYYYMMDD`.
+
+> The monitor pod must stay running for stats to accumulate — it is designed
+> to (tini init, `restartPolicy: Always`, no GPU held).
+
+## Deploy / operate
+
+Image push works **from anywhere** via the registry's public endpoint
+(`sgvr-registry.kr.ncr.ntruss.com`); the cluster pulls via the private
+endpoint (`vnxb4cz3.kr.private-ncr.ntruss.com`, in-cluster only — preferred
+per the MLXP guide). API key: NCP console → Access Management.
+
+```bash
+docker login sgvr-registry.kr.ncr.ntruss.com
+docker build -f docker/Dockerfile.gpu-monitor -t sgvr-registry.kr.ncr.ntruss.com/sangmin/gpu-monitor:TAG .
+docker push sgvr-registry.kr.ncr.ntruss.com/sangmin/gpu-monitor:TAG
+
+kubectl delete pod sangmin-gpu-monitor -n p-sgvr-node-02 --ignore-not-found
+kubectl apply -f k8s/gpu-monitor.yaml   # pods are immutable: delete + apply
+kubectl wait --for=condition=Ready pod/sangmin-gpu-monitor -n p-sgvr-node-02 --timeout=180s
+```
+
+Optional (enables the pod-allocation view + idle stats; kubelet syncs it in
+within a minute, no restart):
+
+```bash
+kubectl -n p-sgvr-node-02 create secret generic sgpu-kubeconfig --from-file=config=$HOME/.kube/config
+```
+
+> Anyone with exec access to the monitor pod can read that token — fine
+> inside a trusting lab namespace; use your least-privileged kubeconfig.
+
+## kubectl setup (Linux/WSL)
+
+```bash
+mkdir -p ~/.local/bin ~/.kube
+V=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
+curl -fsSL -o ~/.local/bin/kubectl "https://dl.k8s.io/release/${V}/bin/linux/amd64/kubectl" && chmod +x ~/.local/bin/kubectl
+cp /path/to/sgvr-node-02-kubeconfig.yaml ~/.kube/config && chmod 600 ~/.kube/config
+kubectl get pods -n p-sgvr-node-02   # connectivity test
+```
+
+## Development
+
+```bash
+SGPU_MOCK=1 python3 tools/gpu-monitor/server.py   # full pipeline, no GPU needed
+SGPU_MOCK=1 python3 tools/gpu-monitor/tui.py
+python3 -m unittest discover -s tests
+```
+
+How it works: `sgpu` (thin Python client) → `kubectl exec` → monitor pod
+(privileged, hostPID) where server.py renders everything; process→pod
+attribution reads `/proc/<pid>/environ` (`HOSTNAME` = pod name), owner =
+pod-name prefix. Known limits: pods overriding `spec.hostname` and MPS show
+as `?`.
+
+Troubleshooting: broken terminal after a dropped TUI → `reset` · frozen TUI
+→ rerun `sgpu` · garbled bars → Windows Terminal or `--no-color`.

sgpu-0.6.0/sgpu.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE
+README.md
+pyproject.toml
+sgpu/__init__.py
+sgpu/__main__.py
+sgpu/cli.py
+sgpu.egg-info/PKG-INFO
+sgpu.egg-info/SOURCES.txt
+sgpu.egg-info/dependency_links.txt
+sgpu.egg-info/entry_points.txt
+sgpu.egg-info/top_level.txt
+tests/test_statsagg.py
+tests/test_statsdb.py

sgpu-0.6.0/sgpu.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

sgpu-0.6.0/sgpu.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+sgpu = sgpu.cli:main

sgpu-0.6.0/sgpu.egg-info/top_level.txt

@@ -0,0 +1 @@
+sgpu

sgpu-0.6.0/tests/test_statsagg.py

@@ -0,0 +1,325 @@
+"""Unit tests for statsagg (aggregation math + rendering)."""
+
+import gzip
+import json
+import os
+import sys
+import tempfile
+import unittest
+from datetime import datetime, timezone
+
+# --- sys.path shim: make tools/gpu-monitor importable ---
+_HERE = os.path.dirname(os.path.abspath(__file__))
+_MOD_DIR = os.path.join(_HERE, "..", "tools", "gpu-monitor")
+sys.path.insert(0, os.path.abspath(_MOD_DIR))
+
+import statsagg  # noqa: E402
+
+
+def ts_at(base_epoch, offset_s):
+    dt = datetime.fromtimestamp(base_epoch + offset_s, tz=timezone.utc)
+    return dt.strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def write_day(d, date_str, records, gzipped=False):
+    """Write a list of sample dicts as JSONL; return the path."""
+    name = "samples-%s.jsonl" % date_str
+    path = os.path.join(d, name)
+    text = "".join(json.dumps(r, separators=(",", ":")) + "\n" for r in records)
+    if gzipped:
+        path += ".gz"
+        with gzip.open(path, "wt", encoding="utf-8") as fh:
+            fh.write(text)
+    else:
+        with open(path, "w", encoding="utf-8") as fh:
+            fh.write(text)
+    return path
+
+
+def sample(ts, gpus=None, procs=None, pods=None):
+    r = {"v": 1, "ts": ts, "node": "n1", "gpus": gpus or [], "procs": procs or []}
+    if pods is not None:
+        r["pods"] = pods
+    return r
+
+
+def gpu(i, util=None, mem_total=143771, name="NVIDIA H200"):
+    return {"i": i, "util": util, "mem_total": mem_total, "name": name}
+
+
+def proc(pid, gi, owner, mem=None, sm=None, pod=None):
+    return {"pid": pid, "gpu": gi, "owner": owner, "mem": mem, "sm": sm,
+            "pod": pod}
+
+
+class TestDtGapRule(unittest.TestCase):
+    def test_gap_credit_bounded(self):
+        # 3 samples 15s apart, then a 2h gap, then 1 more.
+        base = 1751860000  # arbitrary epoch, aligned to a second
+        recs = [
+            sample(ts_at(base, 0), gpus=[gpu(0, 50)],
+                   procs=[proc(1, 0, "a", sm=90)]),
+            sample(ts_at(base, 15), gpus=[gpu(0, 50)],
+                   procs=[proc(1, 0, "a", sm=90)]),
+            sample(ts_at(base, 30), gpus=[gpu(0, 50)],
+                   procs=[proc(1, 0, "a", sm=90)]),
+            sample(ts_at(base, 30 + 7200), gpus=[gpu(0, 50)],
+                   procs=[proc(1, 0, "a", sm=90)]),
+        ]
+        with tempfile.TemporaryDirectory() as d:
+            path = write_day(d, "20260707", recs)
+            roll = statsagg.build_rollup(path)
+        self.assertEqual(roll["samples"], 4)
+        self.assertEqual(roll["interval_hint"], 15)
+        # dt: 15 (first) + 15 + 15 + min(7200, 30) = 75
+        self.assertAlmostEqual(roll["coverage_seconds"], 75.0, places=6)
+        # owner "a" active on 1 gpu each sample -> gpu_seconds == coverage
+        self.assertAlmostEqual(roll["owners"]["a"]["gpu_seconds"], 75.0,
+                               places=6)
+
+
+class TestSharedGpuUtil(unittest.TestCase):
+    def test_two_owners_share_one_gpu(self):
+        # One gpu, util 80, two owners each with a proc on it. Single sample.
+        base = 1751860000
+        recs = [
+            sample(ts_at(base, 0), gpus=[gpu(0, 80)],
+                   procs=[proc(1, 0, "a"), proc(2, 0, "b")]),
+        ]
+        with tempfile.TemporaryDirectory() as d:
+            path = write_day(d, "20260707", recs)
+            roll = statsagg.build_rollup(path)
+        # First sample dt = interval_hint (fallback 15, no gaps).
+        dt = roll["interval_hint"]
+        a = roll["owners"]["a"]
+        b = roll["owners"]["b"]
+        # Both charged gpu_seconds for the shared gpu.
+        self.assertAlmostEqual(a["gpu_seconds"], 1 * dt, places=6)
+        self.assertAlmostEqual(b["gpu_seconds"], 1 * dt, places=6)
+        # Device util split by 2 sharers: 80*dt/2 each.
+        self.assertAlmostEqual(a["util_wsum"], 80 * dt / 2, places=6)
+        self.assertAlmostEqual(a["util_weight"], dt, places=6)
+        self.assertAlmostEqual(b["util_wsum"], 80 * dt / 2, places=6)
+
+
+class TestIdleAllocation(unittest.TestCase):
+    def test_idle_when_pod_uses_fewer_gpus(self):
+        # Pod req=4 but procs only on 1 gpu -> idle 3*dt.
+        base = 1751860000
+        pods = [{"pod": "a-x", "owner": "a", "req": 4, "phase": "Running",
+                 "start": ts_at(base, -3600)}]
+        recs = [
+            sample(ts_at(base, 0), gpus=[gpu(0, 50)],
+                   procs=[proc(1, 0, "a", pod="a-x")], pods=pods),
+        ]
+        with tempfile.TemporaryDirectory() as d:
+            path = write_day(d, "20260707", recs)
+            roll = statsagg.build_rollup(path)
+        dt = roll["interval_hint"]
+        a = roll["owners"]["a"]
+        self.assertAlmostEqual(a["alloc_gpu_seconds"], 4 * dt, places=6)
+        self.assertAlmostEqual(a["idle_gpu_seconds"], 3 * dt, places=6)
+        self.assertAlmostEqual(roll["pods_coverage_seconds"], dt, places=6)
+
+
+class TestKstBinning(unittest.TestCase):
+    def test_ts_2330_utc_maps_to_kst_hour_8(self):
+        # 23:30 UTC + 9h = 08:30 next day -> KST hour 8.
+        recs = [
+            sample("2026-07-07T23:30:00Z", gpus=[gpu(0, 50)],
+                   procs=[proc(1, 0, "a")]),
+        ]
+        with tempfile.TemporaryDirectory() as d:
+            path = write_day(d, "20260707", recs)
+            roll = statsagg.build_rollup(path)
+        hist = roll["owners"]["a"]["hour_hist_kst"]
+        dt = roll["interval_hint"]
+        self.assertAlmostEqual(hist[8], 1 * dt, places=6)
+        for h in range(24):
+            if h != 8:
+                self.assertEqual(hist[h], 0.0)
+
+
+class TestMergeEqualsConcat(unittest.TestCase):
+    def test_merge_two_days_equals_build_over_concat(self):
+        # The two groups are contiguous (group 2 starts exactly one interval
+        # after group 1's last sample) so that in the concatenated build the
+        # first sample of group 2 is credited dt = min(15, 2*15) = 15, matching
+        # the first-of-day credit it receives when group 2 is built alone. This
+        # is the condition under which merge(day1,day2) == build(day1+day2).
+        base = 1751860000  # some time
+        day1 = [
+            sample(ts_at(base, 0), gpus=[gpu(0, 60)],
+                   procs=[proc(1, 0, "a", sm=70, mem=1000)]),
+            sample(ts_at(base, 15), gpus=[gpu(0, 60)],
+                   procs=[proc(1, 0, "a", sm=70, mem=1000)]),
+        ]
+        base2 = base + 30  # 15s after day1's last sample -> contiguous
+        day2 = [
+            sample(ts_at(base2, 0), gpus=[gpu(1, 40)],
+                   procs=[proc(2, 1, "a", sm=30, mem=500),
+                          proc(3, 1, "b", sm=None, mem=200)]),
+            sample(ts_at(base2, 15), gpus=[gpu(1, 40)],
+                   procs=[proc(2, 1, "a", sm=30, mem=500),
+                          proc(3, 1, "b", sm=None, mem=200)]),
+        ]
+        with tempfile.TemporaryDirectory() as d:
+            p1 = write_day(d, "20260707", day1)
+            p2 = write_day(d, "20260708", day2)
+            r1 = statsagg.build_rollup(p1)
+            r2 = statsagg.build_rollup(p2)
+            merged = statsagg.merge_rollups([r1, r2])
+
+            # Build over the concatenation of both days (same interval hint 15).
+            pc = write_day(d, "20260709", day1 + day2)
+            concat = statsagg.build_rollup(pc)
+
+        self.assertAlmostEqual(merged["coverage_seconds"],
+                               concat["coverage_seconds"], places=5)
+        for owner in ("a", "b"):
+            m = merged["owners"][owner]
+            c = concat["owners"][owner]
+            for key in ("gpu_seconds", "sm_wsum", "sm_weight", "util_wsum",
+                        "util_weight", "mem_wsum", "mem_weight"):
+                self.assertAlmostEqual(m[key], c[key], places=5,
+                                       msg="%s.%s" % (owner, key))
+            self.assertEqual(m["mem_peak_mib"], c["mem_peak_mib"])
+            for h in range(24):
+                self.assertAlmostEqual(m["hour_hist_kst"][h],
+                                       c["hour_hist_kst"][h], places=5)
+
+
+class TestMalformedLineSkipped(unittest.TestCase):
+    def test_truncated_trailing_line(self):
+        base = 1751860000
+        good = sample(ts_at(base, 0), gpus=[gpu(0, 50)],
+                      procs=[proc(1, 0, "a")])
+        with tempfile.TemporaryDirectory() as d:
+            path = os.path.join(d, "samples-20260707.jsonl")
+            with open(path, "w", encoding="utf-8") as fh:
+                fh.write(json.dumps(good, separators=(",", ":")) + "\n")
+                fh.write('{"v":1,"ts":"2026-07-07T00:00:15Z","gpus":[{"i":0')
+            roll = statsagg.build_rollup(path)
+        # Only the one complete line counted.
+        self.assertEqual(roll["samples"], 1)
+
+
+class TestGzipTransparent(unittest.TestCase):
+    def test_reads_gz(self):
+        base = 1751860000
+        recs = [sample(ts_at(base, 0), gpus=[gpu(0, 50)],
+                       procs=[proc(1, 0, "a")])]
+        with tempfile.TemporaryDirectory() as d:
+            path = write_day(d, "20260707", recs, gzipped=True)
+            roll = statsagg.build_rollup(path)
+        self.assertEqual(roll["samples"], 1)
+
+
+class TestCurrentIdle(unittest.TestCase):
+    def test_detects_running_pod_with_no_procs(self):
+        # Today's samples in the last 30 min: pod running, zero procs on it.
+        now = datetime(2026, 7, 7, 12, 0, 0, tzinfo=timezone.utc)
+        now_epoch = now.timestamp()
+        pods = [{"pod": "z-idle", "owner": "z", "req": 4, "phase": "Running",
+                 "start": "2026-07-07T09:00:00Z"}]
+        recs = [
+            sample(ts_at(now_epoch, -600), gpus=[gpu(0, 0)], procs=[],
+                   pods=pods),
+            sample(ts_at(now_epoch, -300), gpus=[gpu(0, 0)], procs=[],
+                   pods=pods),
+        ]
+        with tempfile.TemporaryDirectory() as d:
+            write_day(d, "20260707", recs)
+            result = statsagg.query(d, days=1, now_fn=lambda: now)
+        idle = result["current_idle"]
+        self.assertEqual(len(idle), 1)
+        self.assertEqual(idle[0]["pod"], "z-idle")
+        self.assertEqual(idle[0]["owner"], "z")
+        self.assertEqual(idle[0]["req"], 4)
+        self.assertGreaterEqual(idle[0]["idle_minutes"], 9)
+
+    def test_pod_with_proc_not_idle(self):
+        now = datetime(2026, 7, 7, 12, 0, 0, tzinfo=timezone.utc)
+        now_epoch = now.timestamp()
+        pods = [{"pod": "z-busy", "owner": "z", "req": 4, "phase": "Running",
+                 "start": "2026-07-07T09:00:00Z"}]
+        recs = [
+            sample(ts_at(now_epoch, -600), gpus=[gpu(0, 90)],
+                   procs=[proc(1, 0, "z", pod="z-busy")], pods=pods),
+        ]
+        with tempfile.TemporaryDirectory() as d:
+            write_day(d, "20260707", recs)
+            result = statsagg.query(d, days=1, now_fn=lambda: now)
+        self.assertEqual(result["current_idle"], [])
+
+
+class TestQueryAndRender(unittest.TestCase):
+    def test_query_empty_dir(self):
+        with tempfile.TemporaryDirectory() as d:
+            now = datetime(2026, 7, 7, 12, 0, 0, tzinfo=timezone.utc)
+            result = statsagg.query(d, days=7, now_fn=lambda: now)
+        self.assertEqual(result["merged"]["samples"], 0)
+        self.assertEqual(result["dates_covered"], [])
+        self.assertEqual(result["current_idle"], [])
+        text = statsagg.render_stats_text(result)
+        self.assertIn("no samples recorded yet", text)
+
+    def test_render_smoke_with_owners(self):
+        base = 1751860000
+        pods = [{"pod": "yoonki-x", "owner": "yoonki", "req": 4,
+                 "phase": "Running", "start": ts_at(base, -3600)}]
+        recs = []
+        for i in range(400):  # enough gpu-hours to trip LOW-UTIL
+            recs.append(sample(
+                ts_at(base, i * 15),
+                gpus=[gpu(0, 10)],
+                procs=[proc(1, 0, "yoonki", sm=10, mem=1000, pod="yoonki-x")],
+                pods=pods))
+        now = datetime.fromtimestamp(base + 400 * 15, tz=timezone.utc)
+        with tempfile.TemporaryDirectory() as d:
+            # Put it on "today" so query reads it as raw.
+            date_str = now.strftime("%Y%m%d")
+            write_day(d, date_str, recs)
+            result = statsagg.query(d, days=1, now_fn=lambda: now)
+        text = statsagg.render_stats_text(result, color=False)
+        self.assertIn("yoonki", text)
+        self.assertIn("SGPU usage report", text)
+        self.assertIn("KST", text)
+        # Color variant must not raise and must differ (ANSI codes present).
+        ctext = statsagg.render_stats_text(result, color=True)
+        self.assertIn("\x1b[", ctext)
+
+    def test_list_files(self):
+        with tempfile.TemporaryDirectory() as d:
+            write_day(d, "20260706", [sample("2026-07-06T00:00:00Z")])
+            with open(os.path.join(d, "rollup-20260706.json"), "w",
+                      encoding="utf-8") as fh:
+                fh.write("{}")
+            with open(os.path.join(d, "ignore.txt"), "w",
+                      encoding="utf-8") as fh:
+                fh.write("x")
+            files = statsagg.list_files(d)
+            names = [f["name"] for f in files]
+            self.assertIn("samples-20260706.jsonl", names)
+            self.assertIn("rollup-20260706.json", names)
+            self.assertNotIn("ignore.txt", names)
+            for f in files:
+                self.assertIn("size", f)
+                self.assertIn("mtime_iso", f)
+
+    def test_iter_raw_lines_missing_raises(self):
+        with tempfile.TemporaryDirectory() as d:
+            with self.assertRaises(FileNotFoundError):
+                list(statsagg.iter_raw_lines(d, "20990101"))
+
+    def test_iter_raw_lines_reads_gz(self):
+        with tempfile.TemporaryDirectory() as d:
+            write_day(d, "20260706", [sample("2026-07-06T00:00:00Z")],
+                      gzipped=True)
+            lines = list(statsagg.iter_raw_lines(d, "20260706"))
+            self.assertEqual(len(lines), 1)
+            self.assertIn("2026-07-06T00:00:00Z", lines[0])
+
+
+if __name__ == "__main__":
+    unittest.main()

sgpu-0.6.0/tests/test_statsdb.py

@@ -0,0 +1,267 @@
+"""Unit tests for statsdb (sample recording + rotation)."""
+
+import gzip
+import json
+import os
+import sys
+import tempfile
+import unittest
+from datetime import datetime, timezone
+
+# --- sys.path shim: make tools/gpu-monitor importable ---
+_HERE = os.path.dirname(os.path.abspath(__file__))
+_MOD_DIR = os.path.join(_HERE, "..", "tools", "gpu-monitor")
+sys.path.insert(0, os.path.abspath(_MOD_DIR))
+
+import statsdb  # noqa: E402
+import statsagg  # noqa: E402
+
+
+def make_snapshot(ts="2026-07-07T04:12:30Z", with_pods=True, pods_ok=True):
+    snap = {
+        "schema": 2,
+        "source": "nvml",
+        "time_utc": ts,
+        "node": "h200-04-w-4b11",
+        "driver": "580.126.16",
+        "sgpu_version": "0.5.0",
+        "gpus": [{
+            "index": 0, "uuid": "GPU-abc", "name": "NVIDIA H200",
+            "util": 97, "mem_used_mib": 121004, "mem_total_mib": 143771,
+            "temp_c": 61, "power_w": 540.2, "power_limit_w": 700.0,
+            "owners": ["yoonki"],
+        }],
+        "procs": [{
+            "pid": 1234, "gpu_index": 0, "gpu_uuid": "GPU-abc",
+            "mem_mib": 40213, "owner": "yoonki", "pod": "yoonki-ume-xvrdr",
+            "pod_uid": None, "cmd": "x" * 200,
+            "started_utc": "2026-07-06T09:00:00Z", "sm_util": 85,
+            "attribution": "environ",
+        }],
+        "pods": {
+            "ok": pods_ok, "source": "kubeconfig", "error": None,
+            "rows": [{
+                "owner": "yoonki", "pod": "yoonki-ume-xvrdr", "node": "n",
+                "phase": "Running", "gpu": 4, "age": "19h7m",
+                "uid": "u", "start_iso": "2026-07-06T09:00:00Z", "active": 4,
+            }],
+        },
+    }
+    if not with_pods:
+        del snap["pods"]
+    return snap
+
+
+class TestSnapshotToSample(unittest.TestCase):
+    def test_field_mapping(self):
+        s = statsdb.snapshot_to_sample(make_snapshot())
+        self.assertEqual(s["v"], 1)
+        self.assertEqual(s["ts"], "2026-07-07T04:12:30Z")
+        self.assertEqual(s["node"], "h200-04-w-4b11")
+        self.assertEqual(s["driver"], "580.126.16")
+        g = s["gpus"][0]
+        self.assertEqual(g["i"], 0)
+        self.assertEqual(g["uuid"], "GPU-abc")
+        self.assertEqual(g["util"], 97)
+        self.assertEqual(g["mem"], 121004)
+        self.assertEqual(g["mem_total"], 143771)
+        self.assertEqual(g["temp"], 61)
+        self.assertEqual(g["pw"], 540.2)
+        self.assertEqual(g["pw_lim"], 700.0)
+        p = s["procs"][0]
+        self.assertEqual(p["pid"], 1234)
+        self.assertEqual(p["gpu"], 0)
+        self.assertEqual(p["mem"], 40213)
+        self.assertEqual(p["owner"], "yoonki")
+        self.assertEqual(p["pod"], "yoonki-ume-xvrdr")
+        self.assertEqual(p["started"], "2026-07-06T09:00:00Z")
+        self.assertEqual(p["sm"], 85)
+
+    def test_cmd_truncation(self):
+        s = statsdb.snapshot_to_sample(make_snapshot())
+        self.assertEqual(len(s["procs"][0]["cmd"]), 120)
+
+    def test_none_passthrough(self):
+        snap = make_snapshot()
+        snap["gpus"][0]["util"] = None
+        snap["gpus"][0]["temp_c"] = None
+        snap["gpus"][0]["power_w"] = None
+        snap["procs"][0]["owner"] = None
+        snap["procs"][0]["sm_util"] = None
+        snap["procs"][0]["mem_mib"] = None
+        s = statsdb.snapshot_to_sample(snap)
+        self.assertIsNone(s["gpus"][0]["util"])
+        self.assertIsNone(s["gpus"][0]["temp"])
+        self.assertIsNone(s["gpus"][0]["pw"])
+        self.assertIsNone(s["procs"][0]["owner"])
+        self.assertIsNone(s["procs"][0]["sm"])
+        self.assertIsNone(s["procs"][0]["mem"])
+        # None survives a JSON round trip as null.
+        rt = json.loads(json.dumps(s))
+        self.assertIsNone(rt["procs"][0]["owner"])
+
+    def test_pods_key_present_when_ok(self):
+        s = statsdb.snapshot_to_sample(make_snapshot(pods_ok=True))
+        self.assertIn("pods", s)
+        self.assertEqual(s["pods"][0]["pod"], "yoonki-ume-xvrdr")
+        self.assertEqual(s["pods"][0]["req"], 4)
+        self.assertEqual(s["pods"][0]["phase"], "Running")
+
+    def test_pods_key_omitted_when_not_ok(self):
+        s = statsdb.snapshot_to_sample(make_snapshot(pods_ok=False))
+        self.assertNotIn("pods", s)
+
+    def test_pods_key_omitted_when_absent(self):
+        s = statsdb.snapshot_to_sample(make_snapshot(with_pods=False))
+        self.assertNotIn("pods", s)
+
+
+class TestSampleOnce(unittest.TestCase):
+    def test_write_and_read_back(self):
+        with tempfile.TemporaryDirectory() as d:
+            fixed = datetime(2026, 7, 7, 4, 12, 30, tzinfo=timezone.utc)
+            calls = {"n": 0}
+
+            def collect(force=False):
+                calls["n"] += 1
+                self.assertTrue(force)
+                return make_snapshot()
+
+            path = statsdb._sample_once(collect, d, now_fn=lambda: fixed)
+            self.assertTrue(path.endswith("samples-20260707.jsonl"))
+            with open(path, "r", encoding="utf-8") as fh:
+                lines = fh.read().splitlines()
+            self.assertEqual(len(lines), 1)
+            rec = json.loads(lines[0])
+            self.assertEqual(rec["ts"], "2026-07-07T04:12:30Z")
+            self.assertEqual(calls["n"], 1)
+
+    def test_append_not_truncate(self):
+        with tempfile.TemporaryDirectory() as d:
+            fixed = datetime(2026, 7, 7, 4, 12, 30, tzinfo=timezone.utc)
+
+            def collect(force=False):
+                return make_snapshot()
+
+            statsdb._sample_once(collect, d, now_fn=lambda: fixed)
+            path = statsdb._sample_once(collect, d, now_fn=lambda: fixed)
+            with open(path, "r", encoding="utf-8") as fh:
+                lines = fh.read().splitlines()
+            self.assertEqual(len(lines), 2)
+
+
+class TestRotation(unittest.TestCase):
+    def _write_raw_day(self, d, date_str, n=3):
+        path = os.path.join(d, "samples-%s.jsonl" % date_str)
+        with open(path, "w", encoding="utf-8") as fh:
+            for i in range(n):
+                ts = "%s-%s-%sT00:%02d:00Z" % (
+                    date_str[:4], date_str[4:6], date_str[6:8], i)
+                rec = statsdb.snapshot_to_sample(make_snapshot(ts=ts))
+                fh.write(json.dumps(rec, separators=(",", ":")) + "\n")
+        return path
+
+    def test_rotate_gzips_and_rolls_up(self):
+        with tempfile.TemporaryDirectory() as d:
+            raw = self._write_raw_day(d, "20260706")
+            now = datetime(2026, 7, 7, 0, 0, 5, tzinfo=timezone.utc)
+            statsdb._rotate(d, "20260706", 365, 2048, now_fn=lambda: now)
+            gz = os.path.join(d, "samples-20260706.jsonl.gz")
+            self.assertTrue(os.path.isfile(gz))
+            self.assertFalse(os.path.isfile(raw))
+            rollup = os.path.join(d, "rollup-20260706.json")
+            self.assertTrue(os.path.isfile(rollup))
+            with open(rollup, "r", encoding="utf-8") as fh:
+                r = json.load(fh)
+            self.assertEqual(r["date"], "20260706")
+            self.assertEqual(r["samples"], 3)
+
+    def test_backfill_rotates_past_only(self):
+        with tempfile.TemporaryDirectory() as d:
+            self._write_raw_day(d, "20260705")
+            today_raw = self._write_raw_day(d, "20260707")
+            now = datetime(2026, 7, 7, 3, 0, 0, tzinfo=timezone.utc)
+            statsdb._backfill(d, 365, 2048, now_fn=lambda: now)
+            # Past day rotated.
+            self.assertTrue(os.path.isfile(
+                os.path.join(d, "samples-20260705.jsonl.gz")))
+            self.assertFalse(os.path.isfile(
+                os.path.join(d, "samples-20260705.jsonl")))
+            # Today untouched.
+            self.assertTrue(os.path.isfile(today_raw))
+            self.assertFalse(os.path.isfile(
+                os.path.join(d, "samples-20260707.jsonl.gz")))
+
+    def test_size_cap_deletes_oldest_first_never_today(self):
+        with tempfile.TemporaryDirectory() as d:
+            # Today's raw file must never be deleted.
+            today_raw = os.path.join(d, "samples-20260707.jsonl")
+            with open(today_raw, "w", encoding="utf-8") as fh:
+                fh.write("x\n")
+            # Fake old .gz files, 1 MiB each, three days.
+            payload = b"0" * (1024 * 1024)
+            for date_str in ("20260701", "20260702", "20260703"):
+                gz = os.path.join(d, "samples-%s.jsonl.gz" % date_str)
+                with gzip.open(gz, "wb") as fh:
+                    fh.write(payload)
+                # gzip compresses zeros heavily; pad the on-disk file so the
+                # size cap actually triggers. Re-open as raw and append filler.
+                with open(gz, "ab") as fh:
+                    fh.write(b"\x00" * (1024 * 1024))
+            now = datetime(2026, 7, 7, 5, 0, 0, tzinfo=timezone.utc)
+            # Cap of 2 MB with ~3 MB present -> delete oldest .gz first.
+            statsdb._enforce_size_cap(d, 2, now_fn=lambda: now)
+            self.assertFalse(os.path.isfile(
+                os.path.join(d, "samples-20260701.jsonl.gz")),
+                "oldest gz should be deleted first")
+            # today's raw untouched
+            self.assertTrue(os.path.isfile(today_raw))
+
+    def test_size_cap_never_deletes_today_even_if_over(self):
+        with tempfile.TemporaryDirectory() as d:
+            today_raw = os.path.join(d, "samples-20260707.jsonl")
+            with open(today_raw, "wb") as fh:
+                fh.write(b"0" * (3 * 1024 * 1024))
+            now = datetime(2026, 7, 7, 5, 0, 0, tzinfo=timezone.utc)
+            statsdb._enforce_size_cap(d, 1, now_fn=lambda: now)
+            self.assertTrue(os.path.isfile(today_raw))
+
+    def test_retention_deletes_old_gz(self):
+        with tempfile.TemporaryDirectory() as d:
+            old = os.path.join(d, "samples-20250101.jsonl.gz")
+            recent = os.path.join(d, "samples-20260706.jsonl.gz")
+            for p in (old, recent):
+                with gzip.open(p, "wb") as fh:
+                    fh.write(b"{}\n")
+            now = datetime(2026, 7, 7, 0, 0, 0, tzinfo=timezone.utc)
+            statsdb._enforce_retention(d, 30, now_fn=lambda: now)
+            self.assertFalse(os.path.isfile(old))
+            self.assertTrue(os.path.isfile(recent))
+
+
+class TestStatus(unittest.TestCase):
+    def test_status_before_start(self):
+        # status() must not blow up even if data_dir is None. We can't rely on
+        # module state here (other tests may have started a sampler), so we
+        # only assert the shape/keys.
+        st = statsdb.status()
+        for key in ("data_dir", "writable", "fallback", "files", "total_mb",
+                    "sampling", "interval"):
+            self.assertIn(key, st)
+
+
+@unittest.skipIf(os.name == "nt", "read-only dir simulation is POSIX-only")
+class TestFallback(unittest.TestCase):
+    def test_unwritable_dir_falls_back(self):
+        with tempfile.TemporaryDirectory() as d:
+            # Use a file-as-dir: a regular file cannot host children.
+            bogus = os.path.join(d, "not-a-dir")
+            with open(bogus, "w", encoding="utf-8") as fh:
+                fh.write("i am a file")
+            resolved, fallback = statsdb._resolve_data_dir(bogus)
+            self.assertTrue(fallback)
+            self.assertNotEqual(resolved, bogus)
+
+
+if __name__ == "__main__":
+    unittest.main()