spark-smi 3.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- spark_smi-3.6.0/LICENSE +21 -0
- spark_smi-3.6.0/PKG-INFO +189 -0
- spark_smi-3.6.0/README.md +146 -0
- spark_smi-3.6.0/pyproject.toml +38 -0
- spark_smi-3.6.0/setup.cfg +4 -0
- spark_smi-3.6.0/spark_smi/__init__.py +1 -0
- spark_smi-3.6.0/spark_smi/__main__.py +16 -0
- spark_smi-3.6.0/spark_smi/_core.py +434 -0
- spark_smi-3.6.0/spark_smi.egg-info/PKG-INFO +189 -0
- spark_smi-3.6.0/spark_smi.egg-info/SOURCES.txt +12 -0
- spark_smi-3.6.0/spark_smi.egg-info/dependency_links.txt +1 -0
- spark_smi-3.6.0/spark_smi.egg-info/entry_points.txt +2 -0
- spark_smi-3.6.0/spark_smi.egg-info/requires.txt +2 -0
- spark_smi-3.6.0/spark_smi.egg-info/top_level.txt +1 -0
spark_smi-3.6.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 [Your Name or EvilTCell]
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
spark_smi-3.6.0/PKG-INFO
ADDED
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: spark-smi
|
|
3
|
+
Version: 3.6.0
|
|
4
|
+
Summary: A terminal-based system monitor (TUI) for NVIDIA Grace Blackwell (GB10) and hybrid CPU architectures
|
|
5
|
+
Author-email: chappa-ai-llc <admin@chappa.ai>
|
|
6
|
+
License: MIT License
|
|
7
|
+
|
|
8
|
+
Copyright (c) 2025 [Your Name or EvilTCell]
|
|
9
|
+
|
|
10
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
11
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
12
|
+
in the Software without restriction, including without limitation the rights
|
|
13
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
14
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
15
|
+
furnished to do so, subject to the following conditions:
|
|
16
|
+
|
|
17
|
+
The above copyright notice and this permission notice shall be included in all
|
|
18
|
+
copies or substantial portions of the Software.
|
|
19
|
+
|
|
20
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
21
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
22
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
23
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
24
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
25
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
26
|
+
SOFTWARE.
|
|
27
|
+
|
|
28
|
+
Project-URL: Homepage, https://github.com/chappa-ai-llc/spark-smi
|
|
29
|
+
Project-URL: Repository, https://github.com/chappa-ai-llc/spark-smi
|
|
30
|
+
Project-URL: Issues, https://github.com/chappa-ai-llc/spark-smi/issues
|
|
31
|
+
Keywords: nvidia,dgx,spark,gb10,grace-blackwell,monitoring,tui,gpu
|
|
32
|
+
Classifier: Programming Language :: Python :: 3
|
|
33
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
34
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
35
|
+
Classifier: Environment :: Console :: Curses
|
|
36
|
+
Classifier: Topic :: System :: Monitoring
|
|
37
|
+
Requires-Python: >=3.6
|
|
38
|
+
Description-Content-Type: text/markdown
|
|
39
|
+
License-File: LICENSE
|
|
40
|
+
Requires-Dist: psutil
|
|
41
|
+
Requires-Dist: nvidia-ml-py
|
|
42
|
+
Dynamic: license-file
|
|
43
|
+
|
|
44
|
+
# SPARK-SMI
|
|
45
|
+
|
|
46
|
+
> A specialized terminal-based system monitor (TUI) built for **NVIDIA Grace Blackwell (GB10)** and hybrid ARM architectures — because `nvidia-smi` alone doesn't tell the full story.
|
|
47
|
+
|
|
48
|
+

|
|
49
|
+

|
|
50
|
+

|
|
51
|
+

|
|
52
|
+

|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## Demo
|
|
57
|
+
|
|
58
|
+

|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## Why SPARK-SMI?
|
|
63
|
+
|
|
64
|
+
The NVIDIA DGX Spark (GB10) is a unique system — a Grace Blackwell chip with unified CPU+GPU memory, hybrid Cortex-X925/A725 core clusters, and high-speed MT2910 200G networking. Standard tools like `nvidia-smi`, `htop`, and `nvtop` were not built with this topology in mind. SPARK-SMI was.
|
|
65
|
+
|
|
66
|
+
| What it handles correctly | Standard tools |
|
|
67
|
+
|:---|:---:|
|
|
68
|
+
| Hybrid P-core / E-core CPU clusters | ❌ |
|
|
69
|
+
| GB10 Unified Memory (CPU+GPU shared) | ❌ |
|
|
70
|
+
| MT2910 200G NIC bandwidth monitoring | ❌ |
|
|
71
|
+
| Mixed GPU architectures in one system | ❌ |
|
|
72
|
+
| NVML with graceful CLI fallback | ✅ |
|
|
73
|
+
| Zero system dependencies | ✅ |
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## Features
|
|
78
|
+
|
|
79
|
+
- **Snapshot Mode** — Runs once and prints to stdout with full ANSI colors, just like `nvidia-smi`. Pipe it, log it, script it.
|
|
80
|
+
- **Live Mode (`-l`)** — Flicker-free TUI that refreshes every second with responsive terminal resize handling.
|
|
81
|
+
- **Hybrid CPU Topology** — Correctly splits and labels Cortex-X925 (Performance) and Cortex-A725 (Efficiency) core clusters with individual per-core load bars.
|
|
82
|
+
- **Unified Memory Aware** — Detects GB10 unified memory architecture and maps system RAM to GPU memory display accurately.
|
|
83
|
+
- **Dual GPU Support** — Handles mixed architectures (e.g. sm_121 GB10 + sm_86 RTX 3090 via OcuLink) simultaneously.
|
|
84
|
+
- **NIC Monitoring** — Real-time bandwidth utilization across all interfaces: MT2910 200G ports (1–4) and Realtek 10G (port 5), read directly from sysfs.
|
|
85
|
+
- **Driver & CUDA Info** — Footer displays live Driver version and CUDA version via NVML or nvidia-smi fallback.
|
|
86
|
+
- **Robust Fallbacks** — NVML → nvidia-smi CLI → graceful degradation. Adapts to missing sensors, fan controllers, and unsupported queries without crashing.
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## Screenshots
|
|
91
|
+
|
|
92
|
+
| Full Dashboard | Resize-Safe |
|
|
93
|
+
|:---:|:---:|
|
|
94
|
+
|  | Scales cleanly from narrow to full-width |
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## Prerequisites
|
|
99
|
+
|
|
100
|
+
- Linux (aarch64 recommended — built and tested on DGX Spark)
|
|
101
|
+
- Python 3.6+
|
|
102
|
+
- NVIDIA Drivers installed
|
|
103
|
+
- `nvidia-smi` in PATH
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Installation
|
|
108
|
+
|
|
109
|
+
### Option 1: Quick Run (Virtual Environment)
|
|
110
|
+
The safest method on DGX appliances — no system libraries touched.
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
git clone https://github.com/chappa-ai-llc/spark-smi.git
|
|
114
|
+
cd spark-smi
|
|
115
|
+
python3 -m venv venv
|
|
116
|
+
./venv/bin/pip install -r requirements.txt
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
# Snapshot (single output)
|
|
121
|
+
./venv/bin/python3 spark-smi.py
|
|
122
|
+
|
|
123
|
+
# Live mode
|
|
124
|
+
./venv/bin/python3 spark-smi.py -l
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
### Option 2: System Alias (Recommended)
|
|
128
|
+
Type `spark-smi` from anywhere.
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
echo "alias spark-smi='~/spark-smi/venv/bin/python3 ~/spark-smi/spark-smi.py'" >> ~/.bashrc
|
|
132
|
+
source ~/.bashrc
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## Usage
|
|
138
|
+
|
|
139
|
+
| Command | Action |
|
|
140
|
+
|:---|:---|
|
|
141
|
+
| `spark-smi` | Snapshot — print once and exit |
|
|
142
|
+
| `spark-smi -l` | Live mode — interactive TUI |
|
|
143
|
+
| `spark-smi -n 0.5 -l` | Live mode at 0.5s refresh rate |
|
|
144
|
+
|
|
145
|
+
**Interactive Controls**
|
|
146
|
+
|
|
147
|
+
| Key | Action |
|
|
148
|
+
|:---:|:---|
|
|
149
|
+
| `q` | Quit |
|
|
150
|
+
| `t` | Toggle temperature units (°C / °F) |
|
|
151
|
+
| `u` | Toggle memory units (GiB / GB) |
|
|
152
|
+
|
|
153
|
+
---
|
|
154
|
+
|
|
155
|
+
## Tested Hardware
|
|
156
|
+
|
|
157
|
+
| Component | Details |
|
|
158
|
+
|:---|:---|
|
|
159
|
+
| System | NVIDIA DGX Spark |
|
|
160
|
+
| SoC | GB10 Grace Blackwell (sm_121) |
|
|
161
|
+
| External GPU | RTX 3090 via M.2 OcuLink (sm_86) — mixed architecture, single dashboard |
|
|
162
|
+
| NICs | MT2910 × 4 (200G, 100G & 40G DAC), Realtek × 1 (10G, 5G, 2.5G, 1G) |
|
|
163
|
+
| OS | Linux 6.17.0-nvidia |
|
|
164
|
+
| Driver | 580.126.09 |
|
|
165
|
+
| CUDA | 13.0 |
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
|
|
169
|
+
## Roadmap
|
|
170
|
+
|
|
171
|
+
- [ ] **Fan Monitoring** — Read GB10 chassis fan speeds without `sudo` (currently blocked by `nvsm`/IPMI privilege requirements)
|
|
172
|
+
- [ ] **REST API / Prometheus Exporter** — Expose a lightweight JSON HTTP endpoint for Grafana and Prometheus integration
|
|
173
|
+
- [ ] **CSV Logging Mode** — `--csv` flag to pipe raw metrics to stdout or file for external processing
|
|
174
|
+
- [ ] **PyPI Package** — `pip install spark-smi` one-liner install
|
|
175
|
+
- [ ] **Multi-node Support** — Monitor clustered DGX Spark nodes from a single dashboard
|
|
176
|
+
|
|
177
|
+
---
|
|
178
|
+
|
|
179
|
+
## About
|
|
180
|
+
|
|
181
|
+
Built by [chappa-ai-llc](https://github.com/chappa-ai-llc) — a solo homelab project born out of frustration with existing tools on novel hardware.
|
|
182
|
+
|
|
183
|
+
If this saved you time, a ⭐ on the repo is appreciated.
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
## License
|
|
188
|
+
|
|
189
|
+
MIT — see [LICENSE](LICENSE) for details.
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# SPARK-SMI
|
|
2
|
+
|
|
3
|
+
> A specialized terminal-based system monitor (TUI) built for **NVIDIA Grace Blackwell (GB10)** and hybrid ARM architectures — because `nvidia-smi` alone doesn't tell the full story.
|
|
4
|
+
|
|
5
|
+

|
|
6
|
+

|
|
7
|
+

|
|
8
|
+

|
|
9
|
+

|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## Demo
|
|
14
|
+
|
|
15
|
+

|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Why SPARK-SMI?
|
|
20
|
+
|
|
21
|
+
The NVIDIA DGX Spark (GB10) is a unique system — a Grace Blackwell chip with unified CPU+GPU memory, hybrid Cortex-X925/A725 core clusters, and high-speed MT2910 200G networking. Standard tools like `nvidia-smi`, `htop`, and `nvtop` were not built with this topology in mind. SPARK-SMI was.
|
|
22
|
+
|
|
23
|
+
| What it handles correctly | Standard tools |
|
|
24
|
+
|:---|:---:|
|
|
25
|
+
| Hybrid P-core / E-core CPU clusters | ❌ |
|
|
26
|
+
| GB10 Unified Memory (CPU+GPU shared) | ❌ |
|
|
27
|
+
| MT2910 200G NIC bandwidth monitoring | ❌ |
|
|
28
|
+
| Mixed GPU architectures in one system | ❌ |
|
|
29
|
+
| NVML with graceful CLI fallback | ✅ |
|
|
30
|
+
| Zero system dependencies | ✅ |
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## Features
|
|
35
|
+
|
|
36
|
+
- **Snapshot Mode** — Runs once and prints to stdout with full ANSI colors, just like `nvidia-smi`. Pipe it, log it, script it.
|
|
37
|
+
- **Live Mode (`-l`)** — Flicker-free TUI that refreshes every second with responsive terminal resize handling.
|
|
38
|
+
- **Hybrid CPU Topology** — Correctly splits and labels Cortex-X925 (Performance) and Cortex-A725 (Efficiency) core clusters with individual per-core load bars.
|
|
39
|
+
- **Unified Memory Aware** — Detects GB10 unified memory architecture and maps system RAM to GPU memory display accurately.
|
|
40
|
+
- **Dual GPU Support** — Handles mixed architectures (e.g. sm_121 GB10 + sm_86 RTX 3090 via OcuLink) simultaneously.
|
|
41
|
+
- **NIC Monitoring** — Real-time bandwidth utilization across all interfaces: MT2910 200G ports (1–4) and Realtek 10G (port 5), read directly from sysfs.
|
|
42
|
+
- **Driver & CUDA Info** — Footer displays live Driver version and CUDA version via NVML or nvidia-smi fallback.
|
|
43
|
+
- **Robust Fallbacks** — NVML → nvidia-smi CLI → graceful degradation. Adapts to missing sensors, fan controllers, and unsupported queries without crashing.
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## Screenshots
|
|
48
|
+
|
|
49
|
+
| Full Dashboard | Resize-Safe |
|
|
50
|
+
|:---:|:---:|
|
|
51
|
+
|  | Scales cleanly from narrow to full-width |
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## Prerequisites
|
|
56
|
+
|
|
57
|
+
- Linux (aarch64 recommended — built and tested on DGX Spark)
|
|
58
|
+
- Python 3.6+
|
|
59
|
+
- NVIDIA Drivers installed
|
|
60
|
+
- `nvidia-smi` in PATH
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## Installation
|
|
65
|
+
|
|
66
|
+
### Option 1: Quick Run (Virtual Environment)
|
|
67
|
+
The safest method on DGX appliances — no system libraries touched.
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
git clone https://github.com/chappa-ai-llc/spark-smi.git
|
|
71
|
+
cd spark-smi
|
|
72
|
+
python3 -m venv venv
|
|
73
|
+
./venv/bin/pip install -r requirements.txt
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
# Snapshot (single output)
|
|
78
|
+
./venv/bin/python3 spark-smi.py
|
|
79
|
+
|
|
80
|
+
# Live mode
|
|
81
|
+
./venv/bin/python3 spark-smi.py -l
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### Option 2: System Alias (Recommended)
|
|
85
|
+
Type `spark-smi` from anywhere.
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
echo "alias spark-smi='~/spark-smi/venv/bin/python3 ~/spark-smi/spark-smi.py'" >> ~/.bashrc
|
|
89
|
+
source ~/.bashrc
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
---
|
|
93
|
+
|
|
94
|
+
## Usage
|
|
95
|
+
|
|
96
|
+
| Command | Action |
|
|
97
|
+
|:---|:---|
|
|
98
|
+
| `spark-smi` | Snapshot — print once and exit |
|
|
99
|
+
| `spark-smi -l` | Live mode — interactive TUI |
|
|
100
|
+
| `spark-smi -n 0.5 -l` | Live mode at 0.5s refresh rate |
|
|
101
|
+
|
|
102
|
+
**Interactive Controls**
|
|
103
|
+
|
|
104
|
+
| Key | Action |
|
|
105
|
+
|:---:|:---|
|
|
106
|
+
| `q` | Quit |
|
|
107
|
+
| `t` | Toggle temperature units (°C / °F) |
|
|
108
|
+
| `u` | Toggle memory units (GiB / GB) |
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
## Tested Hardware
|
|
113
|
+
|
|
114
|
+
| Component | Details |
|
|
115
|
+
|:---|:---|
|
|
116
|
+
| System | NVIDIA DGX Spark |
|
|
117
|
+
| SoC | GB10 Grace Blackwell (sm_121) |
|
|
118
|
+
| External GPU | RTX 3090 via M.2 OcuLink (sm_86) — mixed architecture, single dashboard |
|
|
119
|
+
| NICs | MT2910 × 4 (200G, 100G & 40G DAC), Realtek × 1 (10G, 5G, 2.5G, 1G) |
|
|
120
|
+
| OS | Linux 6.17.0-nvidia |
|
|
121
|
+
| Driver | 580.126.09 |
|
|
122
|
+
| CUDA | 13.0 |
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## Roadmap
|
|
127
|
+
|
|
128
|
+
- [ ] **Fan Monitoring** — Read GB10 chassis fan speeds without `sudo` (currently blocked by `nvsm`/IPMI privilege requirements)
|
|
129
|
+
- [ ] **REST API / Prometheus Exporter** — Expose a lightweight JSON HTTP endpoint for Grafana and Prometheus integration
|
|
130
|
+
- [ ] **CSV Logging Mode** — `--csv` flag to pipe raw metrics to stdout or file for external processing
|
|
131
|
+
- [ ] **PyPI Package** — `pip install spark-smi` one-liner install
|
|
132
|
+
- [ ] **Multi-node Support** — Monitor clustered DGX Spark nodes from a single dashboard
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
136
|
+
## About
|
|
137
|
+
|
|
138
|
+
Built by [chappa-ai-llc](https://github.com/chappa-ai-llc) — a solo homelab project born out of frustration with existing tools on novel hardware.
|
|
139
|
+
|
|
140
|
+
If this saved you time, a ⭐ on the repo is appreciated.
|
|
141
|
+
|
|
142
|
+
---
|
|
143
|
+
|
|
144
|
+
## License
|
|
145
|
+
|
|
146
|
+
MIT — see [LICENSE](LICENSE) for details.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=61.0", "wheel"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "spark-smi"
|
|
7
|
+
version = "3.6.0"
|
|
8
|
+
description = "A terminal-based system monitor (TUI) for NVIDIA Grace Blackwell (GB10) and hybrid CPU architectures"
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
license = { file = "LICENSE" }
|
|
11
|
+
requires-python = ">=3.6"
|
|
12
|
+
authors = [
|
|
13
|
+
{ name = "chappa-ai-llc", email = "admin@chappa.ai" }
|
|
14
|
+
]
|
|
15
|
+
keywords = ["nvidia", "dgx", "spark", "gb10", "grace-blackwell", "monitoring", "tui", "gpu"]
|
|
16
|
+
classifiers = [
|
|
17
|
+
"Programming Language :: Python :: 3",
|
|
18
|
+
"License :: OSI Approved :: MIT License",
|
|
19
|
+
"Operating System :: POSIX :: Linux",
|
|
20
|
+
"Environment :: Console :: Curses",
|
|
21
|
+
"Topic :: System :: Monitoring",
|
|
22
|
+
]
|
|
23
|
+
dependencies = [
|
|
24
|
+
"psutil",
|
|
25
|
+
"nvidia-ml-py",
|
|
26
|
+
]
|
|
27
|
+
|
|
28
|
+
[project.urls]
|
|
29
|
+
Homepage = "https://github.com/chappa-ai-llc/spark-smi"
|
|
30
|
+
Repository = "https://github.com/chappa-ai-llc/spark-smi"
|
|
31
|
+
Issues = "https://github.com/chappa-ai-llc/spark-smi/issues"
|
|
32
|
+
|
|
33
|
+
[project.scripts]
|
|
34
|
+
spark-smi = "spark_smi.__main__:main"
|
|
35
|
+
|
|
36
|
+
[tool.setuptools.packages.find]
|
|
37
|
+
where = ["."]
|
|
38
|
+
include = ["spark_smi*"]
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__ = "3.6.0"
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Entry point for spark-smi when installed via pip.
|
|
3
|
+
The actual monitoring code lives in spark_smi/_core.py.
|
|
4
|
+
"""
|
|
5
|
+
import os
|
|
6
|
+
import runpy
|
|
7
|
+
import sys
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
def main():
|
|
11
|
+
core = os.path.join(os.path.dirname(__file__), "_core.py")
|
|
12
|
+
runpy.run_path(core, run_name="__main__")
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
if __name__ == "__main__":
|
|
16
|
+
main()
|
|
@@ -0,0 +1,434 @@
|
|
|
1
|
+
import time
|
|
2
|
+
import subprocess
|
|
3
|
+
import psutil
|
|
4
|
+
import argparse
|
|
5
|
+
import shutil
|
|
6
|
+
import re
|
|
7
|
+
import platform
|
|
8
|
+
import os
|
|
9
|
+
import sys
|
|
10
|
+
from datetime import datetime
|
|
11
|
+
|
|
12
|
+
# --- NVML Import Strategy ---
|
|
13
|
+
# Force-check for pynvml, but don't crash if the venv isn't active
|
|
14
|
+
try:
|
|
15
|
+
import pynvml
|
|
16
|
+
pynvml.nvmlInit()
|
|
17
|
+
HAS_NVML = True
|
|
18
|
+
except Exception:
|
|
19
|
+
HAS_NVML = False
|
|
20
|
+
|
|
21
|
+
# --- Configuration ---
|
|
22
|
+
VERSION = "3.5.7-stable"
|
|
23
|
+
REFRESH_RATE = 1.0
|
|
24
|
+
MAX_WIDTH = 110
|
|
25
|
+
|
|
26
|
+
# Toggles
|
|
27
|
+
USE_FAHRENHEIT = False
|
|
28
|
+
USE_DECIMAL_UNITS = False
|
|
29
|
+
|
|
30
|
+
class NetMonitor:
|
|
31
|
+
def __init__(self):
|
|
32
|
+
try: self.prev_stats = psutil.net_io_counters(pernic=True)
|
|
33
|
+
except: self.prev_stats = {}
|
|
34
|
+
self.prev_time = time.time()
|
|
35
|
+
# Mapping for DGX Spark: 1-4 are Mellanox (MT2910), 5 is Realtek
|
|
36
|
+
self.mapping = ["enp1s0f0np0", "enp1s0f1np1", "enP2p1s0f0np0", "enP2p1s0f1np1", "enP7s7"]
|
|
37
|
+
|
|
38
|
+
def get_interface_speed(self, iface_name):
|
|
39
|
+
"""Reads the negotiated link speed from sysfs."""
|
|
40
|
+
try:
|
|
41
|
+
with open(f"/sys/class/net/{iface_name}/speed", "r") as f:
|
|
42
|
+
speed = int(f.read().strip())
|
|
43
|
+
return speed
|
|
44
|
+
except:
|
|
45
|
+
return 0
|
|
46
|
+
|
|
47
|
+
def poll(self):
|
|
48
|
+
curr_time = time.time()
|
|
49
|
+
dt = max(curr_time - self.prev_time, 0.1)
|
|
50
|
+
try: curr_stats = psutil.net_io_counters(pernic=True)
|
|
51
|
+
except: return []
|
|
52
|
+
|
|
53
|
+
nics = []
|
|
54
|
+
for i, name in enumerate(self.mapping):
|
|
55
|
+
if name in curr_stats:
|
|
56
|
+
s, p = curr_stats[name], self.prev_stats.get(name, curr_stats[name])
|
|
57
|
+
bps = (((s.bytes_recv + s.bytes_sent) - (p.bytes_recv + p.bytes_sent)) * 8) / dt
|
|
58
|
+
|
|
59
|
+
# Get dynamic speed from system
|
|
60
|
+
speed_mbit = self.get_interface_speed(name)
|
|
61
|
+
limit_bps = speed_mbit * 1_000_000
|
|
62
|
+
|
|
63
|
+
# Format Speed String (e.g., 200G, 40G, 10G, 1G)
|
|
64
|
+
if speed_mbit >= 1000:
|
|
65
|
+
speed_display = f"{speed_mbit // 1000}G"
|
|
66
|
+
else:
|
|
67
|
+
speed_display = f"{speed_mbit}M"
|
|
68
|
+
|
|
69
|
+
# Determine Label based on Device Index (1-4: MT2910, 5: Realtek)
|
|
70
|
+
if i < 4:
|
|
71
|
+
label = f"MT2910 {speed_display}"
|
|
72
|
+
else:
|
|
73
|
+
label = f"Realtek {speed_display}"
|
|
74
|
+
|
|
75
|
+
if speed_mbit == 0:
|
|
76
|
+
label = "Link Down"
|
|
77
|
+
|
|
78
|
+
nics.append({"label": label, "usage": (bps / limit_bps) * 100 if limit_bps > 0 else 0})
|
|
79
|
+
else:
|
|
80
|
+
nics.append({"label": "Offline", "usage": 0})
|
|
81
|
+
|
|
82
|
+
self.prev_stats, self.prev_time = curr_stats, curr_time
|
|
83
|
+
return nics
|
|
84
|
+
|
|
85
|
+
monitor = NetMonitor()
|
|
86
|
+
|
|
87
|
+
class VirtualCurses:
|
|
88
|
+
def __init__(self):
|
|
89
|
+
self.update_dims()
|
|
90
|
+
self.grid = [[(" ", None) for _ in range(self.cols)] for _ in range(self.rows)]
|
|
91
|
+
self.colors = {1: "\033[32m", 2: "\033[36m", 3: "\033[37m", 4: "\033[31m", 5: "\033[33m", 0: "\033[0m"}
|
|
92
|
+
def update_dims(self):
|
|
93
|
+
try: self.cols, self.rows = shutil.get_terminal_size()
|
|
94
|
+
except: self.cols, self.rows = 120, 50
|
|
95
|
+
def getmaxyx(self):
|
|
96
|
+
self.update_dims()
|
|
97
|
+
return self.rows, self.cols
|
|
98
|
+
def erase(self):
|
|
99
|
+
self.update_dims()
|
|
100
|
+
self.grid = [[(" ", None) for _ in range(self.cols)] for _ in range(self.rows)]
|
|
101
|
+
def addstr(self, y, x, text, attr=None):
|
|
102
|
+
if 0 <= y < self.rows and 0 <= x < self.cols:
|
|
103
|
+
for i, char in enumerate(str(text)):
|
|
104
|
+
if x + i < self.cols: self.grid[y][x + i] = (char, attr)
|
|
105
|
+
def render(self):
|
|
106
|
+
output = []
|
|
107
|
+
last_row = max((r for r in range(self.rows) if any(c[0] != " " for c in self.grid[r])), default=0)
|
|
108
|
+
for r in range(last_row + 1):
|
|
109
|
+
row_str, current_fmt = "", None
|
|
110
|
+
for c, fmt in self.grid[r]:
|
|
111
|
+
if fmt != current_fmt:
|
|
112
|
+
row_str += self.colors.get(fmt, self.colors[0])
|
|
113
|
+
current_fmt = fmt
|
|
114
|
+
row_str += c
|
|
115
|
+
output.append(row_str + self.colors[0])
|
|
116
|
+
return "\n".join(output)
|
|
117
|
+
|
|
118
|
+
def fmt_temp(celsius_val):
|
|
119
|
+
try:
|
|
120
|
+
c = float(celsius_val)
|
|
121
|
+
return f"{int((c * 9/5) + 32)}F" if USE_FAHRENHEIT else f"{int(c)}C"
|
|
122
|
+
except: return "N/A"
|
|
123
|
+
|
|
124
|
+
def fmt_mem(bytes_val):
|
|
125
|
+
if bytes_val in [None, "N/A"]: return "N/A"
|
|
126
|
+
div = 1000.0 if USE_DECIMAL_UNITS else 1024.0
|
|
127
|
+
s_m, s_g = ("MB", "GB") if USE_DECIMAL_UNITS else ("MiB", "GiB")
|
|
128
|
+
try:
|
|
129
|
+
if bytes_val > (div**3): return f"{bytes_val/(div**3):.1f}{s_g}"
|
|
130
|
+
return f"{int(bytes_val/(div**2))}{s_m}"
|
|
131
|
+
except: return "N/A"
|
|
132
|
+
|
|
133
|
+
def make_bar(percent, width, color_good, color_mid, color_bad):
|
|
134
|
+
if width < 3: return "[]", None
|
|
135
|
+
pct = max(0, min(int(percent), 100))
|
|
136
|
+
inner_w = width - 2
|
|
137
|
+
filled = int((pct / 100.0) * inner_w)
|
|
138
|
+
bar_str = "[" + "|" * filled + " " * (inner_w - filled) + "]"
|
|
139
|
+
c = color_good if pct < 50 else (color_mid if pct < 80 else color_bad)
|
|
140
|
+
return bar_str, c
|
|
141
|
+
|
|
142
|
+
def get_cpu_temp():
|
|
143
|
+
try:
|
|
144
|
+
temps = psutil.sensors_temperatures()
|
|
145
|
+
for k in ['cpu_thermal', 'soc_thermal', 'coretemp', 'thermal_zone0']:
|
|
146
|
+
if k in temps: return temps[k][0].current
|
|
147
|
+
return list(temps.values())[0][0].current
|
|
148
|
+
except: return "N/A"
|
|
149
|
+
|
|
150
|
+
def get_system_fan():
|
|
151
|
+
if shutil.which("sensors"):
|
|
152
|
+
try:
|
|
153
|
+
res = subprocess.run(["sensors"], capture_output=True, text=True, timeout=0.5)
|
|
154
|
+
highest = 0
|
|
155
|
+
for line in res.stdout.splitlines():
|
|
156
|
+
if "RPM" in line:
|
|
157
|
+
m = re.search(r'(\d+)', line)
|
|
158
|
+
if m and int(m.group(1)) > highest: highest = int(m.group(1))
|
|
159
|
+
if highest > 0: return f"{highest} RPM"
|
|
160
|
+
except: pass
|
|
161
|
+
return "Err"
|
|
162
|
+
|
|
163
|
+
# Cache so we don't shell out to nvidia-smi on every render tick
|
|
164
|
+
_CACHED_DRIVER_INFO = None
|
|
165
|
+
|
|
166
|
+
def get_driver_info_safe():
|
|
167
|
+
global _CACHED_DRIVER_INFO
|
|
168
|
+
if _CACHED_DRIVER_INFO is not None:
|
|
169
|
+
return _CACHED_DRIVER_INFO
|
|
170
|
+
driver, cuda = "Unknown", "Unknown"
|
|
171
|
+
if HAS_NVML:
|
|
172
|
+
try:
|
|
173
|
+
pynvml.nvmlInit()
|
|
174
|
+
drv = pynvml.nvmlSystemGetDriverVersion()
|
|
175
|
+
if isinstance(drv, bytes): drv = drv.decode('utf-8')
|
|
176
|
+
driver = drv
|
|
177
|
+
cuda_ver = pynvml.nvmlSystemGetCudaDriverVersion()
|
|
178
|
+
cuda = f"{cuda_ver // 1000}.{(cuda_ver % 1000) // 10}"
|
|
179
|
+
_CACHED_DRIVER_INFO = (driver, cuda)
|
|
180
|
+
return _CACHED_DRIVER_INFO
|
|
181
|
+
except: pass
|
|
182
|
+
try:
|
|
183
|
+
res = subprocess.run(["nvidia-smi"], capture_output=True, text=True, timeout=3)
|
|
184
|
+
if res.returncode == 0:
|
|
185
|
+
d = re.search(r"Driver Version:\s*([\d\.]+)", res.stdout)
|
|
186
|
+
c = re.search(r"CUDA Version:\s*([\d\.]+)", res.stdout)
|
|
187
|
+
if d: driver = d.group(1)
|
|
188
|
+
if c: cuda = c.group(1)
|
|
189
|
+
except: pass
|
|
190
|
+
_CACHED_DRIVER_INFO = (driver, cuda)
|
|
191
|
+
return _CACHED_DRIVER_INFO
|
|
192
|
+
|
|
193
|
+
def get_gpu_data():
|
|
194
|
+
if not shutil.which("nvidia-smi"): return []
|
|
195
|
+
try:
|
|
196
|
+
res_l = subprocess.run(["nvidia-smi", "-L"], capture_output=True, text=True, timeout=1)
|
|
197
|
+
gpu_lines = [line for line in res_l.stdout.strip().split('\n') if line.startswith("GPU")]
|
|
198
|
+
gpus = []
|
|
199
|
+
|
|
200
|
+
for i, line in enumerate(gpu_lines):
|
|
201
|
+
gid = str(i)
|
|
202
|
+
# 1. Initialize with defaults
|
|
203
|
+
gpu = {"id": gid, "name": "Unknown", "temp": "N/A", "util": 0, "mem_used": "N/A", "mem_total": "N/A", "pwr_str": "N/A", "fan": "N/A"}
|
|
204
|
+
|
|
205
|
+
# 2. NVML Logic (The original robust way)
|
|
206
|
+
nvml_success = False
|
|
207
|
+
if HAS_NVML:
|
|
208
|
+
try:
|
|
209
|
+
pynvml.nvmlInit()
|
|
210
|
+
handle = pynvml.nvmlDeviceGetHandleByIndex(i)
|
|
211
|
+
gpu["name"] = pynvml.nvmlDeviceGetName(handle)
|
|
212
|
+
if isinstance(gpu["name"], bytes): gpu["name"] = gpu["name"].decode('utf-8')
|
|
213
|
+
|
|
214
|
+
gpu["temp"] = pynvml.nvmlDeviceGetTemperature(handle, pynvml.NVML_TEMPERATURE_GPU)
|
|
215
|
+
gpu["util"] = pynvml.nvmlDeviceGetUtilizationRates(handle).gpu
|
|
216
|
+
|
|
217
|
+
mem = pynvml.nvmlDeviceGetMemoryInfo(handle)
|
|
218
|
+
gpu["mem_used"], gpu["mem_total"] = mem.used, mem.total
|
|
219
|
+
|
|
220
|
+
try:
|
|
221
|
+
gpu["fan"] = f"{pynvml.nvmlDeviceGetFanSpeed(handle)}%"
|
|
222
|
+
except:
|
|
223
|
+
gpu["fan"] = "None"
|
|
224
|
+
|
|
225
|
+
try:
|
|
226
|
+
gpu["pwr_str"] = f"{int(pynvml.nvmlDeviceGetPowerUsage(handle) / 1000.0)}W"
|
|
227
|
+
except:
|
|
228
|
+
pass
|
|
229
|
+
nvml_success = True
|
|
230
|
+
except:
|
|
231
|
+
nvml_success = False
|
|
232
|
+
|
|
233
|
+
# 3. CLI Fallback (If NVML failed or is missing data)
|
|
234
|
+
if not nvml_success or gpu["pwr_str"] == "N/A" or gpu["fan"] in ["N/A", "0%"]:
|
|
235
|
+
try:
|
|
236
|
+
# Query only what we need to avoid N/A poisoning the CSV
|
|
237
|
+
query = "name,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw,fan.speed"
|
|
238
|
+
cmd = ["nvidia-smi", f"--id={gid}", f"--query-gpu={query}", "--format=csv,noheader,nounits"]
|
|
239
|
+
res = subprocess.run(cmd, capture_output=True, text=True, timeout=1)
|
|
240
|
+
if res.returncode == 0:
|
|
241
|
+
r = [x.strip() for x in res.stdout.split(',')]
|
|
242
|
+
if gpu["name"] == "Unknown": gpu["name"] = r[0]
|
|
243
|
+
if gpu["temp"] == "N/A": gpu["temp"] = r[1]
|
|
244
|
+
gpu["util"] = float(r[2]) if "N/A" not in r[2] else gpu["util"]
|
|
245
|
+
|
|
246
|
+
# Power and Fan specific fallbacks
|
|
247
|
+
if gpu["pwr_str"] == "N/A" and "N/A" not in r[5]:
|
|
248
|
+
gpu["pwr_str"] = f"{int(float(r[5]))}W"
|
|
249
|
+
if (gpu["fan"] in ["N/A", "None"]) and "N/A" not in r[6]:
|
|
250
|
+
gpu["fan"] = f"{r[6]}%"
|
|
251
|
+
except:
|
|
252
|
+
pass
|
|
253
|
+
|
|
254
|
+
# 4. GB10 / Architecture Special Cases
|
|
255
|
+
if "GB10" in gpu['name'] or gpu['mem_used'] in ["N/A", 0]:
|
|
256
|
+
sys_ram = psutil.virtual_memory()
|
|
257
|
+
gpu["mem_used"], gpu["mem_total"] = sys_ram.used, sys_ram.total
|
|
258
|
+
if "Unified" not in gpu['name']: gpu['name'] += " (Unified)"
|
|
259
|
+
if "GB10" in gpu['name']: gpu["fan"] = "None"
|
|
260
|
+
|
|
261
|
+
gpus.append(gpu)
|
|
262
|
+
return gpus
|
|
263
|
+
except:
|
|
264
|
+
return []
|
|
265
|
+
|
|
266
|
+
def draw_line(stdscr, y, x, width, h):
|
|
267
|
+
if y >= h or width < 2: return
|
|
268
|
+
try: stdscr.addstr(y, x, "+" + "-" * (width - 2) + "+")
|
|
269
|
+
except: pass
|
|
270
|
+
|
|
271
|
+
def draw_row_parts(stdscr, y, x, width, parts, graph_data, colors_map, h, gpu_mode=False):
|
|
272
|
+
if y >= h or width < 5: return
|
|
273
|
+
try:
|
|
274
|
+
curr_x = x
|
|
275
|
+
for p in parts:
|
|
276
|
+
stdscr.addstr(y, curr_x, p)
|
|
277
|
+
curr_x += len(p)
|
|
278
|
+
if graph_data:
|
|
279
|
+
pct_val, _ = graph_data
|
|
280
|
+
reserved = 7 if gpu_mode else 8
|
|
281
|
+
space_left = (x + width - 1) - curr_x
|
|
282
|
+
if space_left > 5:
|
|
283
|
+
bar, color = make_bar(pct_val, space_left - reserved, colors_map[1], colors_map[5], colors_map[4])
|
|
284
|
+
stdscr.addstr(y, curr_x, bar, color)
|
|
285
|
+
stdscr.addstr(y, curr_x + len(bar), f" {int(pct_val):>3}% ")
|
|
286
|
+
if x + width - 1 < stdscr.getmaxyx()[1]:
|
|
287
|
+
stdscr.addstr(y, x + width - 1, "|")
|
|
288
|
+
except: pass
|
|
289
|
+
|
|
290
|
+
def render_dashboard(stdscr, colors_map):
|
|
291
|
+
h, w = stdscr.getmaxyx()
|
|
292
|
+
draw_w = min(w, MAX_WIDTH)
|
|
293
|
+
start_x = max(0, (w - draw_w) // 2)
|
|
294
|
+
if h < 10 or draw_w < 40: return
|
|
295
|
+
|
|
296
|
+
now = datetime.now().strftime("%H:%M:%S")
|
|
297
|
+
stdscr.addstr(0, start_x, f"SPARK-SMI {now}", colors_map[3])
|
|
298
|
+
stdscr.addstr(0, start_x+draw_w-10, f"Ref: {REFRESH_RATE}s", colors_map[2])
|
|
299
|
+
draw_line(stdscr, 1, start_x, draw_w, h)
|
|
300
|
+
|
|
301
|
+
y, cpu = 2, psutil.cpu_percent(percpu=True)
|
|
302
|
+
temp_str, ram, swap, fan_disp = fmt_temp(get_cpu_temp()), psutil.virtual_memory(), psutil.swap_memory(), (get_system_fan())[:4]
|
|
303
|
+
|
|
304
|
+
stdscr.addstr(y, start_x, f"| CPU {platform.node():<20} | Load | Temp Fan | {'Power(W)':<9} | {'RAM / Swap':<21} | Util")
|
|
305
|
+
stdscr.addstr(y, start_x + draw_w - 1, "|")
|
|
306
|
+
draw_line(stdscr, y+1, start_x, draw_w, h)
|
|
307
|
+
|
|
308
|
+
p1 = f"| 0 {'Cortex-X925 (00-09)':<20} | {int(sum(cpu[0:10])/10):>3}% | {temp_str:<4} {fan_disp:<4} | {'Shared':<9} | {f'RAM:{fmt_mem(ram.used)}/{fmt_mem(ram.total)}':<21} | "
|
|
309
|
+
draw_row_parts(stdscr, y+2, start_x, draw_w, [p1], (ram.percent, colors_map), colors_map, h)
|
|
310
|
+
draw_line(stdscr, y+3, start_x, draw_w, h)
|
|
311
|
+
|
|
312
|
+
p2 = f"| 1 {'Cortex-A725 (10-19)':<20} | {int(sum(cpu[10:20])/10):>3}% | {temp_str:<4} {fan_disp:<4} | {'Shared':<9} | {f'Swp:{fmt_mem(swap.used)}/{fmt_mem(swap.total)}':<21} | "
|
|
313
|
+
draw_row_parts(stdscr, y+4, start_x, draw_w, [p2], (swap.percent, colors_map), colors_map, h)
|
|
314
|
+
draw_line(stdscr, y+5, start_x, draw_w, h)
|
|
315
|
+
y += 6
|
|
316
|
+
|
|
317
|
+
for name, cores, offset in [("Performance Cluster", cpu[0:10], 0), ("Efficiency Cluster", cpu[10:20], 10)]:
|
|
318
|
+
if y >= h: break
|
|
319
|
+
stdscr.addstr(y, start_x, f"| {name} " + "-" * (draw_w - len(name) - 4) + "|")
|
|
320
|
+
y += 1
|
|
321
|
+
cols, col_w = 4, (draw_w - 4) // 4
|
|
322
|
+
for i, p in enumerate(cores):
|
|
323
|
+
r, c = i // cols, i % cols
|
|
324
|
+
if y + r >= h: break
|
|
325
|
+
cx = start_x + 4 + (c * col_w)
|
|
326
|
+
bar, color = make_bar(p, col_w - 9, colors_map[1], colors_map[5], colors_map[4])
|
|
327
|
+
try:
|
|
328
|
+
if c == 0: stdscr.addstr(y + r, start_x, "|")
|
|
329
|
+
stdscr.addstr(y + r, cx, f"{i+offset:02}", colors_map[2])
|
|
330
|
+
stdscr.addstr(y + r, cx+3, bar, color)
|
|
331
|
+
stdscr.addstr(y + r, cx+3+len(bar)+1, f"{int(p)}%", colors_map[3])
|
|
332
|
+
stdscr.addstr(y + r, start_x + draw_w - 1, "|")
|
|
333
|
+
except: pass
|
|
334
|
+
y += 3
|
|
335
|
+
|
|
336
|
+
gpus = get_gpu_data()
|
|
337
|
+
if y < h:
|
|
338
|
+
stdscr.addstr(y, start_x, f"| GPU {'Name':<26} | Temp Fan | {'Power (W)':<11} | {'Memory-Usage':<20}| GPU-Util")
|
|
339
|
+
stdscr.addstr(y, start_x + draw_w - 1, "|")
|
|
340
|
+
draw_line(stdscr, y+1, start_x, draw_w, h)
|
|
341
|
+
y += 2
|
|
342
|
+
# ... inside render_dashboard, find the 'for gpu in gpus' loop ...
|
|
343
|
+
for gpu in gpus:
|
|
344
|
+
if y >= h: break
|
|
345
|
+
|
|
346
|
+
# --- NEW FAN LOGIC START ---
|
|
347
|
+
# 1. Start with what was collected in get_gpu_data
|
|
348
|
+
fan_val = str(gpu.get('fan', 'N/A'))
|
|
349
|
+
|
|
350
|
+
# 2. Universal Logic: If it's a standard GPU and reporting 0% or N/A,
|
|
351
|
+
# hammer the CLI specifically for the fan to bypass VENV/Driver lag
|
|
352
|
+
if ("GB10" not in gpu['name']) and (fan_val in ["0%", "N/A", "Err", "0"]):
|
|
353
|
+
try:
|
|
354
|
+
f_res = subprocess.run(
|
|
355
|
+
["nvidia-smi", f"--id={gpu['id']}", "--query-gpu=fan.speed", "--format=csv,noheader,nounits"],
|
|
356
|
+
capture_output=True, text=True, timeout=0.5
|
|
357
|
+
)
|
|
358
|
+
if f_res.returncode == 0 and f_res.stdout.strip().isdigit():
|
|
359
|
+
fan_val = f"{f_res.stdout.strip()}%"
|
|
360
|
+
except:
|
|
361
|
+
pass
|
|
362
|
+
|
|
363
|
+
# 3. Final Formatting for the UI
|
|
364
|
+
display_fan = fan_val[:4] if len(fan_val) > 4 else fan_val
|
|
365
|
+
if "GB10" in gpu['name']: display_fan = "None" # Keep GB10 clean
|
|
366
|
+
# --- NEW FAN LOGIC END ---
|
|
367
|
+
|
|
368
|
+
mem_str = f"{fmt_mem(gpu['mem_used'])}/{fmt_mem(gpu['mem_total'])}".ljust(20)
|
|
369
|
+
|
|
370
|
+
# Trailing | is split as a separate part so the bar gains 1 char of width
|
|
371
|
+
# and renders as "| [bar]" with a space between the pipe and bracket
|
|
372
|
+
p_row = f"| {gpu['id']:<4} {gpu['name'][:26]:<26} | {fmt_temp(gpu['temp']):<5} {display_fan:<4} | {gpu['pwr_str']:<11} | {mem_str}"
|
|
373
|
+
|
|
374
|
+
draw_row_parts(stdscr, y, start_x, draw_w, [p_row, "| "], (gpu['util'], colors_map), colors_map, h, gpu_mode=True)
|
|
375
|
+
draw_line(stdscr, y+1, start_x, draw_w, h)
|
|
376
|
+
y += 2
|
|
377
|
+
|
|
378
|
+
nics = monitor.poll()
|
|
379
|
+
if y + 2 < h:
|
|
380
|
+
stdscr.addstr(y, start_x, "|")
|
|
381
|
+
stdscr.addstr(y+1, start_x, "|")
|
|
382
|
+
for i, n in enumerate(nics):
|
|
383
|
+
base_x = start_x + 1 + (i * 21)
|
|
384
|
+
if base_x + 20 < start_x + draw_w:
|
|
385
|
+
stdscr.addstr(y, base_x, f" {i+1} {n['label']:<12} {int(n['usage']):>3}% ")
|
|
386
|
+
bar, col = make_bar(n['usage'], 19, colors_map[1], colors_map[5], colors_map[4])
|
|
387
|
+
stdscr.addstr(y+1, base_x, " ")
|
|
388
|
+
stdscr.addstr(y+1, base_x + 1, bar, col)
|
|
389
|
+
stdscr.addstr(y, start_x + draw_w - 1, "|")
|
|
390
|
+
stdscr.addstr(y+1, start_x + draw_w - 1, "|")
|
|
391
|
+
draw_line(stdscr, y+2, start_x, draw_w, h)
|
|
392
|
+
y += 3
|
|
393
|
+
|
|
394
|
+
if y + 2 < h:
|
|
395
|
+
driver, cuda = get_driver_info_safe()
|
|
396
|
+
l1 = f" Ver: {VERSION} | Kernel: {platform.release()} | Driver: {driver} | CUDA: {cuda}"
|
|
397
|
+
l2 = " Controls: 'q':Quit | 'u':Unit | 't':Temp | Mode: Live"
|
|
398
|
+
stdscr.addstr(y, start_x, "|"); stdscr.addstr(y, start_x+1, l1.ljust(draw_w-2), colors_map[2]); stdscr.addstr(y, start_x+draw_w-1, "|")
|
|
399
|
+
stdscr.addstr(y+1, start_x, "|"); stdscr.addstr(y+1, start_x+1, l2.ljust(draw_w-2), colors_map[3]); stdscr.addstr(y+1, start_x+draw_w-1, "|")
|
|
400
|
+
draw_line(stdscr, y+2, start_x, draw_w, h)
|
|
401
|
+
|
|
402
|
+
def main_loop(stdscr):
|
|
403
|
+
global USE_FAHRENHEIT, USE_DECIMAL_UNITS
|
|
404
|
+
import curses
|
|
405
|
+
curses.start_color(); curses.use_default_colors(); curses.curs_set(0); stdscr.nodelay(True)
|
|
406
|
+
colors = {i: curses.color_pair(i) for i in range(1, 6)}
|
|
407
|
+
for i, cl in enumerate([curses.COLOR_GREEN, curses.COLOR_CYAN, curses.COLOR_WHITE, curses.COLOR_RED, curses.COLOR_YELLOW], 1):
|
|
408
|
+
curses.init_pair(i, cl, -1)
|
|
409
|
+
|
|
410
|
+
while True:
|
|
411
|
+
try:
|
|
412
|
+
stdscr.erase()
|
|
413
|
+
render_dashboard(stdscr, colors)
|
|
414
|
+
stdscr.refresh()
|
|
415
|
+
except Exception: pass
|
|
416
|
+
|
|
417
|
+
start_wait = time.time()
|
|
418
|
+
while time.time() - start_wait < REFRESH_RATE:
|
|
419
|
+
ch = stdscr.getch()
|
|
420
|
+
if ch == ord('q'): return
|
|
421
|
+
if ch == ord('t'): USE_FAHRENHEIT = not USE_FAHRENHEIT; break
|
|
422
|
+
if ch == ord('u'): USE_DECIMAL_UNITS = not USE_DECIMAL_UNITS; break
|
|
423
|
+
if ch == curses.KEY_RESIZE: stdscr.clear(); break
|
|
424
|
+
time.sleep(0.05)
|
|
425
|
+
|
|
426
|
+
if __name__ == "__main__":
|
|
427
|
+
if "-l" in sys.argv or "--loop" in sys.argv:
|
|
428
|
+
import curses
|
|
429
|
+
try: curses.wrapper(main_loop)
|
|
430
|
+
except KeyboardInterrupt: pass
|
|
431
|
+
else:
|
|
432
|
+
v = VirtualCurses()
|
|
433
|
+
render_dashboard(v, {i:i for i in range(1,6)})
|
|
434
|
+
print(v.render())
|
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: spark-smi
|
|
3
|
+
Version: 3.6.0
|
|
4
|
+
Summary: A terminal-based system monitor (TUI) for NVIDIA Grace Blackwell (GB10) and hybrid CPU architectures
|
|
5
|
+
Author-email: chappa-ai-llc <admin@chappa.ai>
|
|
6
|
+
License: MIT License
|
|
7
|
+
|
|
8
|
+
Copyright (c) 2025 [Your Name or EvilTCell]
|
|
9
|
+
|
|
10
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
11
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
12
|
+
in the Software without restriction, including without limitation the rights
|
|
13
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
14
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
15
|
+
furnished to do so, subject to the following conditions:
|
|
16
|
+
|
|
17
|
+
The above copyright notice and this permission notice shall be included in all
|
|
18
|
+
copies or substantial portions of the Software.
|
|
19
|
+
|
|
20
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
21
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
22
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
23
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
24
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
25
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
26
|
+
SOFTWARE.
|
|
27
|
+
|
|
28
|
+
Project-URL: Homepage, https://github.com/chappa-ai-llc/spark-smi
|
|
29
|
+
Project-URL: Repository, https://github.com/chappa-ai-llc/spark-smi
|
|
30
|
+
Project-URL: Issues, https://github.com/chappa-ai-llc/spark-smi/issues
|
|
31
|
+
Keywords: nvidia,dgx,spark,gb10,grace-blackwell,monitoring,tui,gpu
|
|
32
|
+
Classifier: Programming Language :: Python :: 3
|
|
33
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
34
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
35
|
+
Classifier: Environment :: Console :: Curses
|
|
36
|
+
Classifier: Topic :: System :: Monitoring
|
|
37
|
+
Requires-Python: >=3.6
|
|
38
|
+
Description-Content-Type: text/markdown
|
|
39
|
+
License-File: LICENSE
|
|
40
|
+
Requires-Dist: psutil
|
|
41
|
+
Requires-Dist: nvidia-ml-py
|
|
42
|
+
Dynamic: license-file
|
|
43
|
+
|
|
44
|
+
# SPARK-SMI
|
|
45
|
+
|
|
46
|
+
> A specialized terminal-based system monitor (TUI) built for **NVIDIA Grace Blackwell (GB10)** and hybrid ARM architectures — because `nvidia-smi` alone doesn't tell the full story.
|
|
47
|
+
|
|
48
|
+

|
|
49
|
+

|
|
50
|
+

|
|
51
|
+

|
|
52
|
+

|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## Demo
|
|
57
|
+
|
|
58
|
+

|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## Why SPARK-SMI?
|
|
63
|
+
|
|
64
|
+
The NVIDIA DGX Spark (GB10) is a unique system — a Grace Blackwell chip with unified CPU+GPU memory, hybrid Cortex-X925/A725 core clusters, and high-speed MT2910 200G networking. Standard tools like `nvidia-smi`, `htop`, and `nvtop` were not built with this topology in mind. SPARK-SMI was.
|
|
65
|
+
|
|
66
|
+
| What it handles correctly | Standard tools |
|
|
67
|
+
|:---|:---:|
|
|
68
|
+
| Hybrid P-core / E-core CPU clusters | ❌ |
|
|
69
|
+
| GB10 Unified Memory (CPU+GPU shared) | ❌ |
|
|
70
|
+
| MT2910 200G NIC bandwidth monitoring | ❌ |
|
|
71
|
+
| Mixed GPU architectures in one system | ❌ |
|
|
72
|
+
| NVML with graceful CLI fallback | ✅ |
|
|
73
|
+
| Zero system dependencies | ✅ |
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## Features
|
|
78
|
+
|
|
79
|
+
- **Snapshot Mode** — Runs once and prints to stdout with full ANSI colors, just like `nvidia-smi`. Pipe it, log it, script it.
|
|
80
|
+
- **Live Mode (`-l`)** — Flicker-free TUI that refreshes every second with responsive terminal resize handling.
|
|
81
|
+
- **Hybrid CPU Topology** — Correctly splits and labels Cortex-X925 (Performance) and Cortex-A725 (Efficiency) core clusters with individual per-core load bars.
|
|
82
|
+
- **Unified Memory Aware** — Detects GB10 unified memory architecture and maps system RAM to GPU memory display accurately.
|
|
83
|
+
- **Dual GPU Support** — Handles mixed architectures (e.g. sm_121 GB10 + sm_86 RTX 3090 via OcuLink) simultaneously.
|
|
84
|
+
- **NIC Monitoring** — Real-time bandwidth utilization across all interfaces: MT2910 200G ports (1–4) and Realtek 10G (port 5), read directly from sysfs.
|
|
85
|
+
- **Driver & CUDA Info** — Footer displays live Driver version and CUDA version via NVML or nvidia-smi fallback.
|
|
86
|
+
- **Robust Fallbacks** — NVML → nvidia-smi CLI → graceful degradation. Adapts to missing sensors, fan controllers, and unsupported queries without crashing.
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## Screenshots
|
|
91
|
+
|
|
92
|
+
| Full Dashboard | Resize-Safe |
|
|
93
|
+
|:---:|:---:|
|
|
94
|
+
|  | Scales cleanly from narrow to full-width |
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## Prerequisites
|
|
99
|
+
|
|
100
|
+
- Linux (aarch64 recommended — built and tested on DGX Spark)
|
|
101
|
+
- Python 3.6+
|
|
102
|
+
- NVIDIA Drivers installed
|
|
103
|
+
- `nvidia-smi` in PATH
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Installation
|
|
108
|
+
|
|
109
|
+
### Option 1: Quick Run (Virtual Environment)
|
|
110
|
+
The safest method on DGX appliances — no system libraries touched.
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
git clone https://github.com/chappa-ai-llc/spark-smi.git
|
|
114
|
+
cd spark-smi
|
|
115
|
+
python3 -m venv venv
|
|
116
|
+
./venv/bin/pip install -r requirements.txt
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
# Snapshot (single output)
|
|
121
|
+
./venv/bin/python3 spark-smi.py
|
|
122
|
+
|
|
123
|
+
# Live mode
|
|
124
|
+
./venv/bin/python3 spark-smi.py -l
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
### Option 2: System Alias (Recommended)
|
|
128
|
+
Type `spark-smi` from anywhere.
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
echo "alias spark-smi='~/spark-smi/venv/bin/python3 ~/spark-smi/spark-smi.py'" >> ~/.bashrc
|
|
132
|
+
source ~/.bashrc
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## Usage
|
|
138
|
+
|
|
139
|
+
| Command | Action |
|
|
140
|
+
|:---|:---|
|
|
141
|
+
| `spark-smi` | Snapshot — print once and exit |
|
|
142
|
+
| `spark-smi -l` | Live mode — interactive TUI |
|
|
143
|
+
| `spark-smi -n 0.5 -l` | Live mode at 0.5s refresh rate |
|
|
144
|
+
|
|
145
|
+
**Interactive Controls**
|
|
146
|
+
|
|
147
|
+
| Key | Action |
|
|
148
|
+
|:---:|:---|
|
|
149
|
+
| `q` | Quit |
|
|
150
|
+
| `t` | Toggle temperature units (°C / °F) |
|
|
151
|
+
| `u` | Toggle memory units (GiB / GB) |
|
|
152
|
+
|
|
153
|
+
---
|
|
154
|
+
|
|
155
|
+
## Tested Hardware
|
|
156
|
+
|
|
157
|
+
| Component | Details |
|
|
158
|
+
|:---|:---|
|
|
159
|
+
| System | NVIDIA DGX Spark |
|
|
160
|
+
| SoC | GB10 Grace Blackwell (sm_121) |
|
|
161
|
+
| External GPU | RTX 3090 via M.2 OcuLink (sm_86) — mixed architecture, single dashboard |
|
|
162
|
+
| NICs | MT2910 × 4 (200G, 100G & 40G DAC), Realtek × 1 (10G, 5G, 2.5G, 1G) |
|
|
163
|
+
| OS | Linux 6.17.0-nvidia |
|
|
164
|
+
| Driver | 580.126.09 |
|
|
165
|
+
| CUDA | 13.0 |
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
|
|
169
|
+
## Roadmap
|
|
170
|
+
|
|
171
|
+
- [ ] **Fan Monitoring** — Read GB10 chassis fan speeds without `sudo` (currently blocked by `nvsm`/IPMI privilege requirements)
|
|
172
|
+
- [ ] **REST API / Prometheus Exporter** — Expose a lightweight JSON HTTP endpoint for Grafana and Prometheus integration
|
|
173
|
+
- [ ] **CSV Logging Mode** — `--csv` flag to pipe raw metrics to stdout or file for external processing
|
|
174
|
+
- [ ] **PyPI Package** — `pip install spark-smi` one-liner install
|
|
175
|
+
- [ ] **Multi-node Support** — Monitor clustered DGX Spark nodes from a single dashboard
|
|
176
|
+
|
|
177
|
+
---
|
|
178
|
+
|
|
179
|
+
## About
|
|
180
|
+
|
|
181
|
+
Built by [chappa-ai-llc](https://github.com/chappa-ai-llc) — a solo homelab project born out of frustration with existing tools on novel hardware.
|
|
182
|
+
|
|
183
|
+
If this saved you time, a ⭐ on the repo is appreciated.
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
## License
|
|
188
|
+
|
|
189
|
+
MIT — see [LICENSE](LICENSE) for details.
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
spark_smi/__init__.py
|
|
5
|
+
spark_smi/__main__.py
|
|
6
|
+
spark_smi/_core.py
|
|
7
|
+
spark_smi.egg-info/PKG-INFO
|
|
8
|
+
spark_smi.egg-info/SOURCES.txt
|
|
9
|
+
spark_smi.egg-info/dependency_links.txt
|
|
10
|
+
spark_smi.egg-info/entry_points.txt
|
|
11
|
+
spark_smi.egg-info/requires.txt
|
|
12
|
+
spark_smi.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
spark_smi
|