tasc-hpc-daemon 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- tasc_hpc_daemon-0.1.0/MANIFEST.in +10 -0
- tasc_hpc_daemon-0.1.0/PKG-INFO +155 -0
- tasc_hpc_daemon-0.1.0/README.md +146 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/__init__.py +2 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/__main__.py +6 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/api_client/__init__.py +2 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/api_client/auth.py +85 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/api_client/daemon_registry.py +134 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/api_client/http.py +90 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/assets/DISCLAIMER.txt +41 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/core/__init__.py +2 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/core/cli.py +276 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/core/config.py +85 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/core/pty_session.py +248 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/core/ws.py +204 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/db.py +263 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/guardrails.py +440 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/jobs/__init__.py +2 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/jobs/handlers.py +245 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/jobs/interfaces.py +324 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/jobs/models.py +43 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/jobs/monitor.py +197 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/path_utils.py +40 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/setup/__init__.py +3 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/setup/prompts.py +344 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/setup/wizard.py +225 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/skills/__init__.py +68 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/skills/gcp-hpc.md +52 -0
- tasc_hpc_daemon-0.1.0/hpc_daemon/skills/mila-hpc.md +313 -0
- tasc_hpc_daemon-0.1.0/pyproject.toml +26 -0
- tasc_hpc_daemon-0.1.0/setup.cfg +4 -0
- tasc_hpc_daemon-0.1.0/tasc_hpc_daemon.egg-info/PKG-INFO +155 -0
- tasc_hpc_daemon-0.1.0/tasc_hpc_daemon.egg-info/SOURCES.txt +35 -0
- tasc_hpc_daemon-0.1.0/tasc_hpc_daemon.egg-info/dependency_links.txt +1 -0
- tasc_hpc_daemon-0.1.0/tasc_hpc_daemon.egg-info/entry_points.txt +2 -0
- tasc_hpc_daemon-0.1.0/tasc_hpc_daemon.egg-info/requires.txt +1 -0
- tasc_hpc_daemon-0.1.0/tasc_hpc_daemon.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: tasc-hpc-daemon
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: HPC cluster daemon for bridging AI agents to remote compute resources
|
|
5
|
+
Author: Tiptree Advanced Systems Corporation, Miles Qi Li
|
|
6
|
+
Requires-Python: >=3.10
|
|
7
|
+
Description-Content-Type: text/markdown
|
|
8
|
+
Requires-Dist: websockets>=14.0
|
|
9
|
+
|
|
10
|
+
# HPC Daemon
|
|
11
|
+
|
|
12
|
+
A lightweight daemon that runs on HPC clusters or remote servers, bridging AI agents to local PTY shells and SLURM job schedulers over WebSocket.
|
|
13
|
+
|
|
14
|
+
## How It Works
|
|
15
|
+
|
|
16
|
+
AI agents run in isolated cloud sandboxes and cannot directly reach machines behind firewalls. The daemon solves this with a reverse-proxy pattern:
|
|
17
|
+
|
|
18
|
+
1. The daemon runs on your server and opens an **outbound** WebSocket to the Tiptree platform
|
|
19
|
+
2. The agent client also connects **outbound** to the same platform
|
|
20
|
+
3. The platform routes messages between them, keyed on your user identity
|
|
21
|
+
|
|
22
|
+
Because the daemon initiates the connection, no inbound firewall rules are needed.
|
|
23
|
+
|
|
24
|
+
### Execution Modes
|
|
25
|
+
|
|
26
|
+
- **PTY mode** (synchronous) — interactive shell sessions for quick commands
|
|
27
|
+
- **Job mode** (asynchronous) — batch job submission with automatic wake-on-complete callbacks
|
|
28
|
+
that are persisted locally and retried until delivery succeeds
|
|
29
|
+
|
|
30
|
+
The daemon auto-detects SLURM. If `sbatch` is available, jobs go through SLURM; otherwise they run as local background processes.
|
|
31
|
+
|
|
32
|
+
## Installation
|
|
33
|
+
|
|
34
|
+
- Python 3.10+
|
|
35
|
+
- A [Tiptree](https://tiptreesystems.com) account
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
pip install . # or: python3 -m pip install .
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
This installs the `hpc-daemon` command and its only dependency (`websockets`).
|
|
42
|
+
|
|
43
|
+
## Setup
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
hpc-daemon setup \
|
|
47
|
+
--email you@example.com \
|
|
48
|
+
--url https://tiptreesystems.com
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
The setup wizard:
|
|
52
|
+
|
|
53
|
+
1. Authenticates via a one-time code sent to your email
|
|
54
|
+
2. Presents a **disclaimer** about remote code execution risks
|
|
55
|
+
3. Creates an API key for daemon authentication
|
|
56
|
+
4. Prompts for an optional **skill** (built-in cluster-specific guidance, or a custom server description)
|
|
57
|
+
5. Prompts for **directory restrictions** (where the agent is allowed to write; defaults to `$SCRATCH/tiptree-workspace` or `~/tiptree-workspace`)
|
|
58
|
+
6. Prompts for a **job working directory** and optional **server instructions**
|
|
59
|
+
7. Registers the daemon with the platform
|
|
60
|
+
|
|
61
|
+
The daemon ID defaults to the machine's hostname. Override with `--daemon-id`.
|
|
62
|
+
|
|
63
|
+
Re-running setup for the same daemon ID updates the existing profile (no duplicates).
|
|
64
|
+
|
|
65
|
+
### Non-Interactive Setup
|
|
66
|
+
|
|
67
|
+
For automated deployments:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
hpc-daemon setup \
|
|
71
|
+
--email you@example.com \
|
|
72
|
+
--url https://tiptreesystems.com \
|
|
73
|
+
--no-interview \
|
|
74
|
+
--allowed-dirs ~/workspace ~/scratch \
|
|
75
|
+
--skill mila-hpc \
|
|
76
|
+
--cluster-name my-cluster
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Use `--cluster-name` to set a human-readable name for the cluster (defaults to the machine's hostname).
|
|
80
|
+
|
|
81
|
+
> **Warning:** `--no-interview` skips **all** interactive prompts (disclaimer, skill selection, directory restrictions, working directory, server instructions). OTP is still required. Without `--allowed-dirs`, the agent gets unrestricted filesystem write access.
|
|
82
|
+
|
|
83
|
+
## Running
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
# Start in foreground
|
|
87
|
+
hpc-daemon start
|
|
88
|
+
|
|
89
|
+
# Start in background (persists after logout)
|
|
90
|
+
nohup hpc-daemon start 2>&1 &
|
|
91
|
+
|
|
92
|
+
# Check status
|
|
93
|
+
hpc-daemon status
|
|
94
|
+
|
|
95
|
+
# View logs
|
|
96
|
+
tail -f ~/.hpc_daemon/<daemon_id>.log
|
|
97
|
+
|
|
98
|
+
# Stop
|
|
99
|
+
hpc-daemon stop
|
|
100
|
+
|
|
101
|
+
# List registered daemons
|
|
102
|
+
hpc-daemon list
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
If only one daemon profile exists, `--daemon-id` is auto-detected. With multiple profiles, specify it explicitly (e.g., `hpc-daemon start --daemon-id mila-login-1`).
|
|
106
|
+
|
|
107
|
+
To force local mode on a SLURM cluster (jobs run as background processes instead of `sbatch`):
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
LOCAL_MODE=1 hpc-daemon start
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
## State
|
|
114
|
+
|
|
115
|
+
All configuration, job records, and callback delivery state are stored in
|
|
116
|
+
`~/.hpc_daemon/state.db` (SQLite). PID files and logs live in the same directory.
|
|
117
|
+
|
|
118
|
+
## Safety
|
|
119
|
+
|
|
120
|
+
The daemon enforces directory restrictions via bash function wrappers injected into PTY sessions. File-modifying commands (`rm`, `rmdir`, `mv`, `cp`, `mkdir`, `touch`, `tee`), directory navigation (`cd`, `pushd`, `popd`), and output redirections are validated against the allowed directory list configured during setup.
|
|
121
|
+
|
|
122
|
+
Job ownership is also enforced: code assistants can only cancel jobs they submitted.
|
|
123
|
+
|
|
124
|
+
These guardrails are shell-level and not a hard security boundary. They prevent accidental damage, not a determined adversary.
|
|
125
|
+
|
|
126
|
+
> **Shared machines:** Your API key is stored in `~/.hpc_daemon/state.db`. The file and directory are restricted to your user account (`0600`/`0700`), so other users on the same login node cannot read it. However, if multiple people share the same Unix account, they all have access. Do not use the daemon on a shared account.
|
|
127
|
+
|
|
128
|
+
## Project Structure
|
|
129
|
+
|
|
130
|
+
```
|
|
131
|
+
hpc_daemon/
|
|
132
|
+
├── core/
|
|
133
|
+
│ ├── cli.py # CLI (setup, start, stop, status, list)
|
|
134
|
+
│ ├── config.py # Runtime config, SLURM detection
|
|
135
|
+
│ ├── ws.py # WebSocket connection and message routing
|
|
136
|
+
│ └── pty_session.py # PTY shell session management
|
|
137
|
+
├── setup/
|
|
138
|
+
│ ├── wizard.py # Setup wizard and daemon registration
|
|
139
|
+
│ └── prompts.py # Interactive setup prompts
|
|
140
|
+
├── jobs/
|
|
141
|
+
│ ├── handlers.py # Job submit/status handlers
|
|
142
|
+
│ ├── interfaces.py # SlurmInterface, LocalJobInterface
|
|
143
|
+
│ ├── models.py # JobRecord, JobState
|
|
144
|
+
│ └── monitor.py # Background job polling and callbacks
|
|
145
|
+
├── api_client/
|
|
146
|
+
│ ├── auth.py # OTP signin, API key creation
|
|
147
|
+
│ ├── daemon_registry.py # Daemon registration
|
|
148
|
+
│ └── http.py # HTTP client utilities
|
|
149
|
+
├── skills/
|
|
150
|
+
│ ├── __init__.py # Skill discovery and loading
|
|
151
|
+
│ └── mila-hpc.md # Built-in Mila cluster skill
|
|
152
|
+
├── db.py # SQLite database (profiles, jobs)
|
|
153
|
+
├── guardrails.py # Directory restriction enforcement
|
|
154
|
+
└── __main__.py # Entry point
|
|
155
|
+
```
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# HPC Daemon
|
|
2
|
+
|
|
3
|
+
A lightweight daemon that runs on HPC clusters or remote servers, bridging AI agents to local PTY shells and SLURM job schedulers over WebSocket.
|
|
4
|
+
|
|
5
|
+
## How It Works
|
|
6
|
+
|
|
7
|
+
AI agents run in isolated cloud sandboxes and cannot directly reach machines behind firewalls. The daemon solves this with a reverse-proxy pattern:
|
|
8
|
+
|
|
9
|
+
1. The daemon runs on your server and opens an **outbound** WebSocket to the Tiptree platform
|
|
10
|
+
2. The agent client also connects **outbound** to the same platform
|
|
11
|
+
3. The platform routes messages between them, keyed on your user identity
|
|
12
|
+
|
|
13
|
+
Because the daemon initiates the connection, no inbound firewall rules are needed.
|
|
14
|
+
|
|
15
|
+
### Execution Modes
|
|
16
|
+
|
|
17
|
+
- **PTY mode** (synchronous) — interactive shell sessions for quick commands
|
|
18
|
+
- **Job mode** (asynchronous) — batch job submission with automatic wake-on-complete callbacks
|
|
19
|
+
that are persisted locally and retried until delivery succeeds
|
|
20
|
+
|
|
21
|
+
The daemon auto-detects SLURM. If `sbatch` is available, jobs go through SLURM; otherwise they run as local background processes.
|
|
22
|
+
|
|
23
|
+
## Installation
|
|
24
|
+
|
|
25
|
+
- Python 3.10+
|
|
26
|
+
- A [Tiptree](https://tiptreesystems.com) account
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
pip install . # or: python3 -m pip install .
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
This installs the `hpc-daemon` command and its only dependency (`websockets`).
|
|
33
|
+
|
|
34
|
+
## Setup
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
hpc-daemon setup \
|
|
38
|
+
--email you@example.com \
|
|
39
|
+
--url https://tiptreesystems.com
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The setup wizard:
|
|
43
|
+
|
|
44
|
+
1. Authenticates via a one-time code sent to your email
|
|
45
|
+
2. Presents a **disclaimer** about remote code execution risks
|
|
46
|
+
3. Creates an API key for daemon authentication
|
|
47
|
+
4. Prompts for an optional **skill** (built-in cluster-specific guidance, or a custom server description)
|
|
48
|
+
5. Prompts for **directory restrictions** (where the agent is allowed to write; defaults to `$SCRATCH/tiptree-workspace` or `~/tiptree-workspace`)
|
|
49
|
+
6. Prompts for a **job working directory** and optional **server instructions**
|
|
50
|
+
7. Registers the daemon with the platform
|
|
51
|
+
|
|
52
|
+
The daemon ID defaults to the machine's hostname. Override with `--daemon-id`.
|
|
53
|
+
|
|
54
|
+
Re-running setup for the same daemon ID updates the existing profile (no duplicates).
|
|
55
|
+
|
|
56
|
+
### Non-Interactive Setup
|
|
57
|
+
|
|
58
|
+
For automated deployments:
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
hpc-daemon setup \
|
|
62
|
+
--email you@example.com \
|
|
63
|
+
--url https://tiptreesystems.com \
|
|
64
|
+
--no-interview \
|
|
65
|
+
--allowed-dirs ~/workspace ~/scratch \
|
|
66
|
+
--skill mila-hpc \
|
|
67
|
+
--cluster-name my-cluster
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Use `--cluster-name` to set a human-readable name for the cluster (defaults to the machine's hostname).
|
|
71
|
+
|
|
72
|
+
> **Warning:** `--no-interview` skips **all** interactive prompts (disclaimer, skill selection, directory restrictions, working directory, server instructions). OTP is still required. Without `--allowed-dirs`, the agent gets unrestricted filesystem write access.
|
|
73
|
+
|
|
74
|
+
## Running
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
# Start in foreground
|
|
78
|
+
hpc-daemon start
|
|
79
|
+
|
|
80
|
+
# Start in background (persists after logout)
|
|
81
|
+
nohup hpc-daemon start 2>&1 &
|
|
82
|
+
|
|
83
|
+
# Check status
|
|
84
|
+
hpc-daemon status
|
|
85
|
+
|
|
86
|
+
# View logs
|
|
87
|
+
tail -f ~/.hpc_daemon/<daemon_id>.log
|
|
88
|
+
|
|
89
|
+
# Stop
|
|
90
|
+
hpc-daemon stop
|
|
91
|
+
|
|
92
|
+
# List registered daemons
|
|
93
|
+
hpc-daemon list
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
If only one daemon profile exists, `--daemon-id` is auto-detected. With multiple profiles, specify it explicitly (e.g., `hpc-daemon start --daemon-id mila-login-1`).
|
|
97
|
+
|
|
98
|
+
To force local mode on a SLURM cluster (jobs run as background processes instead of `sbatch`):
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
LOCAL_MODE=1 hpc-daemon start
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
## State
|
|
105
|
+
|
|
106
|
+
All configuration, job records, and callback delivery state are stored in
|
|
107
|
+
`~/.hpc_daemon/state.db` (SQLite). PID files and logs live in the same directory.
|
|
108
|
+
|
|
109
|
+
## Safety
|
|
110
|
+
|
|
111
|
+
The daemon enforces directory restrictions via bash function wrappers injected into PTY sessions. File-modifying commands (`rm`, `rmdir`, `mv`, `cp`, `mkdir`, `touch`, `tee`), directory navigation (`cd`, `pushd`, `popd`), and output redirections are validated against the allowed directory list configured during setup.
|
|
112
|
+
|
|
113
|
+
Job ownership is also enforced: code assistants can only cancel jobs they submitted.
|
|
114
|
+
|
|
115
|
+
These guardrails are shell-level and not a hard security boundary. They prevent accidental damage, not a determined adversary.
|
|
116
|
+
|
|
117
|
+
> **Shared machines:** Your API key is stored in `~/.hpc_daemon/state.db`. The file and directory are restricted to your user account (`0600`/`0700`), so other users on the same login node cannot read it. However, if multiple people share the same Unix account, they all have access. Do not use the daemon on a shared account.
|
|
118
|
+
|
|
119
|
+
## Project Structure
|
|
120
|
+
|
|
121
|
+
```
|
|
122
|
+
hpc_daemon/
|
|
123
|
+
├── core/
|
|
124
|
+
│ ├── cli.py # CLI (setup, start, stop, status, list)
|
|
125
|
+
│ ├── config.py # Runtime config, SLURM detection
|
|
126
|
+
│ ├── ws.py # WebSocket connection and message routing
|
|
127
|
+
│ └── pty_session.py # PTY shell session management
|
|
128
|
+
├── setup/
|
|
129
|
+
│ ├── wizard.py # Setup wizard and daemon registration
|
|
130
|
+
│ └── prompts.py # Interactive setup prompts
|
|
131
|
+
├── jobs/
|
|
132
|
+
│ ├── handlers.py # Job submit/status handlers
|
|
133
|
+
│ ├── interfaces.py # SlurmInterface, LocalJobInterface
|
|
134
|
+
│ ├── models.py # JobRecord, JobState
|
|
135
|
+
│ └── monitor.py # Background job polling and callbacks
|
|
136
|
+
├── api_client/
|
|
137
|
+
│ ├── auth.py # OTP signin, API key creation
|
|
138
|
+
│ ├── daemon_registry.py # Daemon registration
|
|
139
|
+
│ └── http.py # HTTP client utilities
|
|
140
|
+
├── skills/
|
|
141
|
+
│ ├── __init__.py # Skill discovery and loading
|
|
142
|
+
│ └── mila-hpc.md # Built-in Mila cluster skill
|
|
143
|
+
├── db.py # SQLite database (profiles, jobs)
|
|
144
|
+
├── guardrails.py # Directory restriction enforcement
|
|
145
|
+
└── __main__.py # Entry point
|
|
146
|
+
```
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
import http.cookiejar
|
|
2
|
+
import json
|
|
3
|
+
import ssl
|
|
4
|
+
import sys
|
|
5
|
+
import urllib.error
|
|
6
|
+
import urllib.parse
|
|
7
|
+
import urllib.request
|
|
8
|
+
|
|
9
|
+
from hpc_daemon.api_client.http import http_request, SSL_BYPASS_DOMAINS
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
def create_api_key(base_url: str, access_token: str) -> dict:
|
|
13
|
+
url = f"{base_url}/auth/api-key"
|
|
14
|
+
headers = {"Authorization": f"Bearer {access_token}"}
|
|
15
|
+
return http_request(url, method="POST", data={}, headers=headers, exit_on_error=True)
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
def request_otp_signin(base_url: str, email: str) -> str:
|
|
19
|
+
url = f"{base_url}/otp/signin"
|
|
20
|
+
print(f"Requesting OTP from {url}...")
|
|
21
|
+
try:
|
|
22
|
+
resp = http_request(
|
|
23
|
+
url, method="POST", data={"email": email}, exit_on_error=False
|
|
24
|
+
)
|
|
25
|
+
return resp["payload"]["token"]
|
|
26
|
+
except urllib.error.HTTPError as e:
|
|
27
|
+
print(f"Failed: {e.code} {e.reason}")
|
|
28
|
+
body = e.read().decode()
|
|
29
|
+
print(body)
|
|
30
|
+
print("OTP signin failed. Is the server running on this URL?")
|
|
31
|
+
sys.exit(1)
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
def verify_otp_signin(base_url: str, token: str, otp: str) -> str:
|
|
35
|
+
"""Verify OTP and extract access token from Set-Cookie header.
|
|
36
|
+
|
|
37
|
+
The frontend returns the access token as an httpOnly cookie
|
|
38
|
+
(not in the JSON body), so we need to capture cookies from the response.
|
|
39
|
+
"""
|
|
40
|
+
url = f"{base_url}/otp/signin/verify"
|
|
41
|
+
data_bytes = json.dumps({"token": token, "otp": otp}).encode("utf-8")
|
|
42
|
+
headers = {
|
|
43
|
+
"Content-Type": "application/json",
|
|
44
|
+
"ngrok-skip-browser-warning": "true",
|
|
45
|
+
}
|
|
46
|
+
req = urllib.request.Request(url, data=data_bytes, headers=headers, method="POST")
|
|
47
|
+
|
|
48
|
+
ctx = ssl.create_default_context()
|
|
49
|
+
hostname = urllib.parse.urlparse(url).hostname or ""
|
|
50
|
+
if hostname in SSL_BYPASS_DOMAINS:
|
|
51
|
+
ctx.check_hostname = False
|
|
52
|
+
ctx.verify_mode = ssl.CERT_NONE
|
|
53
|
+
|
|
54
|
+
cookie_jar = http.cookiejar.CookieJar()
|
|
55
|
+
opener = urllib.request.build_opener(
|
|
56
|
+
urllib.request.HTTPSHandler(context=ctx),
|
|
57
|
+
urllib.request.HTTPCookieProcessor(cookie_jar),
|
|
58
|
+
)
|
|
59
|
+
|
|
60
|
+
try:
|
|
61
|
+
response = opener.open(req, timeout=30)
|
|
62
|
+
except urllib.error.HTTPError as e:
|
|
63
|
+
print(f"OTP verification failed: {e.code} {e.reason}")
|
|
64
|
+
body = e.read().decode()
|
|
65
|
+
print(body)
|
|
66
|
+
sys.exit(1)
|
|
67
|
+
|
|
68
|
+
if response.status < 200 or response.status >= 300:
|
|
69
|
+
print(f"OTP verification failed: HTTP {response.status}")
|
|
70
|
+
sys.exit(1)
|
|
71
|
+
|
|
72
|
+
# Extract access token from cookies (cookie name varies by environment)
|
|
73
|
+
for cookie in cookie_jar:
|
|
74
|
+
if cookie.name in ("tiptree-at", "__Secure-tiptree-at"):
|
|
75
|
+
return cookie.value
|
|
76
|
+
|
|
77
|
+
# Fallback: try JSON body (older servers may still return access_token)
|
|
78
|
+
body = response.read()
|
|
79
|
+
if body:
|
|
80
|
+
resp = json.loads(body)
|
|
81
|
+
if "access_token" in resp:
|
|
82
|
+
return resp["access_token"]
|
|
83
|
+
|
|
84
|
+
print("Error: Could not extract access token from server response.")
|
|
85
|
+
sys.exit(1)
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
import multiprocessing
|
|
2
|
+
import platform
|
|
3
|
+
import socket
|
|
4
|
+
import subprocess
|
|
5
|
+
import sys
|
|
6
|
+
import time
|
|
7
|
+
|
|
8
|
+
from hpc_daemon.api_client.http import http_request
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
def detect_capabilities() -> dict[str, object]:
|
|
12
|
+
capabilities = {}
|
|
13
|
+
|
|
14
|
+
# OS and kernel
|
|
15
|
+
capabilities["os"] = f"{platform.system()} {platform.release()}"
|
|
16
|
+
|
|
17
|
+
# Distro (Linux only)
|
|
18
|
+
try:
|
|
19
|
+
import distro
|
|
20
|
+
name = distro.name(pretty=True)
|
|
21
|
+
if name:
|
|
22
|
+
capabilities["distro"] = name
|
|
23
|
+
except ImportError:
|
|
24
|
+
try:
|
|
25
|
+
info = platform.freedesktop_os_release()
|
|
26
|
+
pretty = info.get("PRETTY_NAME", "")
|
|
27
|
+
if pretty:
|
|
28
|
+
capabilities["distro"] = pretty
|
|
29
|
+
except (OSError, AttributeError):
|
|
30
|
+
pass
|
|
31
|
+
|
|
32
|
+
# Architecture
|
|
33
|
+
capabilities["arch"] = platform.machine()
|
|
34
|
+
|
|
35
|
+
# Total RAM
|
|
36
|
+
try:
|
|
37
|
+
with open("/proc/meminfo") as f:
|
|
38
|
+
for line in f:
|
|
39
|
+
if line.startswith("MemTotal:"):
|
|
40
|
+
kb = int(line.split()[1])
|
|
41
|
+
capabilities["ram_gb"] = round(kb / (1024 * 1024), 1)
|
|
42
|
+
break
|
|
43
|
+
except OSError:
|
|
44
|
+
pass
|
|
45
|
+
|
|
46
|
+
# SLURM
|
|
47
|
+
try:
|
|
48
|
+
result = subprocess.run(["sinfo", "--version"], capture_output=True, timeout=5)
|
|
49
|
+
capabilities["slurm_available"] = result.returncode == 0
|
|
50
|
+
if result.returncode == 0:
|
|
51
|
+
capabilities["slurm_version"] = result.stdout.decode().strip()
|
|
52
|
+
except (FileNotFoundError, subprocess.TimeoutExpired):
|
|
53
|
+
capabilities["slurm_available"] = False
|
|
54
|
+
|
|
55
|
+
# GPUs
|
|
56
|
+
try:
|
|
57
|
+
result = subprocess.run(
|
|
58
|
+
["nvidia-smi", "--query-gpu=name", "--format=csv,noheader"],
|
|
59
|
+
capture_output=True,
|
|
60
|
+
timeout=5,
|
|
61
|
+
)
|
|
62
|
+
if result.returncode == 0:
|
|
63
|
+
gpus = result.stdout.decode().strip().split("\n")
|
|
64
|
+
capabilities["gpu_count"] = len([g for g in gpus if g.strip()])
|
|
65
|
+
capabilities["gpu_names"] = gpus
|
|
66
|
+
except (FileNotFoundError, subprocess.TimeoutExpired):
|
|
67
|
+
capabilities["gpu_count"] = 0
|
|
68
|
+
|
|
69
|
+
# CPUs
|
|
70
|
+
try:
|
|
71
|
+
capabilities["cpu_count"] = multiprocessing.cpu_count()
|
|
72
|
+
except NotImplementedError:
|
|
73
|
+
pass
|
|
74
|
+
|
|
75
|
+
return capabilities
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
def format_skill_remote(base_url: str, access_token: str, raw_notes: str, daemon_id: str) -> dict:
|
|
79
|
+
"""Send notes to hpc-server for formatting/validation. Returns dict with formatted_skill and was_reformatted."""
|
|
80
|
+
url = f"{base_url}/hpc/skill/format"
|
|
81
|
+
headers = {"Authorization": f"Bearer {access_token}"}
|
|
82
|
+
return http_request(
|
|
83
|
+
url, method="POST",
|
|
84
|
+
data={"raw_notes": raw_notes, "daemon_id": daemon_id},
|
|
85
|
+
headers=headers,
|
|
86
|
+
exit_on_error=False,
|
|
87
|
+
timeout=120,
|
|
88
|
+
)
|
|
89
|
+
|
|
90
|
+
|
|
91
|
+
def register_daemon(
|
|
92
|
+
base_url: str,
|
|
93
|
+
access_token: str,
|
|
94
|
+
daemon_id: str,
|
|
95
|
+
cluster_name: str | None = None,
|
|
96
|
+
server_instructions: str | None = None,
|
|
97
|
+
hpc_skill: str | None = None,
|
|
98
|
+
custom_skills: dict[str, dict[str, str]] | None = None,
|
|
99
|
+
) -> dict:
|
|
100
|
+
"""
|
|
101
|
+
Register the daemon with hpc-server.
|
|
102
|
+
|
|
103
|
+
Args:
|
|
104
|
+
server_instructions: Optional human-readable instructions for the agent (e.g., "module load cuda").
|
|
105
|
+
IMPORTANT: This field MUST NOT contain "magic links", connection strings,
|
|
106
|
+
access tokens, or any other sensitive secrets.
|
|
107
|
+
hpc_skill: Optional markdown content (with YAML frontmatter) providing cluster-specific
|
|
108
|
+
skill guidance for AI agents.
|
|
109
|
+
custom_skills: Optional dict of user-provided markdown skills.
|
|
110
|
+
Shape: {"skill-name": {"SKILL.md": "content", ...}}
|
|
111
|
+
"""
|
|
112
|
+
url = f"{base_url}/hpc/daemon/register"
|
|
113
|
+
headers = {"Authorization": f"Bearer {access_token}"}
|
|
114
|
+
|
|
115
|
+
hostname = socket.gethostname()
|
|
116
|
+
capabilities = detect_capabilities()
|
|
117
|
+
|
|
118
|
+
data = {
|
|
119
|
+
"daemon_id": daemon_id,
|
|
120
|
+
"hostname": hostname,
|
|
121
|
+
"cluster_name": cluster_name or hostname,
|
|
122
|
+
"capabilities": capabilities,
|
|
123
|
+
"metadata": {"setup_time": time.time(), "python_version": sys.version},
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
if server_instructions:
|
|
127
|
+
data["server_instructions"] = server_instructions
|
|
128
|
+
if hpc_skill:
|
|
129
|
+
data["hpc_skill"] = hpc_skill
|
|
130
|
+
if custom_skills:
|
|
131
|
+
data["custom_skills"] = custom_skills
|
|
132
|
+
|
|
133
|
+
return http_request(url, method="POST", data=data, headers=headers, exit_on_error=True)
|
|
134
|
+
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
import json
|
|
2
|
+
import logging
|
|
3
|
+
import ssl
|
|
4
|
+
import sys
|
|
5
|
+
import time
|
|
6
|
+
import urllib.error
|
|
7
|
+
import urllib.parse
|
|
8
|
+
import urllib.request
|
|
9
|
+
|
|
10
|
+
logger = logging.getLogger("daemon")
|
|
11
|
+
|
|
12
|
+
# Domains where SSL verification is skipped (local dev with self-signed certs)
|
|
13
|
+
SSL_BYPASS_DOMAINS = ["localhost", "127.0.0.1"]
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
def http_request(
|
|
17
|
+
url: str,
|
|
18
|
+
method: str = "GET",
|
|
19
|
+
data: dict | None = None,
|
|
20
|
+
headers: dict[str, str] | None = None,
|
|
21
|
+
exit_on_error: bool = False,
|
|
22
|
+
max_retries: int = 3,
|
|
23
|
+
timeout: int = 30,
|
|
24
|
+
) -> dict:
|
|
25
|
+
if headers is None:
|
|
26
|
+
headers = {}
|
|
27
|
+
|
|
28
|
+
headers["ngrok-skip-browser-warning"] = "true"
|
|
29
|
+
|
|
30
|
+
if data is not None:
|
|
31
|
+
data_bytes = json.dumps(data).encode("utf-8")
|
|
32
|
+
headers["Content-Type"] = "application/json"
|
|
33
|
+
else:
|
|
34
|
+
data_bytes = None
|
|
35
|
+
|
|
36
|
+
req = urllib.request.Request(url, data=data_bytes, headers=headers, method=method)
|
|
37
|
+
ctx = ssl.create_default_context()
|
|
38
|
+
|
|
39
|
+
# Disable SSL verification for local development with self-signed certs
|
|
40
|
+
hostname = urllib.parse.urlparse(url).hostname or ""
|
|
41
|
+
if hostname in SSL_BYPASS_DOMAINS:
|
|
42
|
+
ctx.check_hostname = False
|
|
43
|
+
ctx.verify_mode = ssl.CERT_NONE
|
|
44
|
+
|
|
45
|
+
retry_delay = 1.0
|
|
46
|
+
last_error = None
|
|
47
|
+
|
|
48
|
+
for attempt in range(max_retries):
|
|
49
|
+
try:
|
|
50
|
+
with urllib.request.urlopen(req, context=ctx, timeout=timeout) as response:
|
|
51
|
+
if 200 <= response.status < 300:
|
|
52
|
+
resp_data = response.read()
|
|
53
|
+
if resp_data:
|
|
54
|
+
return json.loads(resp_data)
|
|
55
|
+
return {}
|
|
56
|
+
logger.error(f"HTTP {response.status}: {response.read().decode()}")
|
|
57
|
+
if exit_on_error:
|
|
58
|
+
sys.exit(1)
|
|
59
|
+
raise urllib.error.HTTPError(
|
|
60
|
+
url, response.status, "HTTP error", response.headers, None
|
|
61
|
+
)
|
|
62
|
+
except urllib.error.HTTPError as e:
|
|
63
|
+
if exit_on_error:
|
|
64
|
+
logger.error(f"HTTP Error: {e.code} {e.reason}")
|
|
65
|
+
sys.exit(1)
|
|
66
|
+
raise
|
|
67
|
+
except (ssl.SSLError, urllib.error.URLError) as e:
|
|
68
|
+
last_error = e
|
|
69
|
+
error_str = str(e)
|
|
70
|
+
# Retry on SSL/connection errors
|
|
71
|
+
if "SSL" in error_str or "EOF" in error_str or "Connection" in error_str:
|
|
72
|
+
if attempt < max_retries - 1:
|
|
73
|
+
logger.warning(
|
|
74
|
+
f"Connection error (attempt {attempt + 1}/{max_retries}): {e}. "
|
|
75
|
+
f"Retrying in {retry_delay}s..."
|
|
76
|
+
)
|
|
77
|
+
time.sleep(retry_delay)
|
|
78
|
+
retry_delay *= 2
|
|
79
|
+
continue
|
|
80
|
+
if exit_on_error:
|
|
81
|
+
logger.error(f"URL Error: {e}")
|
|
82
|
+
sys.exit(1)
|
|
83
|
+
raise
|
|
84
|
+
|
|
85
|
+
# All retries exhausted
|
|
86
|
+
if exit_on_error:
|
|
87
|
+
logger.error(f"Failed after {max_retries} attempts: {last_error}")
|
|
88
|
+
sys.exit(1)
|
|
89
|
+
raise last_error
|
|
90
|
+
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
============================================================
|
|
2
|
+
DISCLAIMER & USER AGREEMENT
|
|
3
|
+
============================================================
|
|
4
|
+
|
|
5
|
+
By proceeding, you acknowledge and agree to the following:
|
|
6
|
+
|
|
7
|
+
1. REMOTE CODE EXECUTION: The HPC daemon enables AI agents
|
|
8
|
+
(via Claude Code by Anthropic) to execute shell commands on
|
|
9
|
+
this machine. This includes reading, writing, moving, and
|
|
10
|
+
deleting files, running programs, and submitting jobs.
|
|
11
|
+
Commands run with YOUR user permissions.
|
|
12
|
+
|
|
13
|
+
2. DIRECTORY RESTRICTIONS: You will be asked to configure
|
|
14
|
+
allowed working directories. This is the PRIMARY safeguard
|
|
15
|
+
against unintended file modifications and data loss.
|
|
16
|
+
|
|
17
|
+
- WITH restrictions: write commands (rm, mv, cp, etc.) are
|
|
18
|
+
blocked outside your chosen directories.
|
|
19
|
+
- WITHOUT restrictions: the agent can modify or delete ANY
|
|
20
|
+
file your user account can access, including personal
|
|
21
|
+
files, dotfiles, SSH keys, and project directories.
|
|
22
|
+
|
|
23
|
+
Skipping restrictions may result in PERMANENT DATA LOSS.
|
|
24
|
+
On personal computers, we strongly recommend using the
|
|
25
|
+
default workspace directory.
|
|
26
|
+
|
|
27
|
+
3. SKILLS & INSTRUCTIONS: You are strongly encouraged to
|
|
28
|
+
provide an HPC skill and server instructions during setup.
|
|
29
|
+
Without adequate guidance, the AI agent may use incorrect
|
|
30
|
+
commands, wrong partitions, or inappropriate paths.
|
|
31
|
+
Insufficient configuration is at your own risk.
|
|
32
|
+
|
|
33
|
+
4. CREDENTIAL STORAGE: Your API key is stored locally in
|
|
34
|
+
~/.hpc_daemon/state.db. This file is restricted to your
|
|
35
|
+
user account (mode 0600), but anyone with access to your
|
|
36
|
+
Unix account can read it. Do NOT run this daemon on a
|
|
37
|
+
shared account where other people can log in as your user.
|
|
38
|
+
|
|
39
|
+
5. NO WARRANTY: This software is provided as-is. Tiptree
|
|
40
|
+
assumes no liability for data loss, job failures, or any
|
|
41
|
+
other damage resulting from use of the HPC daemon.
|