eray 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- eray-0.1.0/PKG-INFO +561 -0
- eray-0.1.0/README.md +530 -0
- eray-0.1.0/eray/__init__.py +192 -0
- eray-0.1.0/eray/cli/__init__.py +36 -0
- eray-0.1.0/eray/cli/autoscale.py +418 -0
- eray-0.1.0/eray/cli/dashboard.py +437 -0
- eray-0.1.0/eray/cli/fleet.py +422 -0
- eray-0.1.0/eray/cli/hostjob.py +506 -0
- eray-0.1.0/eray/cli/jobs.py +1350 -0
- eray-0.1.0/eray/cli/main.py +802 -0
- eray-0.1.0/eray/cli/qr.py +249 -0
- eray-0.1.0/eray/cli/tpu.py +763 -0
- eray-0.1.0/eray/cli/utils.py +707 -0
- eray-0.1.0/eray/core/__init__.py +63 -0
- eray-0.1.0/eray/core/cluster.py +106 -0
- eray-0.1.0/eray/core/exceptions.py +207 -0
- eray-0.1.0/eray/core/monitoring.py +383 -0
- eray-0.1.0/eray/core/sentinels.py +97 -0
- eray-0.1.0/eray/core/status.py +105 -0
- eray-0.1.0/eray/docker/__init__.py +48 -0
- eray-0.1.0/eray/docker/config.py +131 -0
- eray-0.1.0/eray/docker/runner.py +351 -0
- eray-0.1.0/eray/execution/__init__.py +43 -0
- eray-0.1.0/eray/execution/decorators.py +281 -0
- eray-0.1.0/eray/execution/executor.py +741 -0
- eray-0.1.0/eray/execution/remote.py +655 -0
- eray-0.1.0/eray/gcp.py +177 -0
- eray-0.1.0/eray/pool/__init__.py +40 -0
- eray-0.1.0/eray/pool/base.py +284 -0
- eray-0.1.0/eray/pool/device_host.py +717 -0
- eray-0.1.0/eray/pool/slice.py +1195 -0
- eray-0.1.0/eray/provision/__init__.py +66 -0
- eray-0.1.0/eray/provision/fleet.py +332 -0
- eray-0.1.0/eray/provision/launcher.py +497 -0
- eray-0.1.0/eray/provision/qr.py +442 -0
- eray-0.1.0/eray/provision/registry.py +510 -0
- eray-0.1.0/eray/provision/templates/cluster-template.yaml +63 -0
- eray-0.1.0/eray/provision/tunnel.py +424 -0
- eray-0.1.0/eray/provision/watcher.py +777 -0
- eray-0.1.0/eray/py.typed +0 -0
- eray-0.1.0/eray/resources/__init__.py +50 -0
- eray-0.1.0/eray/resources/configs.py +676 -0
- eray-0.1.0/eray/resources/hardware.py +82 -0
- eray-0.1.0/eray/resources/ray_resources.py +387 -0
- eray-0.1.0/eray/resources/topology.py +344 -0
- eray-0.1.0/eray/swarm.py +845 -0
- eray-0.1.0/pyproject.toml +115 -0
eray-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,561 @@
|
|
|
1
|
+
Metadata-Version: 2.3
|
|
2
|
+
Name: eray
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Ray-based distributed execution, scaling, and resource management for ML workloads on CPUs, GPUs, and TPUs
|
|
5
|
+
Keywords: Ray,Distributed Computing,TPU,GPU,Scaling,EasyDeL,eFormer,JAX
|
|
6
|
+
Author: Erfan Zare Chavoshi
|
|
7
|
+
Author-email: Erfan Zare Chavoshi <Erfanzare810@gmail.com>
|
|
8
|
+
License: Apache-2.0
|
|
9
|
+
Classifier: Development Status :: 3 - Alpha
|
|
10
|
+
Classifier: Intended Audience :: Developers
|
|
11
|
+
Classifier: Intended Audience :: Science/Research
|
|
12
|
+
Classifier: Topic :: System :: Distributed Computing
|
|
13
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
14
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
19
|
+
Requires-Dist: ray[default]>=2.54.0
|
|
20
|
+
Requires-Dist: mergedeep~=1.3.4
|
|
21
|
+
Requires-Dist: tblib~=3.0.0
|
|
22
|
+
Requires-Dist: requests>=2.31.0
|
|
23
|
+
Requires-Dist: click>=8.1.0
|
|
24
|
+
Requires-Dist: pytest ; extra == 'dev'
|
|
25
|
+
Requires-Python: >=3.11, <3.14
|
|
26
|
+
Project-URL: Documentation, https://eray.readthedocs.io/
|
|
27
|
+
Project-URL: Homepage, https://github.com/erfanzar/eray
|
|
28
|
+
Project-URL: Repository, https://github.com/erfanzar/eray
|
|
29
|
+
Provides-Extra: dev
|
|
30
|
+
Description-Content-Type: text/markdown
|
|
31
|
+
|
|
32
|
+
# eray ⚡
|
|
33
|
+
|
|
34
|
+
[](https://pypi.org/project/eray/)
|
|
35
|
+
[](https://www.python.org/)
|
|
36
|
+
[](https://github.com/ray-project/ray)
|
|
37
|
+
[](LICENSE)
|
|
38
|
+
[](#)
|
|
39
|
+
|
|
40
|
+
> Ray-based distributed execution, scaling, and resource management for ML workloads on CPUs, GPUs, and TPUs.
|
|
41
|
+
|
|
42
|
+
## What is eray?
|
|
43
|
+
|
|
44
|
+
**eray** is a standalone extraction of the Ray executor stack from [eFormer](https://github.com/erfanzar/eFormer) / [EasyDeL](https://github.com/erfanzar/EasyDeL). It provides everything you need to run distributed machine learning workloads on Ray-managed clusters:
|
|
45
|
+
|
|
46
|
+
- **Fault-tolerant execution** — automatic retry and resumption for preempted/failed jobs
|
|
47
|
+
- **Multi-slice TPU support** — coordinate workloads across TPU pod slices with MegaScale
|
|
48
|
+
- **GPU & CPU execution** — run on NVIDIA GPUs, Intel/AMD accelerators, or plain CPU
|
|
49
|
+
- **Docker integration** — build, push, and run containers on remote nodes
|
|
50
|
+
- **Resource pooling** — actor pool management with health checks and auto-scaling
|
|
51
|
+
- **Rich type system** — job status tracking, exception serialization across processes
|
|
52
|
+
|
|
53
|
+
## Installation
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
pip install eray
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
For TPU environments, you may also want `jax` and `jaxlib` installed separately.
|
|
60
|
+
|
|
61
|
+
## Quickstart
|
|
62
|
+
|
|
63
|
+
### Single-pod execution
|
|
64
|
+
|
|
65
|
+
```python
|
|
66
|
+
import eray
|
|
67
|
+
|
|
68
|
+
# Run a function on a GPU pod with automatic retries
|
|
69
|
+
@eray.execute(eray.GpuAcceleratorConfig(count=1, gpu_model="A100"))
|
|
70
|
+
def train_step(batch):
|
|
71
|
+
...
|
|
72
|
+
return loss
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### Resumable execution with preemption recovery
|
|
76
|
+
|
|
77
|
+
```python
|
|
78
|
+
result = eray.execute_resumable(
|
|
79
|
+
remote_fn=train_fn,
|
|
80
|
+
accelerator_config=eray.TpuAcceleratorConfig(tpu_version="v4-8", pod_count=1),
|
|
81
|
+
max_preemption_retries=1_000_000,
|
|
82
|
+
)
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### Multi-slice TPU autoscaling
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
results = eray.autoscale_execute_resumable(
|
|
89
|
+
remote_fn=distributed_train,
|
|
90
|
+
accelerator_config=eray.TpuAcceleratorConfig(
|
|
91
|
+
tpu_version="v4-32",
|
|
92
|
+
pod_count=4,
|
|
93
|
+
),
|
|
94
|
+
)
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
### Docker on distributed nodes
|
|
98
|
+
|
|
99
|
+
```python
|
|
100
|
+
from eray import DockerConfig, run_docker_multislice
|
|
101
|
+
|
|
102
|
+
docker_config = DockerConfig(
|
|
103
|
+
image="my-ml-image:latest",
|
|
104
|
+
command="python train.py",
|
|
105
|
+
volumes={"/data": "/data"},
|
|
106
|
+
)
|
|
107
|
+
|
|
108
|
+
outputs = run_docker_multislice(
|
|
109
|
+
docker_config,
|
|
110
|
+
accelerator_config=eray.TpuAcceleratorConfig(tpu_version="v4-32", pod_count=4),
|
|
111
|
+
)
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### `device_remote` decorator
|
|
115
|
+
|
|
116
|
+
```python
|
|
117
|
+
@eray.device_remote(accelerator_config=eray.TpuAcceleratorConfig(tpu_version="v4-8"))
|
|
118
|
+
class Trainer:
|
|
119
|
+
def train(self, data):
|
|
120
|
+
...
|
|
121
|
+
|
|
122
|
+
trainer = Trainer()
|
|
123
|
+
trainer.train.remote(data)
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
### Host chip splitting
|
|
127
|
+
|
|
128
|
+
A TPU host is normally driven by one process owning every chip. eray can carve
|
|
129
|
+
a host into multiple processes with disjoint chip ownership (the same libtpu
|
|
130
|
+
environment mechanism torch_xla uses):
|
|
131
|
+
|
|
132
|
+
```python
|
|
133
|
+
import eray
|
|
134
|
+
|
|
135
|
+
# Plan only (pure, no Ray): inspect the per-split environment.
|
|
136
|
+
plan = eray.plan_host_split(num_chips=4, num_splits=4, mode="isolated")
|
|
137
|
+
|
|
138
|
+
# On a slice: run fn once per split on every host.
|
|
139
|
+
# SliceActor.run_split_remote_fn / DeviceHostActor.run_split_remote_fn
|
|
140
|
+
refs = ray.get(host_actor.run_split_remote_fn.remote(fn, 4, mode="isolated"))
|
|
141
|
+
results = ray.get(refs) # one result per split; ERAY_SPLIT_ID identifies each
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
- `isolated` (default): each split is an independent runtime with 1 or 2 chips
|
|
145
|
+
assigned disjointly by Ray — e.g. four single-chip serving replicas on a
|
|
146
|
+
v4/v5p host.
|
|
147
|
+
- `cooperative`: one process per chip forming a single multi-process runtime
|
|
148
|
+
(communicating over ICI via `jax.distributed`); chip-dependent variables
|
|
149
|
+
(`TPU_VISIBLE_CHIPS`, `CLOUD_TPU_TASK_ID`, `TPU_PROCESS_PORT`) are bound
|
|
150
|
+
inside each task from Ray's chip assignment.
|
|
151
|
+
|
|
152
|
+
### Swarms — heterogeneous runs with a chip plan
|
|
153
|
+
|
|
154
|
+
`eray.swarm` packs several independent workloads onto the same TPU hosts.
|
|
155
|
+
`SwarmConfig` mirrors `TpuAcceleratorConfig` (it takes the chip type) plus the
|
|
156
|
+
number of runs and an optional plan for how to split each host's chips:
|
|
157
|
+
|
|
158
|
+
```python
|
|
159
|
+
import eray
|
|
160
|
+
|
|
161
|
+
# Even split: four replicas, one chip each per host.
|
|
162
|
+
status = eray.swarm_execute(serve_replica, eray.SwarmConfig(tpu_version="v5p-8", num_runs=4))
|
|
163
|
+
|
|
164
|
+
# A plan: three different workloads, chips split (2, 1, 1).
|
|
165
|
+
status = eray.swarm_execute(
|
|
166
|
+
[
|
|
167
|
+
eray.SwarmRun(train_probe, name="train"),
|
|
168
|
+
eray.SwarmRun(serve_small, name="serve"),
|
|
169
|
+
eray.SwarmRun(eval_loop, name="eval"),
|
|
170
|
+
],
|
|
171
|
+
eray.SwarmConfig(tpu_version="v5p-8", chip_split=(2, 1, 1)),
|
|
172
|
+
)
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
**GPU swarms** use `GpuSwarmConfig` — same layout knobs, but no slice pool:
|
|
176
|
+
Ray natively assigns disjoint GPUs (`CUDA_VISIBLE_DEVICES`) per run, any GPU
|
|
177
|
+
count is allowed (fractions of one GPU too), and `colocate=True` packs all
|
|
178
|
+
runs onto one node via a placement group:
|
|
179
|
+
|
|
180
|
+
```python
|
|
181
|
+
status = eray.swarm_execute(
|
|
182
|
+
[
|
|
183
|
+
eray.SwarmRun(train_probe, name="train", num_cores=8), # 2 GPUs, 8 CPU cores
|
|
184
|
+
eray.SwarmRun(serve_small, name="serve"), # 1 GPU
|
|
185
|
+
eray.SwarmRun(eval_loop, name="eval"), # 1 GPU
|
|
186
|
+
],
|
|
187
|
+
eray.GpuSwarmConfig(gpu_model="A100", gpu_split=(2, 1, 1), colocate=True),
|
|
188
|
+
)
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
`SwarmRun.num_cores` reserves CPU cores per run on both TPU and GPU swarms
|
|
192
|
+
(default: the config's `cpu_count` on GPU, the launcher default on TPU).
|
|
193
|
+
GPU runs see `ERAY_RUN_GPUS` instead of `ERAY_RUN_CHIPS`.
|
|
194
|
+
|
|
195
|
+
There is also a decorator form — bare `SwarmRun(...)` specs (no `fn`) bind to
|
|
196
|
+
the decorated function, and calling it launches the swarm:
|
|
197
|
+
|
|
198
|
+
```python
|
|
199
|
+
@eray.swarmed(
|
|
200
|
+
eray.SwarmConfig(tpu_version="v5p-8"),
|
|
201
|
+
runs=[
|
|
202
|
+
eray.SwarmRun(name="lr3e4", f_kwargs={"lr": 3e-4}),
|
|
203
|
+
eray.SwarmRun(name="lr1e4", f_kwargs={"lr": 1e-4}),
|
|
204
|
+
eray.SwarmRun(name="lr5e5", f_kwargs={"lr": 5e-5}),
|
|
205
|
+
eray.SwarmRun(name="lr1e5", f_kwargs={"lr": 1e-5}),
|
|
206
|
+
],
|
|
207
|
+
)
|
|
208
|
+
def train(lr, warmup):
|
|
209
|
+
...
|
|
210
|
+
|
|
211
|
+
status = train(warmup=100) # call-time kwargs broadcast to every run
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
A run may also be a plain **class**: it is instantiated as a long-lived,
|
|
215
|
+
detached named Ray actor on its chip share, addressable from any driver:
|
|
216
|
+
|
|
217
|
+
```python
|
|
218
|
+
runs = [
|
|
219
|
+
eray.SwarmRun(ServeReplica, name="serve-a"), # class -> named actor
|
|
220
|
+
eray.SwarmRun(ServeReplica, name="serve-b"),
|
|
221
|
+
eray.SwarmRun(train_probe, name="train"), # function -> task
|
|
222
|
+
]
|
|
223
|
+
status = eray.swarm_execute(
|
|
224
|
+
runs,
|
|
225
|
+
eray.SwarmConfig(tpu_version="v5p-8", chip_split=(1, 1, 2), namespace="my-swarm"),
|
|
226
|
+
)
|
|
227
|
+
|
|
228
|
+
# later, from any driver:
|
|
229
|
+
serve = ray.get_actor("serve-a", namespace="my-swarm")
|
|
230
|
+
ray.get(serve.generate.remote(prompt))
|
|
231
|
+
|
|
232
|
+
eray.shutdown_swarm("my-swarm") # kill all of the swarm's named actors
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Each run is an isolated runtime with disjoint chips (assigned by Ray) and sees
|
|
236
|
+
`ERAY_RUN_ID` / `ERAY_NUM_RUNS` / `ERAY_RUN_CHIPS` / `ERAY_RUN_NAME` in its
|
|
237
|
+
environment. Chip counts per run must be 1, 2, or the whole host, and the plan
|
|
238
|
+
must not oversubscribe the host. Hardware note (validated on v5p-8): 2-chip
|
|
239
|
+
runs only form a sub-slice on aligned chip pairs ({0,1}, {2,3}); a misaligned
|
|
240
|
+
Ray assignment raises a clear error in-task instead of crashing libtpu. When
|
|
241
|
+
launching a Ray driver on a TPU VM, invoke the venv python directly — Ray's
|
|
242
|
+
`uv run` hook repackages the working dir and workers lose libtpu. With `pod_count > 1` or multi-host slices the
|
|
243
|
+
same layout repeats on every host (actor names get a `-s{slice}h{host}`
|
|
244
|
+
suffix). Class-run actors are detached: they survive the swarm call and hold
|
|
245
|
+
their chips until `shutdown_swarm(namespace)` or `ray.kill`.
|
|
246
|
+
|
|
247
|
+
## Module Map
|
|
248
|
+
|
|
249
|
+
| Module | Description |
|
|
250
|
+
|--------|-------------|
|
|
251
|
+
| `eray.types` | Job status types, exception serialization, sentinels, coordination primitives |
|
|
252
|
+
| `eray.resource_manager` | `ComputeResourceConfig`, `RayResources`, hardware constants |
|
|
253
|
+
| `eray.pool_manager` | Actor pool management, `SlicePoolManager`, `DeviceHostActor` |
|
|
254
|
+
| `eray.executor` | `RayExecutor`, `execute`, `autoscale_execute`, `device_remote` |
|
|
255
|
+
| `eray.docker_executor` | Docker config, build/push, container execution |
|
|
256
|
+
|
|
257
|
+
## Key Types
|
|
258
|
+
|
|
259
|
+
```python
|
|
260
|
+
from eray import (
|
|
261
|
+
# Execution
|
|
262
|
+
execute, execute_resumable,
|
|
263
|
+
autoscale_execute, autoscale_execute_resumable,
|
|
264
|
+
device_remote, RayExecutor,
|
|
265
|
+
|
|
266
|
+
# Resource configs
|
|
267
|
+
CpuAcceleratorConfig, GpuAcceleratorConfig, TpuAcceleratorConfig,
|
|
268
|
+
ComputeResourceConfig, RayResources, AcceleratorConfigType,
|
|
269
|
+
|
|
270
|
+
# Job status
|
|
271
|
+
JobSucceeded, JobFailed, JobPreempted, JobError, JobStatus, JobInfo,
|
|
272
|
+
|
|
273
|
+
# Docker
|
|
274
|
+
DockerConfig, run_docker_on_pod, run_docker_multislice,
|
|
275
|
+
|
|
276
|
+
# Pool management
|
|
277
|
+
ResourcePoolManager, SlicePoolManager, DeviceHostActor,
|
|
278
|
+
|
|
279
|
+
# Host chip splitting
|
|
280
|
+
HostSplitPlan, plan_host_split,
|
|
281
|
+
|
|
282
|
+
# Swarms
|
|
283
|
+
SwarmConfig, GpuSwarmConfig, SwarmRun, swarm_execute, swarmed, shutdown_swarm, HostPartitionPlan, plan_host_partition,
|
|
284
|
+
|
|
285
|
+
# Utilities
|
|
286
|
+
handle_ray_error, ExceptionInfo, StopwatchActor, RefBox, DONE,
|
|
287
|
+
)
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
## Environment Variables
|
|
291
|
+
|
|
292
|
+
| Variable | Default | Description |
|
|
293
|
+
|----------|---------|-------------|
|
|
294
|
+
| `ERAY_SCALE_RETRY_SLEEP` | `60` | Sleep between retry attempts when slices are unavailable |
|
|
295
|
+
| `ERAY_SUBPROCESS_TIMEOUT` | — | Timeout for forked subprocess execution |
|
|
296
|
+
| `ERAY_SAFE_GATHER` | — | Use safe gather (prune dead actors) in slice preparation |
|
|
297
|
+
| `ERAY_MODERATE` | — | Adjust num_hosts/num_devices discovery |
|
|
298
|
+
| `ERAY_KILL_VFIO` | — | Kill VFIO holders on TPU hosts |
|
|
299
|
+
| `ERAY_INSTALL_LSOF` | — | Attempt quiet lsof install for VFIO holder detection |
|
|
300
|
+
| `ERAY_HOST_HEALTH_WAIT` | `60` | Timeout for waiting on all hosts to become healthy |
|
|
301
|
+
| `ERAY_SCALE_POLL` | `30` | Poll interval for actor pool scaling |
|
|
302
|
+
| `ERAY_SCALE_ADD_TIMEOUT` | `604800` | Timeout for adding actors to pool (7 days) |
|
|
303
|
+
|
|
304
|
+
## CLI — TPU Cluster Management
|
|
305
|
+
|
|
306
|
+
`eray` ships with a CLI for connecting TPU hosts into a Ray cluster. It ports the Ray connection logic from `scripts/tpu_setup.sh` into a clean Python tool.
|
|
307
|
+
|
|
308
|
+
Run on the TPU VM itself, no flags are needed at all — the TPU's name, type,
|
|
309
|
+
zone, and every worker's IP are auto-detected from the instance metadata
|
|
310
|
+
server, and commands targeting this machine execute locally (no SSH):
|
|
311
|
+
|
|
312
|
+
```bash
|
|
313
|
+
eray tpu connect # auto-detects the current TPU
|
|
314
|
+
eray tpu status # auto-detects the head address
|
|
315
|
+
eray tpu health
|
|
316
|
+
eray resources # cluster resources: used / total / free / util
|
|
317
|
+
eray tpu disconnect
|
|
318
|
+
```
|
|
319
|
+
|
|
320
|
+
### Prerequisites
|
|
321
|
+
|
|
322
|
+
- `gcloud` CLI installed and authenticated
|
|
323
|
+
- TPU VMs in `READY` state
|
|
324
|
+
- `ray` installed on each TPU host
|
|
325
|
+
|
|
326
|
+
### Commands
|
|
327
|
+
|
|
328
|
+
```bash
|
|
329
|
+
# Connect a TPU slice into a Ray cluster
|
|
330
|
+
eray tpu connect -n my-tpu -p my-project -z us-central2-b
|
|
331
|
+
|
|
332
|
+
# Check cluster status (nodes, TPU resources)
|
|
333
|
+
eray tpu status -a 10.0.0.1:6379
|
|
334
|
+
|
|
335
|
+
# Run health check (JAX devices per host)
|
|
336
|
+
eray tpu health -a 10.0.0.1:6379
|
|
337
|
+
|
|
338
|
+
# Show resource usage (CPU, TPU, memory, custom resources); --per-node for a host table
|
|
339
|
+
eray resources -a 10.0.0.1:6379 --per-node
|
|
340
|
+
|
|
341
|
+
# Stop Ray on all hosts
|
|
342
|
+
eray tpu disconnect -n my-tpu -p my-project -z us-central2-b
|
|
343
|
+
|
|
344
|
+
# List TPUs in a zone
|
|
345
|
+
eray tpu list -p my-project -z us-central2-b
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
### How `connect` works
|
|
349
|
+
|
|
350
|
+
1. **Discovers TPU** via `gcloud compute tpus tpu-vm describe` → gets internal IPs, accelerator type, worker count
|
|
351
|
+
2. **Cleans existing Ray** on all hosts (kills stale processes, frees ports)
|
|
352
|
+
3. **Starts Ray head** on worker 0 with TPU-specific custom resources
|
|
353
|
+
4. **Starts Ray workers** on all other hosts, pointing at the head
|
|
354
|
+
5. **Waits for readiness** — polls `ray.nodes()` until all hosts register
|
|
355
|
+
|
|
356
|
+
### Resource allocation
|
|
357
|
+
|
|
358
|
+
The cluster is configured with custom Ray resources matching the TPU topology:
|
|
359
|
+
|
|
360
|
+
| Resource | Head | Workers |
|
|
361
|
+
|----------|------|---------|
|
|
362
|
+
| `TPU` | `chips_per_host` | `chips_per_host` |
|
|
363
|
+
| `TPU-{version}` | `chips_per_host` | `chips_per_host` |
|
|
364
|
+
| `TPU-{version}-{slice}-head` | `1` | — |
|
|
365
|
+
| `accelerator_type:TPU-{VERSION}` | `1` | `1` |
|
|
366
|
+
| `head-node` | `1` | — |
|
|
367
|
+
|
|
368
|
+
### JSON output
|
|
369
|
+
|
|
370
|
+
All commands support `--json` for machine-readable output:
|
|
371
|
+
|
|
372
|
+
```bash
|
|
373
|
+
eray tpu connect -n my-tpu -p proj -z zone --json | jq .ray_address
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
## Spot fleet management
|
|
377
|
+
|
|
378
|
+
eray manages spot TPU capacity end to end: **queued resources** are the
|
|
379
|
+
provisioning primitive, a **fleet registry** remembers every cluster, and a
|
|
380
|
+
**watcher** re-queues preempted capacity, reconnects Ray, and resubmits jobs.
|
|
381
|
+
|
|
382
|
+
```bash
|
|
383
|
+
# capacity (stateless): spot by default, waits in queue instead of failing
|
|
384
|
+
eray qr create trainer1 --type v5p-64 --wait
|
|
385
|
+
eray qr list
|
|
386
|
+
|
|
387
|
+
# fleet (stateful): registry + reconciliation
|
|
388
|
+
eray fleet init --state gs://my-bucket/eray/clusters.json # shared registry (optional)
|
|
389
|
+
eray fleet add trainer1 --type v5p-64 --setup-easydel # EasyDeL bootstrap (--branch vnext pins a ref)
|
|
390
|
+
eray fleet add gpubox --type v5p-8 --bootstrap-cmd '...' # or any custom per-host bootstrap command
|
|
391
|
+
eray fleet add n_server_spot_m # adopt an existing QR by name
|
|
392
|
+
eray fleet ensure trainer1 --wait 3600 # request capacity + connect Ray, idempotent
|
|
393
|
+
eray fleet status
|
|
394
|
+
|
|
395
|
+
# jobs against a registered cluster; --restartable opts into auto-resubmission
|
|
396
|
+
eray run -c trainer1 --restartable -- python train.py
|
|
397
|
+
|
|
398
|
+
# the watcher: detect preemption -> re-queue -> reconnect -> resubmit
|
|
399
|
+
eray fleet watch --resubmit # foreground; --once for cron; --dry-run to preview
|
|
400
|
+
eray fleet pause / eray fleet resume # kill switch; resume clears HALTED_* park states
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
How recovery works: the watcher (run it OFF the TPU — the head dies with the
|
|
404
|
+
slice) fuses three signals per tick (queued-resource state, node state, head
|
|
405
|
+
TCP reachability). Definitive preemption (node `PREEMPTED`, QR `SUSPENDED`)
|
|
406
|
+
triggers: mark degraded → delete the owned QR → create `{name}-r{gen+1}` (the
|
|
407
|
+
node id never changes, so `eray tpu connect -n NAME` always works) → wait
|
|
408
|
+
ACTIVE → bootstrap once per generation → reconnect → resubmit `--restartable`
|
|
409
|
+
jobs as `{id}-p{N}` with `ERAY_RESTART_COUNT`/`ERAY_PREEMPTED_FROM` env (your
|
|
410
|
+
trainer resumes from its own checkpoints). Budgets (4 recreates/hour, 12/day)
|
|
411
|
+
and quota-flavored failures park the cluster in `HALTED_*` instead of looping.
|
|
412
|
+
Jobs are snapshotted while healthy, and resubmission re-packages the recorded
|
|
413
|
+
cwd, so run the watcher on the machine you submit from.
|
|
414
|
+
|
|
415
|
+
Systemd unit example:
|
|
416
|
+
|
|
417
|
+
```ini
|
|
418
|
+
[Unit]
|
|
419
|
+
Description=eray fleet watcher
|
|
420
|
+
After=network-online.target
|
|
421
|
+
|
|
422
|
+
[Service]
|
|
423
|
+
ExecStart=/path/to/venv/bin/eray fleet watch --resubmit
|
|
424
|
+
Restart=always
|
|
425
|
+
RestartSec=10
|
|
426
|
+
|
|
427
|
+
[Install]
|
|
428
|
+
WantedBy=default.target
|
|
429
|
+
```
|
|
430
|
+
|
|
431
|
+
### From a laptop (macOS)
|
|
432
|
+
|
|
433
|
+
Ray heads listen on internal VPC IPs, so a laptop outside the VPC splits the
|
|
434
|
+
work in two: the laptop is the control plane (everything gcloud-backed), and
|
|
435
|
+
one always-on box inside the VPC — any existing VM works — runs the watcher
|
|
436
|
+
and submits restartable jobs (resubmission re-packages that box's cwd).
|
|
437
|
+
|
|
438
|
+
Laptop, once (Python 3.11–3.13):
|
|
439
|
+
|
|
440
|
+
```bash
|
|
441
|
+
brew install google-cloud-sdk uv && gcloud auth login
|
|
442
|
+
uv tool install "eray @ git+https://github.com/erfanzar/eray"
|
|
443
|
+
eray fleet init --state gs://my-bucket/eray/clusters.json # shared with the watcher box
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
Watcher box, once (same install + `fleet init`, then keep alive via systemd
|
|
447
|
+
or tmux):
|
|
448
|
+
|
|
449
|
+
```bash
|
|
450
|
+
eray fleet watch --resubmit
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
Day to day, from the laptop:
|
|
454
|
+
|
|
455
|
+
```bash
|
|
456
|
+
eray fleet add trainer1 --type v5p-64 --setup-easydel && eray fleet up trainer1
|
|
457
|
+
eray fleet status --no-probe # registry view; probes need VPC access
|
|
458
|
+
gcloud compute ssh <watcher-box> -- 'cd repo && eray run -c trainer1 --restartable -- python train.py'
|
|
459
|
+
eray fleet tunnel trainer1 & # forward the head's dashboard to 127.0.0.1:8265
|
|
460
|
+
eray logs <job-id> -a http://127.0.0.1:8265 -f # live logs / browser dashboard through the tunnel
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
Preemption needs nothing from you: the watcher re-queues, reconnects, and
|
|
464
|
+
resubmits; `eray fleet status` shows the new generation. Non-restartable
|
|
465
|
+
one-off jobs can be submitted straight from the laptop through the tunnel
|
|
466
|
+
(`eray run -a http://127.0.0.1:8265 -- ...`).
|
|
467
|
+
|
|
468
|
+
## Elastic autoscale clusters
|
|
469
|
+
|
|
470
|
+
`eray autoscale` wraps Ray's own cluster-launcher (`ray up/down/dashboard`)
|
|
471
|
+
for the elastic many-small-slices case (prefer `eray fleet` above for
|
|
472
|
+
reliable big slices). `up`/`down`/`status` never need a YAML path: with no
|
|
473
|
+
argument they pick from whatever `generate` already wrote — auto-selecting
|
|
474
|
+
the only profile, or prompting when there's more than one.
|
|
475
|
+
|
|
476
|
+
```bash
|
|
477
|
+
eray autoscale generate --zones us-east5-a # writes ~/.eray/autoscale/easydel-us-east5-a.yaml
|
|
478
|
+
eray autoscale up # no path needed: picks/prompts among generated profiles
|
|
479
|
+
eray autoscale up easydel-us-east5-a # or name one directly (also accepts a YAML path)
|
|
480
|
+
eray autoscale status # every profile: zone, live status, head, uptime/died
|
|
481
|
+
eray autoscale down # picker again; stamps when it went down
|
|
482
|
+
```
|
|
483
|
+
|
|
484
|
+
Every `up`/`down` also registers the cluster into the same fleet registry
|
|
485
|
+
`eray fleet` uses (`kind=launcher`), which is what makes `eray dashboard`
|
|
486
|
+
below see autoscale clusters right alongside `eray fleet` ones.
|
|
487
|
+
|
|
488
|
+
## Dashboard
|
|
489
|
+
|
|
490
|
+
One place to see everything eray manages — `eray fleet` (QR-based TPU pods)
|
|
491
|
+
and `eray autoscale` (cluster-launcher) clusters together — and to open a
|
|
492
|
+
dashboard without hand-building a `ray dashboard`/gcloud SSH command:
|
|
493
|
+
|
|
494
|
+
```bash
|
|
495
|
+
eray dashboard # table of every registered cluster; prompts + opens one if interactive
|
|
496
|
+
eray dashboard ls # same table, no opening (scriptable; --json too)
|
|
497
|
+
eray dashboard open trainer1 -o # open (or reuse) trainer1's dashboard; -o also launches the browser
|
|
498
|
+
eray dashboard stop --all # close every tracked tunnel
|
|
499
|
+
```
|
|
500
|
+
|
|
501
|
+
Opening a dashboard spawns a tracked **background** port-forward instead of
|
|
502
|
+
blocking your terminal — `eray dashboard ls` shows the local port and PID of
|
|
503
|
+
anything already open, and asking to open the same cluster again reuses that
|
|
504
|
+
forward instead of colliding on the port (the failure mode a bare
|
|
505
|
+
`ray dashboard` has no memory to avoid on its own).
|
|
506
|
+
|
|
507
|
+
`eray dashboard` also sees generated autoscale profiles that were never
|
|
508
|
+
registered — brought up by hand, or by an eray version before `autoscale up`
|
|
509
|
+
started writing to the registry — listed as `UNREGISTERED`; you can `open`
|
|
510
|
+
one directly with no extra step, or run `eray autoscale up <name>` once
|
|
511
|
+
(idempotent against an already-running cluster) to make it a first-class
|
|
512
|
+
registered entry.
|
|
513
|
+
|
|
514
|
+
Opening a dashboard forwards **both** Ray's dashboard port (8265) and its GCS
|
|
515
|
+
port (6379) — over one SSH connection — because `ray status`, `eray
|
|
516
|
+
resources`, and `eray tpu status` read cluster state over GCS, not the
|
|
517
|
+
dashboard's HTTP port. (`eray` reads that state the way `ray status` does — a
|
|
518
|
+
pure GCS read via the autoscaler SDK — so it works over a tunnel;
|
|
519
|
+
`ray.init(address=...)` deliberately isn't used, as its driver join can't
|
|
520
|
+
complete over a single tunneled port.) You don't type the address either:
|
|
521
|
+
these commands **auto-detect** the tunnel's local port from eray's own tunnel
|
|
522
|
+
registry, so no `-a` is needed — correct even when the tunnel landed on a
|
|
523
|
+
non-default port because 8265/6379 were busy:
|
|
524
|
+
|
|
525
|
+
```bash
|
|
526
|
+
eray dashboard open trainer1 # one SSH forwarding dashboard (8265) + GCS (6379)
|
|
527
|
+
eray resources # auto-detects the GCS tunnel — no -a
|
|
528
|
+
eray tpu status # ditto
|
|
529
|
+
eray status # auto-detects the dashboard tunnel — no -a
|
|
530
|
+
eray dashboard open trainer1 --no-gcs # dashboard port only
|
|
531
|
+
```
|
|
532
|
+
|
|
533
|
+
Tunnels take ~15-20s to establish (SSH negotiation) before the ports answer.
|
|
534
|
+
`eray fleet tunnel` (the foreground, QR-only variant) forwards both ports the
|
|
535
|
+
same way; `--no-gcs` opts out. (`eray tpu health` runs a device probe on every
|
|
536
|
+
worker, so it needs a real driver join and only works on the cluster itself,
|
|
537
|
+
not from a laptop tunnel.)
|
|
538
|
+
|
|
539
|
+
## Relationship to eFormer
|
|
540
|
+
|
|
541
|
+
`eray` was extracted from [`eformer.executor.ray`](https://github.com/erfanzar/eFormer) to provide a focused, standalone Ray execution toolkit. It has **zero dependencies on eFormer or JAX** — you can use it for any Ray-based distributed workload, ML or otherwise.
|
|
542
|
+
|
|
543
|
+
## License
|
|
544
|
+
|
|
545
|
+
Apache License 2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE).
|
|
546
|
+
|
|
547
|
+
```
|
|
548
|
+
Copyright 2026 The EasyDeL/eray Author Erfan Zare Chavoshi
|
|
549
|
+
|
|
550
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
551
|
+
you may not use this file except in compliance with the License.
|
|
552
|
+
You may obtain a copy of the License at
|
|
553
|
+
|
|
554
|
+
https://www.apache.org/licenses/LICENSE-2.0
|
|
555
|
+
|
|
556
|
+
Unless required by applicable law or agreed to in writing, software
|
|
557
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
558
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
559
|
+
See the License for the specific language governing permissions and
|
|
560
|
+
limitations under the License.
|
|
561
|
+
```
|