slurmray 3.5.7__tar.gz → 3.5.9__tar.gz

{slurmray-3.5.7 → slurmray-3.5.9}/PKG-INFO

@@ -1,14 +1,17 @@
 Metadata-Version: 2.1
 Name: slurmray
-Version: 3.5.7
+Version: 3.5.9
 Summary: SlurmRay is a module for effortlessly distributing tasks on a Slurm cluster using the Ray library. 
 Home-page: https://henri-jamet.vercel.app/
 License: Apache License
 Author: Henri Jamet
 Author-email: henri.jamet@unil.ch
-Requires-Python: ==3.9.13
+Requires-Python: >=3.9,<4.0
 Classifier: License :: Other/Proprietary License
 Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
 Requires-Dist: dill (>=0.3.7,<0.4.0)
 Requires-Dist: paramiko (>=3.3.1,<4.0.0)
 Requires-Dist: pdoc3 (>=0.10.0,<0.11.0)
@@ -38,28 +41,42 @@ pip install slurmray
 
 ```python
 from slurmray.RayLauncher import RayLauncher
+import ray
+import torch
 
-if __name__ == "__main__":
-    def example_func(x):
-        import ray # All packages and resources must be imported inside the function
+def function_inside_function():
+    with open("slurmray/RayLauncher.py", "r") as f:
+        return f.read()[0:10]
 
-        return ray.cluster_resources(), x + 1
-
-    launcher = RayLauncher(
-        project_name="example",
-        func=example_func,
-        args={"x": 1},
-        modules=[],
-        node_nbr=1,
-        use_gpu=True,
-        memory=64,
-        max_running_time=15,
-        server_run=True,
-        server_ssh="curnagl.dcsr.unil.ch",
-        server_username="hjamet",
+def example_func(x):
+    result = (
+        ray.cluster_resources(),
+        f"GPU is available : {torch.cuda.is_available()}",
+        x + 1,
+        function_inside_function(),
     )
+    return result
+
+launcher = RayLauncher(
+    project_name="example", # Name of the project (will create a directory with this name in the current directory)
+    func=example_func, # Function to execute
+    args={"x": 1}, # Arguments of the function
+    files=["slurmray/RayLauncher.py"], # List of files to push to the cluster (file path will be recreated on the cluster)
+    modules=[], # List of modules to load on the curnagl Cluster (CUDA & CUDNN are automatically added if use_gpu=True)
+    node_nbr=1, # Number of nodes to use
+    use_gpu=True, # If you need A100 GPU, you can set it to True
+    memory=8, # In MegaBytes
+    max_running_time=5, # In minutes
+    runtime_env={"env_vars": {"NCCL_SOCKET_IFNAME": "eno1"}}, # Example of environment variable
+    server_run=True, # To run the code on the cluster and not locally
+    server_ssh="curnagl.dcsr.unil.ch", # Address of the SLURM server
+    server_username="hjamet", # Username to connect to the server
+    server_password=None, # Will be asked in the terminal
+)
 
-    result = launcher()
-    print(result)
+result = launcher()
+print(result)
 ```
+## Launcher documentation
 
+The Launcher documentation is available [here](https://htmlpreview.github.io/?https://raw.githubusercontent.com/hjamet/SLURM_RAY/main/documentation/RayLauncher.html).

slurmray-3.5.9/README.md

@@ -0,0 +1,59 @@
+# SLURM_RAY
+
+👉[Full documentation](https://henri-jamet.vercel.app/cards/documentation/slurm-ray/slurm-ray/)
+
+## Description
+
+**SlurmRay** is a module for effortlessly distributing tasks on a [Slurm](https://slurm.schedmd.com/) cluster using the [Ray](https://ray.io/) library. **SlurmRay** was initially designed to work with the [Curnagl](https://wiki.unil.ch/ci/books/high-performance-computing-hpc/page/curnagl) cluster at the *University of Lausanne*. However, it should be able to run on any [Slurm](https://slurm.schedmd.com/) cluster with a minimum of configuration.
+
+## Installation
+
+**SlurmRay** is designed to run both locally and on a cluster without any modification. This design is intended to allow work to be carried out on a local machine until the script seems to be working. It should then be possible to run it using all the resources of the cluster without having to modify the code.
+
+```bash
+pip install slurmray
+```
+
+## Usage
+
+```python
+from slurmray.RayLauncher import RayLauncher
+import ray
+import torch
+
+def function_inside_function():
+    with open("slurmray/RayLauncher.py", "r") as f:
+        return f.read()[0:10]
+
+def example_func(x):
+    result = (
+        ray.cluster_resources(),
+        f"GPU is available : {torch.cuda.is_available()}",
+        x + 1,
+        function_inside_function(),
+    )
+    return result
+
+launcher = RayLauncher(
+    project_name="example", # Name of the project (will create a directory with this name in the current directory)
+    func=example_func, # Function to execute
+    args={"x": 1}, # Arguments of the function
+    files=["slurmray/RayLauncher.py"], # List of files to push to the cluster (file path will be recreated on the cluster)
+    modules=[], # List of modules to load on the curnagl Cluster (CUDA & CUDNN are automatically added if use_gpu=True)
+    node_nbr=1, # Number of nodes to use
+    use_gpu=True, # If you need A100 GPU, you can set it to True
+    memory=8, # In MegaBytes
+    max_running_time=5, # In minutes
+    runtime_env={"env_vars": {"NCCL_SOCKET_IFNAME": "eno1"}}, # Example of environment variable
+    server_run=True, # To run the code on the cluster and not locally
+    server_ssh="curnagl.dcsr.unil.ch", # Address of the SLURM server
+    server_username="hjamet", # Username to connect to the server
+    server_password=None, # Will be asked in the terminal
+)
+
+result = launcher()
+print(result)
+```
+## Launcher documentation
+
+The Launcher documentation is available [here](https://htmlpreview.github.io/?https://raw.githubusercontent.com/hjamet/SLURM_RAY/main/documentation/RayLauncher.html).

{slurmray-3.5.7 → slurmray-3.5.9}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "slurmray"
-version = "3.5.7"
+version = "3.5.9"
 description = "SlurmRay is a module for effortlessly distributing tasks on a Slurm cluster using the Ray library. "
 authors = ["Henri Jamet <henri.jamet@unil.ch>"]
 license = "Apache License"
@@ -9,7 +9,7 @@ documentation = "https://henri-jamet.vercel.app/cards/documentation/slurm-ray/sl
 readme = "README.md"
 
 [tool.poetry.dependencies]
-python = "3.9.13"
+python = "^3.9"
 dill = "^0.3.7"
 ray = {extras = ["data", "serve", "train", "tune"], version = "^2.7.1"}
 pdoc3 = "^0.10.0"

{slurmray-3.5.7 → slurmray-3.5.9}/slurmray/RayLauncher.py

@@ -29,6 +29,7 @@ class RayLauncher:
         server_run: bool = True,
         server_ssh: str = "curnagl.dcsr.unil.ch",
         server_username: str = "hjamet",
+        server_password: str = None,
     ):
         """Initialize the launcher
 
@@ -46,6 +47,7 @@ class RayLauncher:
             server_run (bool, optional): If you run the launcher from your local machine, you can use this parameter to execute your function using online cluster ressources. Defaults to True.
             server_ssh (str, optional): If `server_run` is set to true, the addess of the **SLURM** server to use.
             server_username (str, optional): If `server_run` is set to true, the username with which you wish to connect.
+            server_password (str, optional): If `server_run` is set to true, the password of the user to connect to the server. CAUTION: never write your password in the code. Defaults to None.
         """
         # Save the parameters
         self.project_name = project_name
@@ -60,6 +62,7 @@ class RayLauncher:
         self.server_run = server_run
         self.server_ssh = server_ssh
         self.server_username = server_username
+        self.server_password = server_password
 
         self.modules = ["gcc", "python/3.9.13"] + [
             mod for mod in modules if mod not in ["gcc", "python/3.9.13"]
@@ -94,7 +97,7 @@ class RayLauncher:
             Any: Result of the function
         """
         # Sereialize function and arguments
-        self.serialize_func_and_args(self.func, self.args)
+        self.__serialize_func_and_args(self.func, self.args)
 
         if self.cluster:
             print("Cluster detected, running on cluster...")
@@ -156,7 +159,7 @@ class RayLauncher:
         # Copy the file to the server
         sftp.put(file_path, cluster_path)
 
-    def serialize_func_and_args(self, func: Callable = None, args: list = None):
+    def __serialize_func_and_args(self, func: Callable = None, args: list = None):
         """Serialize the function and the arguments
 
         Args:
@@ -401,16 +404,19 @@ class RayLauncher:
         ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
         while not connected:
             try:
-                # Add ssh key
-                ssh_key = getpass("Enter your cluster password: ")
+                if self.server_password is None:
+                    # Add ssh key
+                    self.server_password = getpass("Enter your cluster password: ")
+                
                 ssh_client.connect(
                     hostname=self.server_ssh,
                     username=self.server_username,
-                    password=ssh_key,
+                    password=self.server_password,
                 )
                 sftp = ssh_client.open_sftp()
                 connected = True
             except paramiko.ssh_exception.AuthenticationException:
+                self.server_password = None
                 print("Wrong password, please try again.")
 
         # Write server script
@@ -535,19 +541,20 @@ if __name__ == "__main__":
         return result
 
     launcher = RayLauncher(
-        project_name="example",
-        func=example_func,
-        args={"x": 1},
-        files=["slurmray/RayLauncher.py"],
-        modules=[],
-        node_nbr=1,
-        use_gpu=True,
-        memory=8,
-        max_running_time=5,
-        runtime_env={"env_vars": {"NCCL_SOCKET_IFNAME": "eno1"}},
-        server_run=True,
-        server_ssh="curnagl.dcsr.unil.ch",
-        server_username="hjamet",
+        project_name="example", # Name of the project (will create a directory with this name in the current directory)
+        func=example_func, # Function to execute
+        args={"x": 1}, # Arguments of the function
+        files=["slurmray/RayLauncher.py"], # List of files to push to the cluster (file path will be recreated on the cluster)
+        modules=[], # List of modules to load on the curnagl Cluster (CUDA & CUDNN are automatically added if use_gpu=True)
+        node_nbr=1, # Number of nodes to use
+        use_gpu=True, # If you need A100 GPU, you can set it to True
+        memory=8, # In MegaBytes
+        max_running_time=5, # In minutes
+        runtime_env={"env_vars": {"NCCL_SOCKET_IFNAME": "eno1"}}, # Example of environment variable
+        server_run=True, # To run the code on the cluster and not locally
+        server_ssh="curnagl.dcsr.unil.ch", # Address of the SLURM server
+        server_username="hjamet", # Username to connect to the server
+        server_password=None, # Will be asked in the terminal
     )
 
     result = launcher()

{slurmray-3.5.7 → slurmray-3.5.9}/slurmray/assets/slurmray_server_template.py

@@ -17,6 +17,6 @@ if __name__ == "__main__":
     )
     
     # Remove serialization
-    launcher.serialize_func_and_args = lambda *args, **kwargs : print("No serialization done.")
+    launcher.__serialize_func_and_args = lambda *args, **kwargs : print("No serialization done.")
 
     result = launcher()

slurmray-3.5.7/README.md

@@ -1,44 +0,0 @@
-# SLURM_RAY
-
-👉[Full documentation](https://henri-jamet.vercel.app/cards/documentation/slurm-ray/slurm-ray/)
-
-## Description
-
-**SlurmRay** is a module for effortlessly distributing tasks on a [Slurm](https://slurm.schedmd.com/) cluster using the [Ray](https://ray.io/) library. **SlurmRay** was initially designed to work with the [Curnagl](https://wiki.unil.ch/ci/books/high-performance-computing-hpc/page/curnagl) cluster at the *University of Lausanne*. However, it should be able to run on any [Slurm](https://slurm.schedmd.com/) cluster with a minimum of configuration.
-
-## Installation
-
-**SlurmRay** is designed to run both locally and on a cluster without any modification. This design is intended to allow work to be carried out on a local machine until the script seems to be working. It should then be possible to run it using all the resources of the cluster without having to modify the code.
-
-```bash
-pip install slurmray
-```
-
-## Usage
-
-```python
-from slurmray.RayLauncher import RayLauncher
-
-if __name__ == "__main__":
-    def example_func(x):
-        import ray # All packages and resources must be imported inside the function
-
-        return ray.cluster_resources(), x + 1
-
-    launcher = RayLauncher(
-        project_name="example",
-        func=example_func,
-        args={"x": 1},
-        modules=[],
-        node_nbr=1,
-        use_gpu=True,
-        memory=64,
-        max_running_time=15,
-        server_run=True,
-        server_ssh="curnagl.dcsr.unil.ch",
-        server_username="hjamet",
-    )
-
-    result = launcher()
-    print(result)
-```