cloudos-cli 2.32.1__tar.gz → 2.34.0__tar.gz

{cloudos_cli-2.32.1 → cloudos_cli-2.34.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.32.1
+Version: 2.34.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -420,6 +420,98 @@ command.
 Other options like `--wait-completion` are also available and work in the same way as for the `cloudos job run` command.
 Check `cloudos bash job --help` for more details.
 
+#### Send a bash array-job to CloudOS (parallel sample processing)
+
+When running a bash array job, the following options are available to customize the behavior:
+
+##### Array File
+- **`--array-file`**: Specifies the path to a file containing a set of columns useful in running the bash job. This option is **required** when using the command `bash array-job`.
+
+##### Separator
+- **`--separator`**: Defines the separator to use in the array file. Supported separators include:
+    - `,` (comma)
+    - `;` (semicolon)
+    - `tab`
+    - `space`
+    - `|` (pipe)
+This option is **required** when using the command `bash array-job`.
+
+##### List Columns
+- **`--list-columns`**: Lists the columns available in the array file. This is useful for inspecting the structure of the file. This flag disables sending the job, it just prints the column list, one per line:
+
+```console
+Columns:
+    - column1
+    - column2
+    - column3
+```
+
+##### Array File Project
+- **`--array-file-project`**: Specifies the name of the project in which the array file is placed, if it is different from the project specified by `--project-name`.
+
+##### Disable Column Check
+- **`--disable-column-check`**: Disables the validation of columns in the array file. This implies that each `--array-parameter` value is not checked against the header of the `--array-file`. For example, `--array-parameter --bar=foo`, without `--disable-column-check`, expects the array file to have column 'foo' inside the file header. If the column is not present, the CLI will throw an error. When `--disable-column-check` flag is added, the column check is not performed and the bash array job is sent to the platform.
+
+> [!NOTE]
+> Adding `--disable-column-check` will make the CLI command run without errors, but the errors might appear when checking the job in the platform, if the columns in the array file do not exists, as depicted with `--array-parameter`.
+
+##### Array Parameter
+- **`-a` / `--array-parameter`**: Allows specifying the column name present in the header of the array file. Each parameter should be in the format `arary_parameter_name=array_file_column`. For example:
+    - `-a --test=value` or
+    - `--array-parameter -test=value`
+specify a column named 'value' in the array file header. Adding array parameters not present in the header will cause an error. This option can be used multiple times to include as many array parameters as needed. This type of parameter is similar to `-p, --parameter`, both parameters can be interpolated in the bash array job command (either with `--command` or `--custom-script-path`), but this parameter can only be used to name the column present in the header of the array file.
+
+For example, the array file has the following header:
+
+```console
+id,bgen,csv
+1,s3://data/adipose.bgen,s3://data/adipose.csv
+2,s3://data/blood.bgen,s3://data/blood.csv
+3,s3://data/brain.bgen,s3://data/brain.csv
+...
+```
+and in the command there is need to go over the `bgen` column, this can be specified as `--array-parameter file=bgen`, refering to the column in the header.
+
+##### Custom Script Path
+- **`--custom-script-path`**: Specifies the path to a custom script to run in the bash array job instead of a command. When adding this command, parameter `--command` is ignored. To ensure the script runs successfully, you must either:
+
+1. Use a Shebang Line at the Top of the Script
+
+The shebang (#!) tells the system which interpreter to use to run the script. The path should match absolute path to python or other interpreter installed inside the docker container.
+
+Examples:
+`#!/usr/bin/python3` –-> for Python scripts
+`#!/usr/bin/Rscript` –-> for R scripts
+`#!/bin/bash`        –-> for Bash scripts
+
+Example Python Script:
+
+```python
+#!/usr/bin/python3
+print("Hello world")
+```
+ 
+2. Or use an interpreter command in the executable field
+
+If your script doesn’t have a shebang line, you can execute it by explicitly specifying the interpreter in the executable command:
+
+```console
+python my_script.py
+Rscript my_script.R
+bash my_script.sh
+```
+This assumes the interpreter is available on the container’s $PATH. If not, you can use the full absolute path instead:
+
+```console
+/usr/bin/python3 my_script.py
+/usr/local/bin/Rscript my_script.R
+```
+
+##### Custom Script Project
+- **`--custom-script-project`**: Specifies the name of the project in which the custom script is placed, if it is different from the project specified by `--project-name`.
+
+These options provide flexibility for configuring and running bash array jobs, allowing to tailor the execution for specific requirements.
+
 #### Get path to logs of job from CloudOS
 
 Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.
@@ -922,6 +1014,36 @@ Please, note that in the above example a preconfigured profile has been used. If
     --project-name $PROJEC_NAME
 ```
 
+#### Copying files and folders
+
+Files and folders can be copied **from** anywhere in the project **to** `Data` or any of its subfolders programmatically (i.e `Data`, `Data/folder/file.txt`).
+
+1. The copy can happen **within the same project** running the following command:
+```
+cloudos datasets cp <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets cp <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets cp AnalysesResults/my_analysis/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.32.1 → cloudos_cli-2.34.0}/README.md

@@ -385,6 +385,98 @@ command.
 Other options like `--wait-completion` are also available and work in the same way as for the `cloudos job run` command.
 Check `cloudos bash job --help` for more details.
 
+#### Send a bash array-job to CloudOS (parallel sample processing)
+
+When running a bash array job, the following options are available to customize the behavior:
+
+##### Array File
+- **`--array-file`**: Specifies the path to a file containing a set of columns useful in running the bash job. This option is **required** when using the command `bash array-job`.
+
+##### Separator
+- **`--separator`**: Defines the separator to use in the array file. Supported separators include:
+    - `,` (comma)
+    - `;` (semicolon)
+    - `tab`
+    - `space`
+    - `|` (pipe)
+This option is **required** when using the command `bash array-job`.
+
+##### List Columns
+- **`--list-columns`**: Lists the columns available in the array file. This is useful for inspecting the structure of the file. This flag disables sending the job, it just prints the column list, one per line:
+
+```console
+Columns:
+    - column1
+    - column2
+    - column3
+```
+
+##### Array File Project
+- **`--array-file-project`**: Specifies the name of the project in which the array file is placed, if it is different from the project specified by `--project-name`.
+
+##### Disable Column Check
+- **`--disable-column-check`**: Disables the validation of columns in the array file. This implies that each `--array-parameter` value is not checked against the header of the `--array-file`. For example, `--array-parameter --bar=foo`, without `--disable-column-check`, expects the array file to have column 'foo' inside the file header. If the column is not present, the CLI will throw an error. When `--disable-column-check` flag is added, the column check is not performed and the bash array job is sent to the platform.
+
+> [!NOTE]
+> Adding `--disable-column-check` will make the CLI command run without errors, but the errors might appear when checking the job in the platform, if the columns in the array file do not exists, as depicted with `--array-parameter`.
+
+##### Array Parameter
+- **`-a` / `--array-parameter`**: Allows specifying the column name present in the header of the array file. Each parameter should be in the format `arary_parameter_name=array_file_column`. For example:
+    - `-a --test=value` or
+    - `--array-parameter -test=value`
+specify a column named 'value' in the array file header. Adding array parameters not present in the header will cause an error. This option can be used multiple times to include as many array parameters as needed. This type of parameter is similar to `-p, --parameter`, both parameters can be interpolated in the bash array job command (either with `--command` or `--custom-script-path`), but this parameter can only be used to name the column present in the header of the array file.
+
+For example, the array file has the following header:
+
+```console
+id,bgen,csv
+1,s3://data/adipose.bgen,s3://data/adipose.csv
+2,s3://data/blood.bgen,s3://data/blood.csv
+3,s3://data/brain.bgen,s3://data/brain.csv
+...
+```
+and in the command there is need to go over the `bgen` column, this can be specified as `--array-parameter file=bgen`, refering to the column in the header.
+
+##### Custom Script Path
+- **`--custom-script-path`**: Specifies the path to a custom script to run in the bash array job instead of a command. When adding this command, parameter `--command` is ignored. To ensure the script runs successfully, you must either:
+
+1. Use a Shebang Line at the Top of the Script
+
+The shebang (#!) tells the system which interpreter to use to run the script. The path should match absolute path to python or other interpreter installed inside the docker container.
+
+Examples:
+`#!/usr/bin/python3` –-> for Python scripts
+`#!/usr/bin/Rscript` –-> for R scripts
+`#!/bin/bash`        –-> for Bash scripts
+
+Example Python Script:
+
+```python
+#!/usr/bin/python3
+print("Hello world")
+```
+ 
+2. Or use an interpreter command in the executable field
+
+If your script doesn’t have a shebang line, you can execute it by explicitly specifying the interpreter in the executable command:
+
+```console
+python my_script.py
+Rscript my_script.R
+bash my_script.sh
+```
+This assumes the interpreter is available on the container’s $PATH. If not, you can use the full absolute path instead:
+
+```console
+/usr/bin/python3 my_script.py
+/usr/local/bin/Rscript my_script.R
+```
+
+##### Custom Script Project
+- **`--custom-script-project`**: Specifies the name of the project in which the custom script is placed, if it is different from the project specified by `--project-name`.
+
+These options provide flexibility for configuring and running bash array jobs, allowing to tailor the execution for specific requirements.
+
 #### Get path to logs of job from CloudOS
 
 Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.
@@ -887,6 +979,36 @@ Please, note that in the above example a preconfigured profile has been used. If
     --project-name $PROJEC_NAME
 ```
 
+#### Copying files and folders
+
+Files and folders can be copied **from** anywhere in the project **to** `Data` or any of its subfolders programmatically (i.e `Data`, `Data/folder/file.txt`).
+
+1. The copy can happen **within the same project** running the following command:
+```
+cloudos datasets cp <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets cp <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets cp AnalysesResults/my_analysis/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
 ### WDL pipeline support
 
 #### Cromwell server managing