yellowdog-python-examples 7.2.5__tar.gz → 7.2.7__tar.gz

{yellowdog-python-examples-7.2.5/yellowdog_python_examples.egg-info → yellowdog-python-examples-7.2.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: yellowdog-python-examples
-Version: 7.2.5
+Version: 7.2.7
 Summary: Example Python commands using the YellowDog Python SDK
 Home-page: https://github.com/yellowdog/python-examples
 Author: YellowDog Limited

{yellowdog-python-examples-7.2.5 → yellowdog-python-examples-7.2.7}/README.md

@@ -18,6 +18,7 @@
    * [User-Defined Variables](#user-defined-variables)
    * [Nested Variables](#nested-variables)
    * [Providing Default Values for User-Defined Variables](#providing-default-values-for-user-defined-variables)
+   * [Variable Substitutions in Worker Pool and Compute Requirement Specifications, and in User Data](#variable-substitutions-in-worker-pool-and-compute-requirement-specifications-and-in-user-data)
 * [Work Requirement Properties](#work-requirement-properties)
    * [Work Requirement JSON File Structure](#work-requirement-json-file-structure)
    * [Property Inheritance](#property-inheritance)
@@ -107,7 +108,7 @@
    * [yd-remove](#yd-remove)
    * [yd-follow](#yd-follow)
 
-<!-- Added by: pwt, at: Wed Sep 13 15:24:39 BST 2023 -->
+<!-- Added by: pwt, at: Mon Sep 18 16:03:18 BST 2023 -->
 
 <!--te-->
 
@@ -335,11 +336,17 @@ Variable substitutions provide a powerful way of introducing variable values int
 
 Variable substitutions are expressed using `{{variable}}` notation, where the expression is replaced by the value of `variable`.
 
-Substitutions can also be performed for non-string (number and boolean) values using the `num:` and `bool:` prefixes within the variable substitution:
+Substitutions can also be performed for non-string (number, boolean, array, and table) values using the `num:`, `bool:`, `array:`, and `table:` prefixes within the variable substitution:
 
-- Define the variable substitution using one of the following patterns: `"{{num:my_int}}"`, `"{{num:my_float}}"`, `"{{bool:my_bool}}"`
-- Variable definitions supplied on the command line would then be of the form: `-m my_int=5 -m my_float=2.5 -m my_bool=true`
-- In the processed JSON or TOML, these values would become `5`, `2.5` and `true`, respectively, converted from strings to their correct JSON types
+- Define the variable substitution using one of the following patterns: `"{{num:my_int}}"`, `"{{num:my_float}}"`, `"{{bool:my_bool}}"`, `"{{array:my_array}}"`, `"{{table:my_table}}"`
+- Variable definitions supplied on the command line would then be of the form, e.g.: 
+
+```shell
+ yd-submit -v my_int=5 -v my_float=2.5 -v my_bool=true \
+           -v my_array="[1,2,3]" -v my_table="{'A': 100, 'B': 200}"
+```
+
+- In the processed JSON (or TOML), these values would become `5`, `2.5`, `true`, `[1,2,3]`, and `{"A": 100, "B": 200}`, respectively, converted from strings to their correct JSON types
 
 ## Default Variables
 
@@ -406,10 +413,12 @@ Nesting can be up to three levels deep including the top level.
 
 Each variable can be supplied with a default value to be used if a value is not provided for that variable name. The syntax for providing a default is:
 
-```shell
+```
 {{variable_name:=default_value}} or
 {{num:numeric_variable_name:=default_numeric_value}} or
-{{bool:boolean_variable_name:=default_boolean_value}}
+{{bool:boolean_variable_name:=default_boolean_value}} or
+{{array:array_name:=default_array}} or
+{{table:table_name:=default_table}}
 ```
 
 Examples of use in a TOML file:
@@ -418,10 +427,24 @@ Examples of use in a TOML file:
 name = "{{name:=my_name}}"
 taskCount = "{{num:task_count:=5}}"
 finishIfAllTasksFinished = "{{bool:fiaft:=true}}"
+arguments = "{{array:args:=[1,2,3]}}"
+environment = "{{table:env:={'A':100,'B':200}}}"
 ```
 
 Default values can be used anywhere that variable substitutions are allowed.
 
+In TOML files only, nested variable substitutions can be used inside default values, e.g.:
+
+```toml
+name = "{{name_var:={{tag}}-{{datetime}}}}"
+```
+
+## Variable Substitutions in Worker Pool and Compute Requirement Specifications, and in User Data
+
+In JSON specifications for Worker Pools and Compute Requirements, variable substitutions can be used, but they must be prefixed and postfixed by a double underscore `__`, e.g., `__{{username}}__`. This is to disambiguate them from variable substitutions intended for Mustache processing at the platform end.
+
+Variable substitutions can also be used within **User Data** to be supplied to instances, for which the same prefix/postfix requirement applies, **including** for User Data supplied directly using the `userData` property in the `workerPool` section of the TOML file.
+
 # Work Requirement Properties
 
 The `workRequirement` section of the configuration file is optional. It's used only by the `yd-submit` command, and controls the Work Requirement that is submitted to the Platform.
@@ -480,54 +503,55 @@ The following table outlines all the properties available for defining Work Requ
 
 All properties are optional except for **`taskType`** (or **`taskTypes`**).
 
-| Property Name              | Description                                                                                                                                                              | TOML | WR  | TGrp | Task |
-|:---------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----|:----|:-----|:-----|
-| `alwaysUpload`             | Whether to attempt to upload task outputs on failure. Default: `true`.                                                                                                   | Yes  | Yes | Yes  | Yes  |
-| `arguments`                | The list of arguments to be passed to the Task when it is executed. E.g.: `[1, "Two"]`.                                                                                  | Yes  | Yes | Yes  | Yes  |
-| `captureTaskOutput`        | Whether the console output of a Task's process should be uploaded to the YellowDog Object Store on Task completion. Default: `true`.                                     | Yes  | Yes | Yes  | Yes  |
-| `completedTaskTtl`         | The time (in minutes) to live for completed Tasks. If set, Tasks that have been completed for longer than this period will be deleted. E.g.: `10.0`.                     | Yes  | Yes | Yes  |      |
-| `csvFile`                  | The name of the CSV file used to derive Task data. An alternative to `csvFiles` that can be used when there's only a single CSV file. E.g. `"file.csv"`.                 | Yes  |     |      |      |
-| `csvFiles`                 | A list of CSV files used to derive Task data. E.g. `["file.csv", "file_2.csv:2]`.                                                                                        | Yes  |     |      |      |
-| `dependentOn`              | The name of another Task Group within the same Work Requirement that must be successfully completed before the Task Group is started. E.g. `"task_group_1"`.             |      |     | Yes  |      |
-| `dockerEnvironment`        | The environment to be passed to a Docker container. Only used by the `docker` Task Type. E.g., JSON: `{"VAR_1": "abc"}`, TOML: `{VAR_1 = "abc", VAR_2 = "def"}`.         | Yes  | Yes | Yes  | Yes  |
-| `dockerPassword`           | The password for DockerHub, only used by the `docker` Task Type. E,g., `"my_password"`.                                                                                  | Yes  | Yes | Yes  | Yes  |
-| `dockerUsername`           | The username for DockerHub, only used by the `docker` Task Type. E,g., `"my_username"`.                                                                                  | Yes  | Yes | Yes  | Yes  |
-| `environment`              | The environment variables to set for a Task when it's executed. E.g., JSON: `{"VAR_1": "abc", "VAR_2": "def"}`, TOML: `{VAR_1 = "abc", VAR_2 = "def"}`.                  | Yes  | Yes | Yes  | Yes  |
-| `exclusiveWorkers`         | If true, then do not allow claimed Workers to be shared with other Task Groups; otherwise, Workers can be shared. Default:`false`.                                       | Yes  | Yes | Yes  |      |
-| `executable`               | The 'executable' to run when a Bash or Docker Task is executed. Bash script for Bash, container image for Docker. Optional: omit to suppress automatic processing.       | Yes  | Yes | Yes  | Yes  |
-| `finishIfAllTasksFinished` | If true, the Task Group will finish automatically if all contained tasks finish. Default:`true`.                                                                         | Yes  | Yes | Yes  |      |
-| `finishIfAnyTaskFailed`    | If true, the Task Group will be failed automatically if any contained tasks fail. Default:`false`.                                                                       | Yes  | Yes | Yes  |      |
-| `flattenInputPaths`        | Determines whether input object paths should be flattened (i.e., directory structure removed) when downloaded to a node. Default: `false`.                               | Yes  | Yes | Yes  | Yes  |
-| `flattenUploadPaths`       | Ignore local directory paths when uploading files to the Object Store; place in `<namespace>:<work-req-name>/`. Default: `false`.                                        | Yes  | Yes |      |      |
-| `fulfilOnSubmit`           | Indicates if the Work Requirement should be fulfilled when it is submitted, rather than being allowed to wait in PENDING status. Default:`false`.                        | Yes  | Yes |      |      |
-| `inputs`                   | The list of input files to be uploaded to the YellowDog Object Store, and required by the Task (implies `verifyAtStart`). E.g. `["a.sh", "b.sh"]` or `["*.sh"]`.         | Yes  | Yes | Yes  | Yes  |
-| `inputsOptional`           | A list of input files required by a Task, but which are not subject to verification. Can contain wildcards. E.g.: `["task_group_1/**/results.txt"]`.                     | Yes  | Yes | Yes  | Yes  |
-| `instanceTypes`            | The machine instance types that can be used to execute Tasks. E.g., `["t3.micro", "t3a.micro"]`.                                                                         | Yes  | Yes | Yes  |      |
-| `maximumTaskRetries`       | The maximum number of times a Task can be retried after it has failed. E.g.: `5`.                                                                                        | Yes  | Yes | Yes  |      |
-| `maxWorkers`               | The maximum number of Workers that can be claimed for the associated Task Group. E.g., `10`.                                                                             | Yes  | Yes | Yes  |      |
-| `minWorkers`               | The minimum number of Workers that the associated Task Group requires. This many workers must be claimed before the associated Task Group will start working. E.g., `1`. | Yes  | Yes | Yes  |      |
-| `name`                     | The name of the Work Requirement, Task Group or Task. E.g., `"wr_name"`. Note that the `name` property is not inherited.                                                 | Yes  | Yes | Yes  | Yes  |
-| `outputs`                  | The files to be uploaded to the YellowDog Object Store by a Worker node on completion of the Task. E.g., `["results_1.txt", "results_2.txt"]`.                           | Yes  | Yes | Yes  | Yes  |
-| `outputsRequired`          | The files that *must* be uploaded to the YellowDog Object Store by a Worker node on completion of the Task. The Task will fail if any outputs are unavailable.           | Yes  | Yes | Yes  | Yes  |
-| `priority`                 | The priority of Work Requirements and Task Groups. Higher priority acquires Workers ahead of lower priority. E.g., `0.0`.                                                | Yes  | Yes | Yes  |      |
-| `providers`                | Constrains the YellowDog Scheduler only to execute tasks from the associated Task Group on the specified providers. E.g., `["AWS", "GOOGLE"]`.                           | Yes  | Yes | Yes  |      |
-| `ram`                      | Range constraint on GB of RAM that are required to execute Tasks. E.g., `[2.5, 4.0]`.                                                                                    | Yes  | Yes | Yes  |      |
-| `regions`                  | Constrains the YellowDog Scheduler only to execute Tasks from the associated Task Group in the specified regions. E.g., `["eu-west-2]`.                                  | Yes  | Yes | Yes  |      |
-| `taskBatchSize`            | Determines the batch size used to add Tasks to Task Groups. Default is 2,000.                                                                                            | Yes  |     |      |      |
-| `taskCount`                | The number of times to execute the Task. Can be set in the TOML file or in any JSON Task Group definition. Note: no inheritance from TOML to JSON.                       | Yes  |     | Yes  |      |
-| `taskData`                 | The data to be passed to the Worker when the Task is started. E.g., `"mydata"`. Becomes file `taskdata.txt` in the Task's working directory when The Task executes.      | Yes  | Yes | Yes  | Yes  |
-| `taskDataFile`             | Populate the `taskData` property above with the contents of the specified file. E.g., `"my_task_data_file.txt"`.                                                         | Yes  | Yes | Yes  | Yes  |
-| `taskName`                 | The name to use for the Task. Only usable in the TOML file. Mostly useful in conjunction with CSV Task data. E.g., `"my_task_number_{{task_number}}"`.                   | Yes  |     |      |      |
-| `taskGroupName`            | The name to use for the Task Group. Only usable in the TOML file. E.g., `"my_tg_number_{{task_group_number}}"`.                                                          | Yes  |     |      |      |
-| `taskType`                 | The Task Type of a Task. E.g., `"docker"`.                                                                                                                               | Yes  |     |      | Yes  |
-| `taskTypes`                | The list of Task Types required by the range of Tasks in a Task Group. E.g., `["docker", bash"]`.                                                                        |      | Yes | Yes  |      |
-| `tasksPerWorker`           | Determines the number of Worker claims based on splitting the number of unfinished Tasks across Workers. E.g., `1`.                                                      | Yes  | Yes | Yes  |      |
-| `uploadFiles`              | The list of files to be uploaded to the YellowDog Object Store. E.g., (JSON): `[{"localPath": file_1.txt", "uploadPath": "file_1.txt"}]`.                                | Yes  | Yes | Yes  | Yes  |
-| `vcpus`                    | Range constraint on number of vCPUs that are required to execute Tasks E.g., `[2.0, 4.0]`.                                                                               | Yes  | Yes | Yes  |      |
-| `verifyAtStart`            | A list of files required by a Task. Must be present when the Task is ready to start or the Task will fail. E.g.: `["task_group_1/task_1/results.txt"]`.                  | Yes  | Yes | Yes  | Yes  |
-| `verifyWait`               | A list of files required by a Task. The Task will wait until the files are available before starting. E.g.: `["task_group_1/task_1/results.txt"]`.                       | Yes  | Yes | Yes  | Yes  |
-| `workerTags`               | The list of Worker Tags that will be used to match against the Worker Tag of a candidate Worker. E.g., `["tag_x", "tag_y"]`.                                             | Yes  | Yes | Yes  |      |
-| `workRequirementData`      | The name of the file containing the JSON document in which the Work Requirement is defined. E.g., `"test_workreq.json"`.                                                 | Yes  |     |      |      |
+| Property Name              | Description                                                                                                                                                                                                                                | TOML | WR  | TGrp | Task |
+|:---------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----|:----|:-----|:-----|
+| `alwaysUpload`             | Whether to attempt to upload task outputs on failure. Default: `true`.                                                                                                                                                                     | Yes  | Yes | Yes  | Yes  |
+| `arguments`                | The list of arguments to be passed to the Task when it is executed. E.g.: `[1, "Two"]`.                                                                                                                                                    | Yes  | Yes | Yes  | Yes  |
+| `captureTaskOutput`        | Whether the console output of a Task's process should be uploaded to the YellowDog Object Store on Task completion. Default: `true`.                                                                                                       | Yes  | Yes | Yes  | Yes  |
+| `completedTaskTtl`         | The time (in minutes) to live for completed Tasks. If set, Tasks that have been completed for longer than this period will be deleted. E.g.: `10.0`.                                                                                       | Yes  | Yes | Yes  |      |
+| `csvFile`                  | The name of the CSV file used to derive Task data. An alternative to `csvFiles` that can be used when there's only a single CSV file. E.g. `"file.csv"`.                                                                                   | Yes  |     |      |      |
+| `csvFiles`                 | A list of CSV files used to derive Task data. E.g. `["file.csv", "file_2.csv:2]`.                                                                                                                                                          | Yes  |     |      |      |
+| `dependentOn`              | The name of another Task Group within the same Work Requirement that must be successfully completed before the Task Group is started. E.g. `"task_group_1"`.                                                                               |      |     | Yes  |      |
+| `dockerEnvironment`        | The environment to be passed to a Docker container. Only used by the `docker` Task Type. E.g., JSON: `{"VAR_1": "abc"}`, TOML: `{VAR_1 = "abc", VAR_2 = "def"}`.                                                                           | Yes  | Yes | Yes  | Yes  |
+| `dockerPassword`           | The password for DockerHub, only used by the `docker` Task Type. E,g., `"my_password"`.                                                                                                                                                    | Yes  | Yes | Yes  | Yes  |
+| `dockerUsername`           | The username for DockerHub, only used by the `docker` Task Type. E,g., `"my_username"`.                                                                                                                                                    | Yes  | Yes | Yes  | Yes  |
+| `environment`              | The environment variables to set for a Task when it's executed. E.g., JSON: `{"VAR_1": "abc", "VAR_2": "def"}`, TOML: `{VAR_1 = "abc", VAR_2 = "def"}`.                                                                                    | Yes  | Yes | Yes  | Yes  |
+| `exclusiveWorkers`         | If true, then do not allow claimed Workers to be shared with other Task Groups; otherwise, Workers can be shared. Default:`false`.                                                                                                         | Yes  | Yes | Yes  |      |
+| `executable`               | The 'executable' to run when a Bash or Docker Task is executed. Bash script for Bash, container image for Docker. Optional: omit to suppress automatic processing.                                                                         | Yes  | Yes | Yes  | Yes  |
+| `finishIfAllTasksFinished` | If true, the Task Group will finish automatically if all contained tasks finish. Default:`true`.                                                                                                                                           | Yes  | Yes | Yes  |      |
+| `finishIfAnyTaskFailed`    | If true, the Task Group will be failed automatically if any contained tasks fail. Default:`false`.                                                                                                                                         | Yes  | Yes | Yes  |      |
+| `flattenInputPaths`        | Determines whether input object paths should be flattened (i.e., directory structure removed) when downloaded to a node. Default: `false`.                                                                                                 | Yes  | Yes | Yes  | Yes  |
+| `flattenUploadPaths`       | Ignore local directory paths when uploading files to the Object Store; place in `<namespace>:<work-req-name>/`. Default: `false`.                                                                                                          | Yes  | Yes |      |      |
+| `fulfilOnSubmit`           | Indicates if the Work Requirement should be fulfilled when it is submitted, rather than being allowed to wait in PENDING status. Default:`false`.                                                                                          | Yes  | Yes |      |      |
+| `inputs`                   | The list of input files to be uploaded to the YellowDog Object Store, and required by the Task (implies `verifyAtStart`). E.g. `["a.sh", "b.sh"]` or `["*.sh"]`.                                                                           | Yes  | Yes | Yes  | Yes  |
+| `inputsOptional`           | A list of input files required by a Task, but which are not subject to verification. Can contain wildcards. E.g.: `["task_group_1/**/results.txt"]`.                                                                                       | Yes  | Yes | Yes  | Yes  |
+| `instanceTypes`            | The machine instance types that can be used to execute Tasks. E.g., `["t3.micro", "t3a.micro"]`.                                                                                                                                           | Yes  | Yes | Yes  |      |
+| `maximumTaskRetries`       | The maximum number of times a Task can be retried after it has failed. E.g.: `5`.                                                                                                                                                          | Yes  | Yes | Yes  |      |
+| `maxWorkers`               | The maximum number of Workers that can be claimed for the associated Task Group. E.g., `10`.                                                                                                                                               | Yes  | Yes | Yes  |      |
+| `minWorkers`               | The minimum number of Workers that the associated Task Group requires. This many workers must be claimed before the associated Task Group will start working. E.g., `1`.                                                                   | Yes  | Yes | Yes  |      |
+| `name`                     | The name of the Work Requirement, Task Group or Task. E.g., `"wr_name"`. Note that the `name` property is not inherited.                                                                                                                   | Yes  | Yes | Yes  | Yes  |
+| `outputs`                  | The files to be uploaded to the YellowDog Object Store by a Worker node on completion of the Task. E.g., `["results_1.txt", "results_2.txt"]`.                                                                                             | Yes  | Yes | Yes  | Yes  |
+| `outputsOther`             | Files to be uploaded to the YellowDog Object Store from outside the Tasks's Working Directory by a Worker node on completion of a Task. E.g., `outputsOther = [{"directoryName" = "tmp", "filePattern" = "out.txt", "required" = false}]`. | Yes  | Yes | Yes  | Yes  |
+| `outputsRequired`          | The files that *must* be uploaded to the YellowDog Object Store by a Worker node on completion of the Task. The Task will fail if any outputs are unavailable.                                                                             | Yes  | Yes | Yes  | Yes  |
+| `priority`                 | The priority of Work Requirements and Task Groups. Higher priority acquires Workers ahead of lower priority. E.g., `0.0`.                                                                                                                  | Yes  | Yes | Yes  |      |
+| `providers`                | Constrains the YellowDog Scheduler only to execute tasks from the associated Task Group on the specified providers. E.g., `["AWS", "GOOGLE"]`.                                                                                             | Yes  | Yes | Yes  |      |
+| `ram`                      | Range constraint on GB of RAM that are required to execute Tasks. E.g., `[2.5, 4.0]`.                                                                                                                                                      | Yes  | Yes | Yes  |      |
+| `regions`                  | Constrains the YellowDog Scheduler only to execute Tasks from the associated Task Group in the specified regions. E.g., `["eu-west-2]`.                                                                                                    | Yes  | Yes | Yes  |      |
+| `taskBatchSize`            | Determines the batch size used to add Tasks to Task Groups. Default is 2,000.                                                                                                                                                              | Yes  |     |      |      |
+| `taskCount`                | The number of times to execute the Task.                                                                                                                                                                                                   | Yes  | Yes | Yes  |      |
+| `taskData`                 | The data to be passed to the Worker when the Task is started. E.g., `"mydata"`. Becomes file `taskdata.txt` in the Task's working directory when The Task executes.                                                                        | Yes  | Yes | Yes  | Yes  |
+| `taskDataFile`             | Populate the `taskData` property above with the contents of the specified file. E.g., `"my_task_data_file.txt"`.                                                                                                                           | Yes  | Yes | Yes  | Yes  |
+| `taskName`                 | The name to use for the Task. Only usable in the TOML file. Mostly useful in conjunction with CSV Task data. E.g., `"my_task_number_{{task_number}}"`.                                                                                     | Yes  |     |      |      |
+| `taskGroupName`            | The name to use for the Task Group. Only usable in the TOML file. E.g., `"my_tg_number_{{task_group_number}}"`.                                                                                                                            | Yes  |     |      |      |
+| `taskType`                 | The Task Type of a Task. E.g., `"docker"`.                                                                                                                                                                                                 | Yes  |     |      | Yes  |
+| `taskTypes`                | The list of Task Types required by the range of Tasks in a Task Group. E.g., `["docker", bash"]`.                                                                                                                                          |      | Yes | Yes  |      |
+| `tasksPerWorker`           | Determines the number of Worker claims based on splitting the number of unfinished Tasks across Workers. E.g., `1`.                                                                                                                        | Yes  | Yes | Yes  |      |
+| `uploadFiles`              | The list of files to be uploaded to the YellowDog Object Store. E.g., (JSON): `[{"localPath": file_1.txt", "uploadPath": "file_1.txt"}]`.                                                                                                  | Yes  | Yes | Yes  | Yes  |
+| `vcpus`                    | Range constraint on number of vCPUs that are required to execute Tasks E.g., `[2.0, 4.0]`.                                                                                                                                                 | Yes  | Yes | Yes  |      |
+| `verifyAtStart`            | A list of files required by a Task. Must be present when the Task is ready to start or the Task will fail. E.g.: `["task_group_1/task_1/results.txt"]`.                                                                                    | Yes  | Yes | Yes  | Yes  |
+| `verifyWait`               | A list of files required by a Task. The Task will wait until the files are available before starting. E.g.: `["task_group_1/task_1/results.txt"]`.                                                                                         | Yes  | Yes | Yes  | Yes  |
+| `workerTags`               | The list of Worker Tags that will be used to match against the Worker Tag of a candidate Worker. E.g., `["tag_x", "tag_y"]`.                                                                                                               | Yes  | Yes | Yes  |      |
+| `workRequirementData`      | The name of the file containing the JSON document in which the Work Requirement is defined. E.g., `"test_workreq.json"`.                                                                                                                   | Yes  |     |      |      |
 
 ## Automatic Properties
 
@@ -618,13 +642,9 @@ If the `executable` property is not supplied, the automatic processing described
 
 ### Task Counts
 
-This property will expand the number of Tasks to match `taskCount`.
-
-The `taskCount` property can be set only in the `workRequirement` section of the `config.toml` file, or in the `taskGroup` section(s) of a JSON Work Requirement definition.
-
-In the former case, the `taskCount` applies only to the Task specified within the `config.toml` file and is not inherited by JSON Work Requirement specifications.
+The `taskCount` property can be used to expand the number of Tasks within a Task Group, by creating duplicates of a single Task. In JSON specifications, there must be zero or one Task(s) listed within each Task Group or `taskCount` is ignored.
 
-In the latter case, the `taskCount` applies to the Task specified within the Task Group, and there must be zero or one Task(s) listed within the group or `taskCount` is ignored.
+This property can also be set on the command line using the `--task-count`/`-C` option of `yd-submit` followed by the required number of Tasks.
 
 ## Examples
 
@@ -662,6 +682,7 @@ Here's an example of the `workRequirement` section of a TOML configuration file,
     minWorkers = 1
     name = "my-work-requirement"
     outputs = ["results.txt"]
+    outputsOther = [{"directoryName" = "my_output_dir", "filePattern" = "out.txt", "required" = true}]
     outputsRequired = ["results_required.txt"]
     priority = 0.0
     providers = ["AWS"]
@@ -712,11 +733,13 @@ Showing all possible properties at the Work Requirement level:
   "minWorkers": 1,
   "name": "my-work-requirement",
   "outputs": ["results.txt"],
+  "outputsOther": [{"directoryName": "my_output_dir", "filePattern": "out.txt", "required": true}],
   "outputsRequired": ["results_required.txt"],
   "priority": 0.0,
   "providers": ["AWS"],
   "ram": [0.5, 2],
   "regions": ["eu-west-2"],
+  "taskCount": 100,
   "taskData": "my_task_data_string",
   "taskDataFile": "my_data_file.txt",
   "taskTypes": ["docker"],
@@ -766,6 +789,7 @@ Showing all possible properties at the Task Group level:
       "minWorkers": 1,
       "name": "first-task-group",
       "outputs": ["results.txt"],
+      "outputsOther": [{"directoryName": "my_output_dir", "filePattern": "out.txt", "required": true}],
       "outputsRequired": ["results_required.txt"],
       "priority": 0.0,
       "providers": ["AWS"],
@@ -819,6 +843,7 @@ Showing all possible properties at the Task level:
           "inputsOptional": ["optional.txt"],
           "name": "my-task",
           "outputs": ["results.txt"],
+          "outputsOther": [{"directoryName": "my_output_dir", "filePattern": "out.txt", "required": true}],
           "outputsRequired": ["results_required.txt"],
           "taskData": "my_task_data_string",
           "taskDataFile": "my_data_file.txt",
@@ -1148,7 +1173,7 @@ The Work Requirement name would then be available to the Task in the environment
 
 ### Files Uploaded from a Node to the Object Store after Task Execution
 
-After Task completion, the Agent will upload specified output files to the Object Store. The files to be uploaded are those listed in the `outputs` and `outputsRequired` properties for the Task.
+After Task completion, the Agent will upload specified output files to the Object Store. The files to be uploaded are those listed in the `outputs`, `outputsRequired`, and `outputsOther` properties for the Task.
 
 In addition, the console output of the Task is captured in a file called `taskoutput.txt` in the root of the Task's working directory. Whether the `taskoutput.txt` file is uploaded to the Object Store is determined by the `captureTaskOutput` property for the Task, and this is set to 'true' by default.
 
@@ -1168,6 +1193,20 @@ The **`outputsRequired`** property can be used instead of (or in addition to) th
 
 `"outputsRequired": ["results/process_output.txt"]`
 
+The **`outputsOther`** property is used to collect outputs from directories that are not contained under the Task's working directory. In this case, the YellowDog Agent must be explicitly configured to allow upload from these directories by establishing this in the `application.yaml` file. For example:
+
+```yaml
+yda.outputSources:
+  - name: "my_output_dir"
+    path: "/tmp/outputs"
+```
+
+Then, in the list of entries in the `outputsOther` property, the `directoryName` property is set to be the **`name`** specified in the `application.yaml`. For example:
+
+```json
+"outputsOther": [{"directoryName": "my_output_dir", "filePattern": "out.txt", "required": true}]
+```
+
 ### Files Downloaded from the Object Store to Local Storage
 
 The `yd-download` command can download all objects from the Object Store to a local directory, on a per Work Requirement basis (including any files that have been uploaded). A local directory is created with the same name as the Namespace and containing the Work Requirement directories.
@@ -1212,7 +1251,7 @@ When a Task is allocated to a Worker on a node by the YellowDog Scheduler, the f
 4. The Agent then gathers any files in the `outputs` and `outputsRequired` lists and uploads them to the Object Store. If a file in the `outputsRequired` list is not found, the Task will be reported as failed. The Agent will also optionally upload the console output (including both `stdout` and `stderr`) of the Task, contained in the `taskoutput.txt` file.
 5. The ephemeral Task directory is then deleted.
 
-Note that if a Task is aborted during execution, the Task's subprocess is sent a `SIGINT`, allowing the Task an opportunity to terminate any child processes or other resources (e.g., containers) that may have been started as part of Task execution.
+Note that if a Task is aborted during execution, the Task's subprocess is sent a `SIGTERM`, allowing the Task an opportunity to terminate any child processes or other resources (e.g., containers) that may have been started as part of Task execution.
 
 Once the steps above have been completed, the Worker is ready to accept its next Task from the YellowDog scheduler.
 
@@ -1337,6 +1376,8 @@ All variable substitutions unrelated to the CSV file data are left unchanged, fo
 
 If the value to be inserted is a number (an integer or floating point value) or Boolean, the `{{num:my_number_var}}` and `{{bool:my_boolean_var}}` forms can be used in the JSON file, as with their use in other parts of the JSON Work Requirement specification. The substituted value will assume the nominated type rather than being a string.
 
+The same is true for `array:` and `table:` for their respective data structures.
+
 ### Property Inheritance
 
 All the usual property inheritance features operate as normal. Properties are inherited from the `config.toml` file, and from the relevant sections of the JSON Work Requirement file. Any properties set within a Task prototype are copied to all the generated Tasks.
@@ -1626,7 +1667,7 @@ When a JSON Worker Pool specification is used, the following properties from the
 
 Variable substitutions can be used within any property value in TOML configuration files or Worker Pool JSON files. See the description [above](#variable-substitutions) for more details on variable substitutions. This is a powerful feature that allows Worker Pools to be parameterised by supplying values on the command line, via environment variables, or via the TOML file.
 
-An important distinction when using variable substitutions within Worker Pool (or Compute Requirement) JSON (or Jsonnet) documents is that each variable directive **must be preceded by a `__` (double underscore)** to disambiguate it from variable substitutions that are to be passed directly to the API. For example, use: `__{{username}}` to apply a substitution for the `username` default substitution.
+An important distinction when using variable substitutions within Worker Pool (or Compute Requirement) JSON (or Jsonnet) documents is that each variable directive **must be prefixed and postfixed by a `__` (double underscore)** to disambiguate it from variable substitutions that are to be passed directly to the API. For example, use: `__{{username}}__` to apply a substitution for the `username` default substitution.
 
 ## Dry-Running Worker Pool Provisioning
 
@@ -2008,7 +2049,7 @@ Please get in touch with YellowDog if you get stuck.
 
 ## Variable Substitutions in Jsonnet Files
 
-The scripts provide full support for variable substitutions in Jsonnet files, using the same rules as for the JSON specifications. Remember that for **Worker Pool** and **Compute Requirement** specifications, variable substitutions must be prefixed by `__`, e.g. `"__{{username}}}"`.
+The scripts provide full support for variable substitutions in Jsonnet files, using the same rules as for the JSON specifications. Remember that for **Worker Pool** and **Compute Requirement** specifications, variable substitutions must be prefixed and postfixed by `__`, e.g. `"__{{username}}}__"`.
 
 Variable substitution is performed before Jsonnet expansion into JSON, and again after the expansion.
 
@@ -2293,15 +2334,17 @@ The `yd-instantiate` command instantiates a Compute Requirement (i.e., a set of
 
 This command uses the data from the `workerPool` configuration section (or, synonymously, the `computeRequirement` section), but only uses the `name`, `templateId`, `targetInstanceCount`, `instanceTags`, `userData`, and `imagesId` properties. In addition, the Boolean property `maintainInstanceCount` (default = `false`) is available for use with `yd-instantiate`.
 
-Variable substitutions must be preceeded by a double underscore (`__`), e.g.: `"__{{my_variable}}"`.
+Compute Requirements can be instantiated directly from JSON (or Jsonnet) specifications, using the `--compute-requirement` (or `-C`) command line option, followed by the filename, or by using the `computeRequirementData` property in the `workerPool`/`computeRequirement` section. The properties listed above will be inherited from the config.toml `workerPool` specification if they are not present in the JSON file.
+
+Variable substitutions must be prefixed and postfixed by a double underscore (`__`), e.g.: `"__{{my_variable}}__"`.
 
-Compute Requirements can be instantiated directly from JSON (or Jsonnet) specifications, using the `--compute-requirement` (or `-C`) command line option, followed by the filename, or by using the `computeRequirementData` property in the `workerPool`/`computeRequirement` section. The properties listed above will be inherited from the config.toml `workerPool` specification if they are not present in the JSON file. An example JSON specification is shown below:
+An example JSON specification is shown below:
 
 ```json
 {
   "imagesId": "ydid:imgfam:000000:41962592-577c-4fde-ab03-d852465e7f8b",
   "instanceTags": {"a1": "one", "a2": "two"},
-  "requirementName": "cr_test_{{datetime}}",
+  "requirementName": "cr_test___{{datetime}}__",
   "requirementNamespace": "pyexamples",
   "requirementTag": "pyexamples-test",
   "templateId": "ydid:crt:000000:230e9a42-97db-4d69-aa91-29ff309951b4",

yellowdog-python-examples-7.2.7/tests/test_dryruns.py

@@ -0,0 +1,50 @@
+"""
+Tests that run the standard demo dry-runs.
+"""
+
+from cli_test_helpers import shell
+
+DEMO_DIR = "../python-examples-demos"
+CMD_SEQ = "yd-provision -D && yd-submit -D"
+
+
+class TestDemoDryRuns:
+    def test_bash(self):
+        result = shell(f"cd {DEMO_DIR}/bash && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_primes(self):
+        result = shell(f"cd {DEMO_DIR}/primes && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_image_montage(self):
+        result = shell(f"cd {DEMO_DIR}/image-montage && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_openfoam(self):
+        result = shell(f"cd {DEMO_DIR}/openfoam && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_slurm_cluster(self):
+        result = shell(f"cd {DEMO_DIR}/slurm-cluster && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_common_factors(self):
+        result = shell(f"cd {DEMO_DIR}/common-factors-csv && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_virtual_screening(self):
+        result = shell(f"cd {DEMO_DIR}/virtual-screening && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_benchmark(self):
+        result = shell(f"cd {DEMO_DIR}/benchmark && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_powershell(self):
+        result = shell(f"cd {DEMO_DIR}/powershell && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_cmd_exe(self):
+        result = shell(f"cd {DEMO_DIR}/cmd.exe && {CMD_SEQ}")
+        assert result.exit_code == 0

yellowdog-python-examples-7.2.7/yd_commands/__init__.py

@@ -0,0 +1 @@
+__version__ = "7.2.7"

{yellowdog-python-examples-7.2.5 → yellowdog-python-examples-7.2.7}/yd_commands/cancel.py

@@ -24,11 +24,10 @@ from yd_commands.object_utilities import (
     get_work_requirement_summary_by_name_or_id,
 )
 from yd_commands.printing import print_error, print_log
+from yd_commands.settings import TASK_ABORT_CHECK_INTERVAL
 from yd_commands.utils import link_entity
 from yd_commands.wrapper import ARGS_PARSER, CLIENT, CONFIG_COMMON, main_wrapper
 
-ABORT_RETRY_INTERVAL = 20  # Seconds
-
 
 @main_wrapper
 def main():
@@ -201,8 +200,10 @@ def abort_and_follow(work_requirement_summaries: List[WorkRequirementSummary]):
             print_log(f"Collecting Tasks to abort (attempt {attempt})")
             if abort_all_tasks(work_requirement_summaries) == 0:
                 break
-            print_log(f"Waiting {ABORT_RETRY_INTERVAL}s for abort confirmation ...")
-            sleep(ABORT_RETRY_INTERVAL)
+            print_log(
+                f"Waiting {TASK_ABORT_CHECK_INTERVAL}s for abort confirmation ..."
+            )
+            sleep(TASK_ABORT_CHECK_INTERVAL)
     else:
         print_log("Aborting all currently running Tasks")
         abort_all_tasks(work_requirement_summaries)

{yellowdog-python-examples-7.2.5 → yellowdog-python-examples-7.2.7}/yd_commands/config_types.py

@@ -5,16 +5,7 @@ Configuration classes and constants.
 from dataclasses import dataclass, field
 from typing import Dict, List, Optional
 
-YD_KEY = "YD_KEY"
-YD_SECRET = "YD_SECRET"
-YD_NAMESPACE = "YD_NAMESPACE"
-YD_TAG = "YD_TAG"
-YD_URL = "YD_URL"
-TASK_BATCH_SIZE_DEFAULT = 2000
-DEFAULT_URL = "https://portal.yellowdog.co/api"
-NAMESPACE_SEPARATOR = "::"
-CR_BATCH_SIZE_DEFAULT = 10000
-WP_VARIABLES_PREFIX = "__"
+from yd_commands.settings import CR_BATCH_SIZE_DEFAULT, TASK_BATCH_SIZE_DEFAULT
 
 
 @dataclass
@@ -52,6 +43,7 @@ class ConfigWorkRequirement:
     max_workers: Optional[int] = None
     min_workers: Optional[int] = None
     outputs_optional: List[str] = field(default_factory=list)
+    outputs_other: List[Dict] = field(default_factory=list)
     outputs_required: List[str] = field(default_factory=list)
     priority: float = 0.0
     providers: Optional[List[str]] = None

{yellowdog-python-examples-7.2.5 → yellowdog-python-examples-7.2.7}/yd_commands/csv_data.py

@@ -15,11 +15,15 @@ from yd_commands.args import ARGS_PARSER
 from yd_commands.config_types import ConfigWorkRequirement
 from yd_commands.printing import print_json, print_log
 from yd_commands.property_names import *
+from yd_commands.settings import (
+    BOOL_TYPE_TAG,
+    CSV_VAR_CLOSING_DELIMITER,
+    CSV_VAR_OPENING_DELIMITER,
+    NUMBER_TYPE_TAG,
+)
 from yd_commands.variables import (
-    BOOL_SUB,
-    NUMBER_SUB,
     load_jsonnet_file_with_variable_substitutions,
-    process_variable_substitutions,
+    process_variable_substitutions_in_dict_insitu,
 )
 
 
@@ -216,7 +220,7 @@ def perform_csv_task_expansion(wr_data: Dict, csv_files: List[str]) -> Dict:
         exit(0)
 
     # Process remaining substitutions
-    process_variable_substitutions(wr_data)
+    process_variable_substitutions_in_dict_insitu(wr_data)
     return wr_data
 
 
@@ -238,9 +242,11 @@ def make_string_substitutions(input: str, var_name: str, value: str) -> str:
     """
     Helper function to make string substitutions for CSV variables only.
     """
-    input = input.replace(f"{{{{{var_name}}}}}", value)
+    input = input.replace(
+        f"{CSV_VAR_OPENING_DELIMITER}{var_name}{CSV_VAR_CLOSING_DELIMITER}", value
+    )
 
-    num_sub_str = f"{{{{{NUMBER_SUB}{var_name}}}}}"
+    num_sub_str = f"{CSV_VAR_OPENING_DELIMITER}{NUMBER_TYPE_TAG}{var_name}{CSV_VAR_CLOSING_DELIMITER}"
     if num_sub_str in input:
         try:
             float(value)
@@ -248,7 +254,7 @@ def make_string_substitutions(input: str, var_name: str, value: str) -> str:
             raise Exception(f"Invalid number substitution in CSV: '{value}'")
         input = input.replace(f"'{num_sub_str}'", value)
 
-    bool_sub_str = f"{{{{{BOOL_SUB}{var_name}}}}}"
+    bool_sub_str = f"{CSV_VAR_OPENING_DELIMITER}{BOOL_TYPE_TAG}{var_name}{CSV_VAR_CLOSING_DELIMITER}"
     if bool_sub_str in input:
         if value.lower() == "true":
             value = "True"
@@ -314,13 +320,20 @@ def substitions_present(var_names: List[str], task_prototype: str) -> bool:
     Check if there are any CSV substitutions present in the Task prototype.
     """
     return (
-        any(f"{{{{{var_name}}}}}" in task_prototype for var_name in var_names)
+        any(
+            f"{CSV_VAR_OPENING_DELIMITER}{var_name}{CSV_VAR_CLOSING_DELIMITER}"
+            in task_prototype
+            for var_name in var_names
+        )
         or any(
-            f"{{{{{NUMBER_SUB}{var_name}}}}}" in task_prototype
+            f"{CSV_VAR_OPENING_DELIMITER}{NUMBER_TYPE_TAG}{var_name}{CSV_VAR_CLOSING_DELIMITER}"
+            in task_prototype
             for var_name in var_names
         )
         or any(
-            f"{{{{{BOOL_SUB}{var_name}}}}}" in task_prototype for var_name in var_names
+            f"{CSV_VAR_OPENING_DELIMITER}{BOOL_TYPE_TAG}{var_name}{CSV_VAR_CLOSING_DELIMITER}"
+            in task_prototype
+            for var_name in var_names
         )
     )
 
@@ -347,6 +360,7 @@ def csv_expand_toml_tasks(config_wr: ConfigWorkRequirement, csv_file: str) -> Di
         (config_wr.inputs_optional, INPUTS_OPTIONAL),
         (config_wr.inputs_required, INPUTS_REQUIRED),
         (config_wr.outputs_optional, OUTPUTS_OPTIONAL),
+        (config_wr.outputs_other, OUTPUTS_OTHER),
         (config_wr.outputs_required, OUTPUTS_REQUIRED),
         (config_wr.task_data, TASK_DATA),
         (config_wr.task_data_file, TASK_DATA_FILE),
@@ -356,6 +370,7 @@ def csv_expand_toml_tasks(config_wr: ConfigWorkRequirement, csv_file: str) -> Di
         (config_wr.upload_files, UPLOAD_FILES),
         (config_wr.verify_at_start, VERIFY_AT_START),
         (config_wr.verify_wait, VERIFY_WAIT),
+        # Note: not TASK_COUNT; count determined by CSV data
     ]:
         if config_value is not None and substitions_present(
             csv_data.var_names, str(config_value)

{yellowdog-python-examples-7.2.5 → yellowdog-python-examples-7.2.7}/yd_commands/instantiate.py

@@ -15,7 +15,7 @@ from yellowdog_client.model import (
     ComputeRequirementTemplateUsage,
 )
 
-from yd_commands.config_types import WP_VARIABLES_PREFIX, ConfigWorkerPool
+from yd_commands.config_types import ConfigWorkerPool
 from yd_commands.follow_utils import follow_events, follow_ids
 from yd_commands.id_utils import YDIDType
 from yd_commands.load_config import load_config_worker_pool
@@ -26,6 +26,7 @@ from yd_commands.printing import (
     print_yd_object,
 )
 from yd_commands.provision_utils import get_user_data_property
+from yd_commands.settings import WP_VARIABLES_POSTFIX, WP_VARIABLES_PREFIX
 from yd_commands.utils import add_batch_number_postfix, generate_id, link_entity
 from yd_commands.variables import (
     load_json_file_with_variable_substitutions,
@@ -60,7 +61,9 @@ def main():
 
     if cr_json_file is not None:
         print_log(f"Loading Compute Requirement data from: '{cr_json_file}'")
-        create_compute_requirement_from_json(cr_json_file, WP_VARIABLES_PREFIX)
+        create_compute_requirement_from_json(
+            cr_json_file, WP_VARIABLES_PREFIX, WP_VARIABLES_POSTFIX
+        )
         return
 
     if CONFIG_WP.template_id is None:
@@ -195,7 +198,9 @@ def _allocate_nodes_to_batches(
     return batches
 
 
-def create_compute_requirement_from_json(cr_json_file: str, prefix: str = "") -> None:
+def create_compute_requirement_from_json(
+    cr_json_file: str, prefix: str = "", postfix: str = ""
+) -> None:
     """
     Directly create the Compute Requirement using the YellowDog REST API.
     """
@@ -208,11 +213,11 @@ def create_compute_requirement_from_json(cr_json_file: str, prefix: str = "") ->
 
     if cr_json_file.lower().endswith(".jsonnet"):
         cr_data = load_jsonnet_file_with_variable_substitutions(
-            cr_json_file, prefix=prefix
+            cr_json_file, prefix=prefix, postfix=postfix
         )
     else:
         cr_data = load_json_file_with_variable_substitutions(
-            cr_json_file, prefix=prefix
+            cr_json_file, prefix=prefix, postfix=postfix
         )
 
     # Use only the 'requirementTemplateUsage' value (if present);