yellowdog-python-examples 7.8.3__tar.gz → 7.8.5__tar.gz

{yellowdog-python-examples-7.8.3/yellowdog_python_examples.egg-info → yellowdog_python_examples-7.8.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: yellowdog-python-examples
-Version: 7.8.3
+Version: 7.8.5
 Summary: Example Python commands using the YellowDog Python SDK
 Home-page: https://github.com/yellowdog/python-examples
 Author: YellowDog Limited

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/README.md

@@ -132,7 +132,7 @@ The commands provide the following capabilities:
 
 - **Provisioning** Worker Pools with the **`yd-provision`** command
 - **Submitting** Work Requirements with the **`yd-submit`** command
-- **Starting** HELD Work Requirements and **Holding** (or pausing) RUNNING Work Requirements
+- **Starting** HELD Work Requirements and **Holding** (or pausing) RUNNING Work Requirements with the **`yd-start`** and **`yd-hold`** commands
 - **Uploading** files to the YellowDog Object Store with the **`yd-upload`** command
 - **Instantiating** Compute Requirements with the **`yd-instantiate`** command
 - **Downloading** Results from the YellowDog Object Store with the **`yd-download`** command
@@ -142,15 +142,17 @@ The commands provide the following capabilities:
 - **Terminating** Compute Requirements with the **`yd-terminate`** command
 - **Deleting** objects in the YellowDog Object Store with the **`yd-delete`** command
 - **Listing** YellowDog items using the **`yd-list`** command
-- **Resizing** Worker Pools and Compute Requirements
-- **Boosting** Allowances
-- **Creating, Updating and Removing** Source Templates, Compute Templates, Keyrings, Credentials, Namespace Storage Configurations, Image Families, Allowances, and Configured Worker Pools
-- **Following Event Streams** for Work Requirements, Worker Pools and Compute Requirements
+- **Resizing** Worker Pools and Compute Requirements with the **`yd-resize`** command
+- **Boosting** Allowances with the **`yd-boost`** command
+- **Creating, Updating and Removing** Source Templates, Compute Templates, Keyrings, Credentials, Namespace Storage Configurations, Image Families, Allowances, and Configured Worker Pools with the **`yd-create`** and **`yd-remove`** commands
+- **Following Event Streams** for Work Requirements, Worker Pools and Compute Requirements with the **`yd-follow`** command
 
 The operation of the commands is controlled using TOML configuration files and/or environment variables and command line arguments. In addition, Work Requirements and Worker Pools can be defined using JSON files providing extensive configurability.
 
 Commands are also provided for the semi-automatic setup of cloud provider accounts for use with YellowDog, and the creation of YellowDog assets to work with these cloud provider accounts. Please see **[Cloud Wizard](README_CLOUDWIZARD.md)** for more details.
 
+Run any command with the `--help`/`-h` option to discover the command's options.
+
 # YellowDog Prerequisites
 
 To submit **Work Requirements** to YellowDog for processing by Configured Worker Pools (on-premise) and/or Provisioned Worker Pools (cloud-provisioned resources), you'll need:
@@ -275,6 +277,10 @@ All entity names used within the YellowDog Platform must comply with the followi
 
 These restrictions apply to entities including Namespaces, Tags, Work Requirements, Task Groups, Tasks, Worker Pools, and Compute Requirements, and also apply to entities that are currently used indirectly by these scripts, including Usernames, Credentials, Keyrings, Compute Sources and Compute Templates.
 
+Later sections of this document describe variable substitutions implemented with user-defined and CSV-file-defined variables. As a type modifier within these substitution expressions, the `format_name:` option is available, and works in the same manner as `num:`, `bool:`, etc. The `format_name:` modifier will convert the substituted string into one that satisfies YellowDog naming, by switching characters to lower case, etc.
+
+For example, a variable substitution `{{format_name:ligand_name}}`, with variable `ligand_name` set to `DCCCDE_00000s`, would substitute to become `dcccde_00000s`, and would be acceptable for use as a component of a YellowDog name.
+
 # Common Properties
 
 The `[common]` section of the configuration file can contain the following properties:
@@ -1803,7 +1809,7 @@ However, this means that **caution is required** when updating or removing resou
 
 The JSON specification used to define each type of resource can be found by inspecting the YellowDog Platform REST API documentation at https://docs.yellowdog.co/api.
 
-For example, to obtain the JSON schema for creating a Compute Source Template, take a look at the REST API call for adding a new Compute Source template: https://docs.yellowdog.co/api/?spec=Compute%20API#tag/compute/post/compute/templates/sources. This will display an **Example Value**, and an adjacent tab will show the **Schema**.
+For example, to obtain the JSON schema for creating a Compute Source Template, take a look at the REST API call for adding a new Compute Source template: https://docs.yellowdog.co/api/?spec=Compute%20API#tag/compute/post/compute/templates/sources.
 
 When using the `yd-create` and `yd-remove` commands, note that an additional property `resource` must be supplied, to identify the type of resource being specified. The `"resource"` property can take the following values:
 
@@ -2563,4 +2569,4 @@ It can optionally be supplied with a list of the names and/or YDIDs of the speci
 
 ## yd-boost
 
-The `yd-boost` command is used to boost an Allowance by the specified number of hours.
+The `yd-boost` command is used to boost Allowances by the specified number of hours.

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/tests/test_demos.py

@@ -53,6 +53,14 @@ class TestDemos:
         result = shell(f"cd {DEMO_DIR}/cmd.exe && {CMD_SEQ}")
         assert result.exit_code == 0
 
+    def test_blender(self):
+        result = shell(f"cd {DEMO_DIR}/blender && {CMD_SEQ}")
+        assert result.exit_code == 0
+
+    def test_montecarlo(self):
+        result = shell(f"cd {DEMO_DIR}/montecarlo && {CMD_SEQ}")
+        assert result.exit_code == 0
+
     def test_nextflow_image_montage(self):
         result = shell(
             f"cd {DEMO_DIR}/nextflow/image-montage && {NEXTFLOW} main.nf "

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/tests/test_dryruns.py

@@ -60,6 +60,10 @@ class TestDemoDryRuns:
         result = shell(f"cd {DEMO_DIR}/blender && {CMD_SEQ}")
         assert result.exit_code == 0
 
+    def test_montecarlo(self):
+        result = shell(f"cd {DEMO_DIR}/montecarlo && {CMD_SEQ}")
+        assert result.exit_code == 0
+
     # Tests run from outside the demo directories
     def test_bash_out(self):
         demo_name = "bash"
@@ -169,3 +173,12 @@ class TestDemoDryRuns:
             f" {demo_name}/config.toml"
         )
         assert result.exit_code == 0
+
+    def test_montecarlo_out(self):
+        demo_name = "montecarlo"
+        result = shell(
+            f"cd {DEMO_DIR} && yd-provision -D -c {demo_name}/config.toml && yd-submit"
+            f" -D -c {demo_name}/config.toml && yd-instantiate -D -c"
+            f" {demo_name}/config.toml"
+        )
+        assert result.exit_code == 0

yellowdog_python_examples-7.8.5/yd_commands/__init__.py

@@ -0,0 +1 @@
+__version__ = "7.8.5"

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/args.py

@@ -513,6 +513,7 @@ class CLIParser:
             )
             parser.add_argument(
                 "--instances",
+                "-i",
                 action="store_true",
                 required=False,
                 help="list compute instances",
@@ -810,8 +811,8 @@ class CLIParser:
                 type=str,
                 required=False,
                 help=(
-                    "the directory in which files for upload (or for user data) are"
-                    " found"
+                    "the directory in which files for upload (or for user data "
+                    "or CSV data) are found"
                 ),
                 metavar="<directory>",
             )

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/csv_data.py

@@ -20,8 +20,10 @@ from yd_commands.settings import (
     BOOL_TYPE_TAG,
     CSV_VAR_CLOSING_DELIMITER,
     CSV_VAR_OPENING_DELIMITER,
+    FORMAT_NAME_TYPE_TAG,
     NUMBER_TYPE_TAG,
 )
+from yd_commands.utils import format_yd_name
 from yd_commands.variables import (
     load_jsonnet_file_with_variable_substitutions,
     process_variable_substitutions_insitu,
@@ -166,7 +168,7 @@ def load_toml_file_with_csv_task_expansion(
     with open(resolve_filename(files_directory, toml_file), "r") as f:
         wr_data = toml_load(f)
 
-    return perform_csv_task_expansion(wr_data, csv_files, f)
+    return perform_csv_task_expansion(wr_data, csv_files, files_directory)
 
 
 def perform_csv_task_expansion(
@@ -273,6 +275,13 @@ def make_string_substitutions(input: str, var_name: str, value: str) -> str:
             raise Exception(f"Invalid Boolean substitution in CSV: '{value}'")
         input = input.replace(f"'{bool_sub_str}'", value)
 
+    lower_sub_str = f"{CSV_VAR_OPENING_DELIMITER}{FORMAT_NAME_TYPE_TAG}{var_name}{CSV_VAR_CLOSING_DELIMITER}"
+    if lower_sub_str in input:
+        input = input.replace(
+            f"{lower_sub_str}",
+            format_yd_name(value, add_prefix=False),
+        )
+
     return input
 
 
@@ -344,6 +353,11 @@ def substitions_present(var_names: List[str], task_prototype: str) -> bool:
             in task_prototype
             for var_name in var_names
         )
+        or any(
+            f"{CSV_VAR_OPENING_DELIMITER}{FORMAT_NAME_TYPE_TAG}{var_name}{CSV_VAR_CLOSING_DELIMITER}"
+            in task_prototype
+            for var_name in var_names
+        )
     )
 
 
@@ -395,4 +409,4 @@ def csv_expand_toml_tasks(
         ):
             task_proto[config_name] = config_value
 
-    return perform_csv_task_expansion(wr_data, [csv_file])
+    return perform_csv_task_expansion(wr_data, [csv_file], files_directory)

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/interactive.py

@@ -75,7 +75,7 @@ def select(
             if single_result
             else (f"Please select items (e.g.: 1,2,4-7 / *){cancel_string}:")
         )
-        selector_string = CONSOLE.input(print_string(input_string) + " ")
+        selector_string = _get_user_input(print_string(input_string) + " ")
         if selector_string.strip() == "*":
             selector_string = f"1-{len(objects)}"
         selector_list = selector_string.split(",")
@@ -153,10 +153,20 @@ def confirmed(msg: str) -> bool:
 
     # Seek user confirmation
     while True:
-        response = CONSOLE.input(print_string(f"{msg} (y/N):") + " ")
+        response = _get_user_input(print_string(f"{msg} (y/N):") + " ")
         if response.lower() in ["y", "yes"]:
             print_log("Action confirmed by user")
             return True
         elif response.lower() in ["n", "no", ""]:
             print_log("Action cancelled by user")
             return False
+
+
+def _get_user_input(input_prompt: str) -> str:
+    """
+    Get user input, respecting the --no-format option.
+    """
+    if ARGS_PARSER.no_format:
+        return input(input_prompt)
+    else:
+        return CONSOLE.input(input_prompt)

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/list.py

@@ -46,10 +46,10 @@ from yd_commands.object_utilities import (
     get_task_groups_from_wr_summary,
 )
 from yd_commands.printing import (
-    CONSOLE_TABLE,
     indent,
     print_log,
     print_numbered_object_list,
+    print_table_core,
     print_yd_object,
     sorted_objects,
 )
@@ -525,11 +525,11 @@ def list_namespaces():
             for index, namespace in enumerate(namespace_list)
         ]
     )
-    print()
-    CONSOLE_TABLE.print(
+    print(flush=True)
+    print_table_core(
         indent(tabulate(rows, headings, tablefmt="simple_outline")),
     )
-    print()
+    print(flush=True)
 
     if ARGS_PARSER.details:  # Print the details for non-default only
         for namespace in select(CLIENT, namespaces_config, showing_all=True):

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/printing.py

@@ -495,9 +495,10 @@ def instances_table(
 ) -> (List[str], List[str]):
     headers = [
         "#",
-        "Type",
         "Provider",
         "Instance Type",
+        "Hostname",
+        "Status",
         "Private IP",
         "Public IP",
     ]
@@ -506,9 +507,10 @@ def instances_table(
         table.append(
             [
                 index + 1,
-                instance.type.split(".")[-1],
                 instance.provider,
                 instance.instanceType,
+                instance.hostname,
+                instance.status,
                 instance.privateIpAddress,
                 instance.publicIpAddress,
             ]
@@ -920,22 +922,12 @@ def print_batch_download_files(
         counter += 1
 
     print(flush=True)
-
-    if ARGS_PARSER.no_format:
-        print(
-            indent(
-                tabulate(table, headers=headers, tablefmt="simple_outline"),
-                indent_width=4,
-            ),
-            flush=True,
-        )
-    else:
-        CONSOLE_TABLE.print(
-            indent(
-                tabulate(table, headers=headers, tablefmt="simple_outline"),
-                indent_width=4,
-            ),
+    print_table_core(
+        indent(
+            tabulate(table, headers=headers, tablefmt="simple_outline"),
+            indent_width=4,
         )
+    )
     print(flush=True)
     return counter
 

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/settings.py

@@ -31,6 +31,7 @@ NUMBER_TYPE_TAG = "num" + TYPE_TAG_TERMINATOR
 BOOL_TYPE_TAG = "bool" + TYPE_TAG_TERMINATOR
 ARRAY_TYPE_TAG = "array" + TYPE_TAG_TERMINATOR
 TABLE_TYPE_TAG = "table" + TYPE_TAG_TERMINATOR
+FORMAT_NAME_TYPE_TAG = "format_name" + TYPE_TAG_TERMINATOR
 TOML_VAR_NESTED_DEPTH = 3
 
 DEFAULT_LOG_WIDTH = 120

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/submit.py

@@ -57,7 +57,6 @@ from yd_commands.settings import (
 )
 from yd_commands.submit_utils import (
     UploadedFiles,
-    format_yd_name,
     generate_task_input_list,
     pause_between_batches,
     update_config_work_requirement_object,
@@ -71,7 +70,7 @@ from yd_commands.type_check import (
     check_str,
 )
 from yd_commands.upload_utils import unique_upload_pathname
-from yd_commands.utils import generate_id, link_entity
+from yd_commands.utils import format_yd_name, generate_id, link_entity
 from yd_commands.validate_properties import validate_properties
 from yd_commands.variables import (
     L_TASK_COUNT,
@@ -156,7 +155,7 @@ def main():
     if wr_data_file is None and csv_files is not None:
         wr_data = csv_expand_toml_tasks(CONFIG_WR, csv_files[0], files_directory)
         submit_work_requirement(
-            files_directory=CONFIG_FILE_DIR,
+            files_directory=files_directory,
             wr_data=wr_data,
         )
 
@@ -1233,7 +1232,7 @@ def create_task(
             args.insert(0, "/c")
         return _make_task(flatten_input_paths)
 
-    # Special processing for Docker tasks if the 'executable property is set.
+    # Special processing for Docker tasks if the 'executable' property is set.
     # Sets up the '--env' environment strings and the DockerHub username and
     # password if specified.
     elif task_type == "docker":
@@ -1318,6 +1317,13 @@ def create_task(
             if docker_registry is not None
             else {}
         )
+        env_copy.update(
+            {
+                "DOCKER_IMAGE": executable,
+            }
+            if executable is not None
+            else {}
+        )
         return _make_task(flatten_input_paths)
 
     else:

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/submit_utils.py

@@ -2,7 +2,6 @@
 Utility functions for use with the submit command.
 """
 
-import re
 from dataclasses import dataclass
 from glob import glob
 from os import chdir, getcwd
@@ -14,7 +13,7 @@ from yellowdog_client import PlatformClient
 from yellowdog_client.model import ObjectPath, TaskInput, TaskInputVerification
 
 from yd_commands.config_types import ConfigCommon, ConfigWorkRequirement
-from yd_commands.printing import print_error, print_log, print_warning
+from yd_commands.printing import print_error, print_log
 from yd_commands.settings import NAMESPACE_PREFIX_SEPARATOR
 from yd_commands.upload_utils import unique_upload_pathname, upload_file_core
 from yd_commands.variables import process_variable_substitutions_insitu
@@ -239,25 +238,6 @@ class UploadedFiles:
         self._uploaded_files = []
 
 
-def format_yd_name(yd_name: str) -> str:
-    """
-    Format a string to be consistent with YellowDog naming requirements.
-    """
-    # Make obvious substitutions
-    new_yd_name = yd_name.replace("/", "-").replace(" ", "_").lower()
-    # Enforce acceptable regex, starting character and name length
-    new_yd_name = re.sub("[^a-z0-9_-]", "", new_yd_name)
-    if not new_yd_name[0].isalpha():
-        new_yd_name = f"yd_{new_yd_name}"
-    new_yd_name = new_yd_name[:60]
-    if new_yd_name != yd_name:
-        print_warning(
-            f"Changing name '{yd_name}' to '{new_yd_name}' to comply with YellowDog"
-            " naming requirements"
-        )
-    return new_yd_name
-
-
 def update_config_work_requirement_object(
     config_wr: ConfigWorkRequirement,
 ) -> ConfigWorkRequirement:

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/utils.py

@@ -218,3 +218,21 @@ def remove_outer_delimiters(
     return input_string.replace(f"{opening_delimiter}", "", 1)[::-1].replace(
         f"{closing_delimiter[::-1]}", "", 1
     )[::-1]
+
+
+def format_yd_name(yd_name: str, add_prefix: bool = True) -> str:
+    """
+    Format a string to be consistent with YellowDog naming requirements.
+    """
+    # Make obvious substitutions
+    new_yd_name = yd_name.replace("/", "-").replace(" ", "_").lower()
+
+    # Enforce acceptable regex
+    new_yd_name = re.sub("[^a-z0-9_-]", "", new_yd_name)
+
+    # Must start with an alphabetic character
+    if add_prefix and not new_yd_name[0].isalpha():
+        new_yd_name = f"y{new_yd_name}"
+
+    # Mustn't exceed 60 chars
+    return new_yd_name[:60]

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5}/yd_commands/variables.py

@@ -22,6 +22,7 @@ from yd_commands.settings import (
     ARRAY_TYPE_TAG,
     BOOL_TYPE_TAG,
     ENV_VAR_PREFIX,
+    FORMAT_NAME_TYPE_TAG,
     NUMBER_TYPE_TAG,
     RAND_VAR_SIZE,
     TABLE_TYPE_TAG,
@@ -33,7 +34,12 @@ from yd_commands.settings import (
     WP_VARIABLES_POSTFIX,
     WP_VARIABLES_PREFIX,
 )
-from yd_commands.utils import UTCNOW, remove_outer_delimiters, split_delimited_string
+from yd_commands.utils import (
+    UTCNOW,
+    format_yd_name,
+    remove_outer_delimiters,
+    split_delimited_string,
+)
 
 # Set up default variable substitutions
 VARIABLE_SUBSTITUTIONS = {
@@ -185,7 +191,8 @@ def process_variable_substitutions(
             type_tag = (
                 re.match(
                     f"^{opening_delimiter}({NUMBER_TYPE_TAG}|{BOOL_TYPE_TAG}"
-                    f"|{TABLE_TYPE_TAG}|{ARRAY_TYPE_TAG})(?!{TAG_DEFAULT_DIFF})",
+                    f"|{TABLE_TYPE_TAG}|{ARRAY_TYPE_TAG}|{FORMAT_NAME_TYPE_TAG})"
+                    f"(?!{TAG_DEFAULT_DIFF})",
                     element,
                 )
                 .group(0)
@@ -313,6 +320,9 @@ def process_typed_variable_substitution(
     Process a single typed substitution, returning the appropriate type.
     Assumes there is a substitution present.
     """
+    if type_string == FORMAT_NAME_TYPE_TAG:
+        return format_yd_name(input_string, add_prefix=False)
+
     if type_string == NUMBER_TYPE_TAG:
         try:
             return int(input_string)

{yellowdog-python-examples-7.8.3 → yellowdog_python_examples-7.8.5/yellowdog_python_examples.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: yellowdog-python-examples
-Version: 7.8.3
+Version: 7.8.5
 Summary: Example Python commands using the YellowDog Python SDK
 Home-page: https://github.com/yellowdog/python-examples
 Author: YellowDog Limited

yellowdog-python-examples-7.8.3/yd_commands/__init__.py

@@ -1 +0,0 @@
-__version__ = "7.8.3"