yellowdog-python-examples 7.18.4__tar.gz → 7.18.6__tar.gz

{yellowdog_python_examples-7.18.4/yellowdog_python_examples.egg-info → yellowdog_python_examples-7.18.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: yellowdog-python-examples
-Version: 7.18.4
+Version: 7.18.6
 Summary: Python CLI commands using the YellowDog Python SDK
 Author-email: YellowDog Limited <support@yellowdog.co>
 Project-URL: Homepage, https://github.com/yellowdog/python-examples
@@ -20,7 +20,7 @@ Requires-Dist: requests
 Requires-Dist: rich==13.9.4
 Requires-Dist: tabulate>=0.9.0
 Requires-Dist: toml
-Requires-Dist: yellowdog-sdk>=9.3.3
+Requires-Dist: yellowdog-sdk>=9.4.0
 Provides-Extra: jsonnet
 Requires-Dist: jsonnet; extra == "jsonnet"
 Provides-Extra: cloudwizard

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/README.md

@@ -84,7 +84,8 @@
       * [Resource Removal](#resource-removal)
       * [Resource Matching](#resource-matching)
    * [Resource Specification Definitions](#resource-specification-definitions)
-   * [Generating Resource Specifications](#generating-resource-specifications)
+   * [Generating Resource Specifications using yd-list](#generating-resource-specifications-using-yd-list)
+      * [Usage Scenario: Moving or Copying Resources to a New Namespace](#usage-scenario-moving-or-copying-resources-to-a-new-namespace)
    * [Preprocessing Resource Specifications](#preprocessing-resource-specifications)
    * [Keyrings](#keyrings)
    * [Credentials](#credentials)
@@ -131,7 +132,7 @@
    * [yd-compare](#yd-compare)
 
 <!-- Created by https://github.com/ekalinin/github-markdown-toc -->
-<!-- Added by: pwt, at: Mon Jul 28 21:41:59 BST 2025 -->
+<!-- Added by: pwt, at: Wed Aug  6 08:55:44 BST 2025 -->
 
 <!--te-->
 
@@ -386,9 +387,11 @@ If all the required common properties are set using the command line or environm
 
 ## Support for `.env` Files
 
-Environment variables can also be set in a `.env` file, typically in the user's home directory. Entries in the `.env` file will not overwrite existing environment variables -- i.e., environment variables take precedence over entries in the `.env` file.
+Environment variables can also be set in a `.env` file, typically in the user's home directory or the current working directory.
 
-Environment variables sourced from a `.env` file whose names start with `YD` will be reported on the command line. Variables that do not start with `YD` will not be reported, but they will still be applied.
+Entries in the `.env` file will not overwrite existing environment variables -- i.e., environment variables take precedence over entries in the `.env` file. This precedence can be reversed by using the `--env-override` command line option.
+
+Environment variables sourced from a `.env` file whose names start with `YD` will be reported on the command line. Variables whose names do not start with `YD` will not be reported, but they will still be applied.
 
 ## Variable Substitutions in Common Properties
 
@@ -1995,18 +1998,19 @@ When using the `yd-create` and `yd-remove` commands, note that an additional pro
 - `"Application"`
 - `"User"`
 
-## Generating Resource Specifications
+## Generating Resource Specifications using `yd-list`
 
-To generate example JSON specifications from resources already included in the platform, the `yd-list` command can be used with the `--details` option, and select the resources for which details are required. E.g.:
+To generate example JSON specifications from resources already included in the platform, the `yd-list` command can be used with the `--details`, `--substitute-ids`/`-U`, and  `--strip-ids` options, and select the resources for which details are required. E.g.:
 
 ```shell
-yd-list --keyrings --details
-yd-list --source-templates --details
-yd-list --compute-templates --details
-yd-list --image-families --details
+yd-list --source-templates --details --substitute-ids --strip-ids
+yd-list --compute-templates --details --substitute-ids --strip-ids
+yd-list --image-families --details --substitute-ids --strip-ids
 ```
 
-This will produce a list of resource specifications that can be copied and used directly with `yd-create` and `yd-remove`. Certain fields, such as the ID, will be ignored, with warnings. The detailed resource list can also be copied directly to an output file in addition to being displayed on the console:
+This will produce a list of resource specifications that can be copied and used directly with `yd-create` and `yd-remove`.
+
+The detailed resource list can also be copied directly to an output file in addition to being displayed on the console using the `--output-file` option:
 
 ```shell
 yd-list yd-list --source-templates --details --output-file my-resources.json
@@ -2015,12 +2019,47 @@ yd-list yd-list --source-templates --details --output-file my-resources.json
 Alternatively, the `yd-show` command can be used with one or more `ydid` arguments to generate the details of each identified resource. E.g.,
 
 ```shell
-yd-show -q ydid:cst:000000:cde265f8-0b17-4e0e-be1c-505174a620e4 > my-compute-source-template.json
+yd-show -q ydid:cst:000000:cde265f8-0b17-4e0e-be1c-505174a620e4 --substitute-ids --strip-ids --output-file my-compute-source-template.json
 ```
 
 would generate a JSON file that can be used with `yd-create` without alteration, or which could be edited.
 
-Both `yd-list` and `yd-show` support the `--substitute-ids`/`-U` option. For Compute Requirement Template detailed output, this will substitute Compute Source Template IDs and Image Family IDs with their names, to make it easier to use the output acrosss YellowDog accounts.
+As illustrated above, both `yd-list` and `yd-show` support the `--substitute-ids`/`-U` option. For Compute Requirement Template detailed output, this will substitute Compute Source Template IDs and Image Family IDs with their names, to make it easier to reuse the outputs. For Compute Source Templates, Image Family IDs will be substituted. (Only Image Family IDs are substituted, not Image Group or Image IDs; if these are used in CRTs/CSTs, the JSON resource output may not be reusable.)
+
+The `--strip-ids` option will remove any YellowDog IDs ('ydids') from the JSON output, as well as any other properties that are not required in order to use the output with `yd-create`.
+
+### Usage Scenario: Moving or Copying Resources to a New Namespace
+
+In the following usage scenario, we want to move a set of resources from one namespace `ns-1`, to another `ns-2`. We'll move all compute source templates, compute requirement templates, and image families.
+
+**Step 1: Capture the target resources in JSON files**
+
+```shell
+yd-list -q --compute-source-templates --namespace ns-1 --substitute-ids --strip-ids --auto-select-all --output-file csts.json
+yd-list -q --compute-requirement-templates --namespace ns-1 --substitute-ids --strip-ids --auto-select-all --output-file crts.json
+yd-list -q --image-families --namespace ns-1 --substitute-ids --strip-ids --auto-select-all --output-file ifs.json
+```
+
+**Step 2: Remove all target resources** if moving resources
+
+The following will remove all target resources included in the JSON resource files **without user confirmation**. If one instead wants to **copy** the resources to the new namespace rather than move them, omit this step.
+
+```shell
+yd-remove -y csts.json crts.json ifs.json
+```
+
+**Step 3: Change the namespace in all the resources**
+
+Use an editor's search and replace function, or a command line tool such as `sed` to replace all occurences of `"ns-1"` with `"ns-2`", for every `namespace` property, in each of the JSON files.
+
+**Step 4: Recreate all resources in the new namespace**
+
+```shell
+yd-create -y csts.json crts.json ifs.json
+```
+
+Once the resources have been created successfully, the JSON files can be deleted (or retained for your records).
+
 
 ## Preprocessing Resource Specifications
 
@@ -2372,7 +2411,7 @@ Namespace Policies are matched by their `namespace` property when using `yd-crea
 
 ## Groups
 
-When creating and removing groups, a list of roles can can be supplied and the group will be created or updated with the roles specified. Roles can be identified by their names or YellowDog IDs.
+When creating and updating groups, a list of roles can can be supplied and the group will be created or updated with the roles specified. Roles can be identified by their names or YellowDog IDs.
 
 Example:
 
@@ -2402,7 +2441,7 @@ Example:
 
 ### Creating and Regenerating Application Keys
 
-When Application is created its Application Key ID and Secret will be displayed (even if the `--quiet` option is used).
+When an Application is created its Application Key ID and Secret will be displayed (even if the `--quiet` option is used).
 
 When an Application is updated, the `--regenerate-app-keys` option can be used. This will invalidate the current Application key and secret, revoke any Keyring access, and generate a new key and secret which will be displayed.
 

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/requirements.txt

@@ -5,4 +5,4 @@ requests
 rich == 13.9.4
 tabulate >= 0.9.0
 toml
-yellowdog-sdk >= 9.3.3
+yellowdog-sdk >= 9.4.0

yellowdog_python_examples-7.18.6/yellowdog_cli/__init__.py

@@ -0,0 +1 @@
+__version__ = "7.18.6"

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/create.py

@@ -532,7 +532,7 @@ def create_image_family(resource):
         if e.response.status_code == 404:
             # This will create the Image Family and all of its constituent
             # Image Group/Image resources
-            image_family = CLIENT.images_client.add_image_family(image_family)
+            image_family = _create_image_family(image_family, fq_name)
             print_log(f"Created Machine Image Family '{fq_name}' ('{image_family.id}')")
             if ARGS_PARSER.quiet:
                 print(image_family.id)
@@ -984,7 +984,7 @@ def create_namespace_policy(resource: Dict):
         return
 
     print_log(
-        f"Created or updated  Namespace Policy '{namespace_policy.namespace}' with "
+        f"Created or updated Namespace Policy '{namespace_policy.namespace}' with "
         f"'autoscalingMaxNodes={namespace_policy.autoscalingMaxNodes}'"
     )
 
@@ -1007,7 +1007,7 @@ def create_group(resource: Dict):
         for role_name in roles:
             role_id = get_role_id_by_name(CLIENT, role_name)
             if role_id is None:
-                print_warning(f"Role name '{role_name}' not found ... ignoring")
+                print_warning(f"Role '{role_name}' not found ... ignoring")
             else:
                 new_role_ids.add(role_id)
 
@@ -1056,6 +1056,9 @@ def create_group(resource: Dict):
         Helper function to update an existing group, including updating
         its roles.
         """
+        if not confirmed(f"Update Group '{name}' ({group_id})?"):
+            return
+
         group: Group = CLIENT.account_client.update_group(
             group_id, _get_model_object(RN_UPDATE_GROUP_REQUEST, resource)
         )
@@ -1088,7 +1091,7 @@ def create_application(resource: Dict):
         for group_name in groups:
             app_id = get_group_id_by_name(CLIENT, group_name)
             if app_id is None:
-                print_warning(f"Group name '{group_name}' not found ... ignoring")
+                print_warning(f"Group '{group_name}' not found ... ignoring")
             else:
                 new_group_ids.add(app_id)
 
@@ -1148,6 +1151,9 @@ def create_application(resource: Dict):
         Helper function to update an existing application, including updating
         its groups.
         """
+        if not confirmed(f"Update Application '{name}' ({app_id})?"):
+            return
+
         app: Application = CLIENT.account_client.update_application(
             app_id, _get_model_object(RN_UPDATE_APPLICATION_REQUEST, resource)
         )
@@ -1192,7 +1198,7 @@ def update_user(resource: Dict):
         for group_name in groups:
             group_id = get_group_id_by_name(CLIENT, group_name)
             if group_id is None:
-                print_warning(f"Group name '{group_name}' not found ... ignoring")
+                print_warning(f"Group '{group_name}' not found ... ignoring")
             else:
                 new_group_ids.add(group_id)
 
@@ -1203,6 +1209,9 @@ def update_user(resource: Dict):
         if groups is None:
             return
 
+        if not confirmed(f"Update Groups for User '{username}' ({user.id})?"):
+            return
+
         current_group_ids = {group.id for group in get_user_groups(CLIENT, user.id)}
 
         if current_group_ids == new_group_ids:
@@ -1245,12 +1254,49 @@ def update_user(resource: Dict):
     username = user.username if isinstance(user, InternalUser) else user.name
 
     if groups is not None:
-        print_log(f"Updating Groups for User '{username}' ({user.id})")
         update_groups(user)
     else:
         print_log(f"Nothing to do for User '{username}' ({user.id})")
 
 
+def _create_image_family(
+    image_family: MachineImageFamily, fq_name: str
+) -> MachineImageFamily:
+    """
+    Creates a new image family. Only one image group can be added at the time of
+    image family creation, so any additional image groups must be added separately.
+    """
+
+    # Remove all except the first image group; keep the rest as a separate list
+    image_groups = image_family.imageGroups
+    if image_groups is not None:
+        image_family.imageGroups = image_groups[:1]
+        image_groups = image_groups[1:]  # Remaining image groups
+
+    # Create the image family
+    try:
+        image_family = CLIENT.images_client.add_image_family(image_family)
+    except Exception as e:
+        raise Exception(f"Failed to create Machine Image Family '{fq_name}': {e}")
+
+    if image_groups is None or len(image_groups) == 0:
+        return image_family
+
+    # Create any additional image groups
+    for image_group in image_groups:
+        try:
+            image_group = CLIENT.images_client.add_image_group(
+                image_family, image_group
+            )
+        except Exception as e:
+            raise Exception(
+                f"Failed to add Machine Image Group '{image_group.name}' to "
+                f"Image Family '{fq_name}': {e}"
+            )
+
+    return image_family
+
+
 # Entry point
 if __name__ == "__main__":
     main()

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/list.py

@@ -75,11 +75,16 @@ from yellowdog_cli.utils.printing import (
     sorted_objects,
 )
 from yellowdog_cli.utils.settings import (
+    PROP_GROUPS,
+    PROP_RESOURCE,
     RN_ALLOWANCE,
+    RN_APPLICATION,
+    RN_GROUP,
     RN_IMAGE_FAMILY,
     RN_KEYRING,
     RN_NUMERIC_ATTRIBUTE_DEFINITION,
     RN_REQUIREMENT_TEMPLATE,
+    RN_ROLE,
     RN_SOURCE_TEMPLATE,
     RN_STRING_ATTRIBUTE_DEFINITION,
 )
@@ -104,7 +109,7 @@ def main():
         ARGS_PARSER.details = True
 
     if ARGS_PARSER.details and ARGS_PARSER.strip_ids:
-        print_log("Omitting YellowDog IDs from detailed JSON objects")
+        print_log("Omitting YellowDog IDs (etc.) from detailed JSON objects")
 
     if ARGS_PARSER.output_file and ARGS_PARSER.details:
         if exists(ARGS_PARSER.output_file):
@@ -584,7 +589,7 @@ def list_compute_requirement_templates():
                 CLIENT,
                 CLIENT.compute_client.get_compute_requirement_template(cr_template.id),
             ),
-            RN_REQUIREMENT_TEMPLATE,
+            {PROP_RESOURCE: RN_REQUIREMENT_TEMPLATE},
         )
         for cr_template in cr_templates
     ]
@@ -633,7 +638,7 @@ def list_compute_source_templates():
                 CLIENT,
                 CLIENT.compute_client.get_compute_source_template(cs_template.id),
             ),
-            RN_SOURCE_TEMPLATE,
+            {PROP_RESOURCE: RN_SOURCE_TEMPLATE},
         )
         for cs_template in cs_templates
     ]
@@ -662,7 +667,7 @@ def list_keyrings():
 
     # Show details
     print_yd_object_list(
-        [(keyring, RN_KEYRING) for keyring in select(CLIENT, keyrings)]
+        [(keyring, {PROP_RESOURCE: RN_KEYRING}) for keyring in select(CLIENT, keyrings)]
     )
 
 
@@ -712,7 +717,7 @@ def list_image_families():
     image_families = [
         (
             CLIENT.images_client.get_image_family_by_id(image_family_summary.id),
-            RN_IMAGE_FAMILY,
+            {PROP_RESOURCE: RN_IMAGE_FAMILY},
         )
         for image_family_summary in image_family_summaries
     ]
@@ -809,7 +814,10 @@ def list_allowances():
 
     # Show details
     print_yd_object_list(
-        [(allowance, RN_ALLOWANCE) for allowance in select(CLIENT, allowances)]
+        [
+            (allowance, {PROP_RESOURCE: RN_ALLOWANCE})
+            for allowance in select(CLIENT, allowances)
+        ]
     )
 
 
@@ -841,11 +849,13 @@ def list_attribute_definitions():
     attribute_definition_list = [
         (
             attribute,
-            (
-                RN_NUMERIC_ATTRIBUTE_DEFINITION
-                if "Numeric" in attribute["type"]
-                else RN_STRING_ATTRIBUTE_DEFINITION
-            ),
+            {
+                PROP_RESOURCE: (
+                    RN_NUMERIC_ATTRIBUTE_DEFINITION
+                    if "Numeric" in attribute["type"]
+                    else RN_STRING_ATTRIBUTE_DEFINITION
+                )
+            },
         )
         for attribute in select(
             CLIENT,
@@ -904,13 +914,20 @@ def list_users():
         print_numbered_object_list(CLIENT, users, object_type_name="User")
         return
 
-    for selected_user in select(CLIENT, users, object_type_name="User"):
-        add_groups_field = {
-            "groups": [
-                group.name for group in get_user_groups(CLIENT, selected_user.id)
-            ]
-        }
-        print_yd_object(selected_user, add_fields=add_groups_field)
+    # Add the list of groups to the user details
+    print_yd_object_list(
+        [
+            (
+                user,
+                {
+                    PROP_GROUPS: [
+                        group.name for group in get_user_groups(CLIENT, user.id)
+                    ]
+                },
+            )
+            for user in select(CLIENT, users)
+        ]
+    )
 
 
 def list_applications():
@@ -929,13 +946,22 @@ def list_applications():
         print_numbered_object_list(CLIENT, applications, object_type_name="Application")
         return
 
-    for selected_app in select(CLIENT, applications, object_type_name="Application"):
-        add_groups_field = {
-            "groups": [
-                group.name for group in get_application_groups(CLIENT, selected_app.id)
-            ]
-        }
-        print_yd_object(selected_app, add_fields=add_groups_field)
+    # Add the list of group names for each application
+    print_yd_object_list(
+        [
+            (
+                application,
+                {
+                    PROP_GROUPS: [
+                        group.name
+                        for group in get_application_groups(CLIENT, application.id)
+                    ],
+                    PROP_RESOURCE: RN_APPLICATION,
+                },
+            )
+            for application in select(CLIENT, applications)
+        ]
+    )
 
 
 def list_groups():
@@ -951,15 +977,24 @@ def list_groups():
     group_summaries.sort(key=lambda group: group.name if group.name is not None else "")
 
     groups: List[Group] = [
-        CLIENT.account_client.get_group(x.id) for x in group_summaries
+        CLIENT.account_client.get_group(group.id) for group in group_summaries
     ]
 
     if not ARGS_PARSER.details:
         print_numbered_object_list(CLIENT, groups, object_type_name="Group")
         return
 
-    for selected_users in select(CLIENT, groups, object_type_name="Group"):
-        print_yd_object(selected_users)
+    selected_groups = select(CLIENT, groups)
+
+    # If stripping IDs, just supply the list of role names;
+    # subverts the type
+    if ARGS_PARSER.strip_ids:
+        for group in selected_groups:
+            group.roles = [group_role.role.name for group_role in group.roles]
+
+    print_yd_object_list(
+        [(group, {PROP_RESOURCE: RN_GROUP}) for group in selected_groups]
+    )
 
 
 def list_roles():
@@ -986,8 +1021,9 @@ def list_roles():
         print_numbered_object_list(CLIENT, roles, object_type_name="Role")
         return
 
-    for selected_users in select(CLIENT, roles, object_type_name="Role"):
-        print_yd_object(selected_users)
+    print_yd_object_list(
+        [(role, {PROP_RESOURCE: RN_ROLE}) for role in select(CLIENT, roles)]
+    )
 
 
 def list_permissions():

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/remove.py

@@ -498,6 +498,9 @@ def remove_namespace_policy(resource: Dict):
         print_error(f"Namespace Policy '{namespace}' not found")
         return
 
+    if not confirmed(f"Remove Namespace Policy '{namespace}'?"):
+        return
+
     try:
         CLIENT.namespaces_client.delete_namespace_policy(namespace)
         print_log(f"Removed Namespace Policy '{namespace}'")
@@ -520,6 +523,9 @@ def remove_group(resource: Dict):
         print_warning(f"Group '{group_name}' not found")
         return
 
+    if not confirmed(f"Remove Group '{group_name}' ({group_id})?"):
+        return
+
     try:
         CLIENT.account_client.delete_group(group_id)
         print_log(f"Removed Group '{group_name}' ({group_id})")
@@ -543,12 +549,15 @@ def remove_application(resource: Dict):
         print_warning(f"Application '{app_name}' not found")
         return
 
+    if not confirmed(f"Remove Application '{app_name}' ({app_id})?"):
+        return
+
     try:
         CLIENT.account_client.delete_application(app_id)
         print_log(f"Removed Application '{app_name}' ({app_id})")
         clear_application_caches()
     except Exception as e:
-        print_error(f"Unable to delete Application '{app_name}' ({app_id}): {e}")
+        print_error(f"Unable to remove Application '{app_name}' ({app_id}): {e}")
 
 
 # Entry point

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/show.py

@@ -8,11 +8,18 @@ from yellowdog_client.model import ConfiguredWorkerPool
 
 from yellowdog_cli.list import get_keyring
 from yellowdog_cli.utils.entity_utils import (
+    get_application_groups,
     substitute_ids_for_names_in_crt,
     substitute_image_family_id_for_name_in_cst,
 )
-from yellowdog_cli.utils.printing import print_error, print_log, print_yd_object
+from yellowdog_cli.utils.printing import (
+    print_error,
+    print_log,
+    print_to_file,
+    print_yd_object,
+)
 from yellowdog_cli.utils.settings import (
+    PROP_GROUPS,
     RESOURCE_PROPERTY_NAME,
     RN_ALLOWANCE,
     RN_APPLICATION,
@@ -36,11 +43,13 @@ def main():
     # and the 'quiet' option is enabled
     generate_json_list = len(ARGS_PARSER.yellowdog_ids) > 1 and ARGS_PARSER.quiet
 
-    if ARGS_PARSER.details and ARGS_PARSER.strip_ids:
-        print_log("Omitting YellowDog IDs from detailed JSON objects")
+    if ARGS_PARSER.strip_ids:
+        print_log("Omitting YellowDog IDs (etc.) from detailed JSON objects")
 
     if generate_json_list:
         print("[")
+        if ARGS_PARSER.output_file is not None:
+            print_to_file("[", ARGS_PARSER.output_file)
 
     for index, ydid in enumerate(ARGS_PARSER.yellowdog_ids):
         if generate_json_list:
@@ -53,6 +62,8 @@ def main():
 
     if generate_json_list:
         print("]")
+        if ARGS_PARSER.output_file is not None:
+            print_to_file("]", ARGS_PARSER.output_file)
 
 
 def show_details(ydid: str, initial_indent: int = 0, with_final_comma: bool = False):
@@ -210,11 +221,17 @@ def show_details(ydid: str, initial_indent: int = 0, with_final_comma: bool = Fa
 
         elif ydid_type == YDIDType.APPLICATION:
             print_log(f"Showing details of Application ID '{ydid}'")
+            group_names = [group.name for group in get_application_groups(CLIENT, ydid)]
             print_yd_object(
                 CLIENT.account_client.get_application(ydid),
                 initial_indent=initial_indent,
                 with_final_comma=with_final_comma,
-                add_fields=({RESOURCE_PROPERTY_NAME: RN_APPLICATION}),
+                add_fields=(
+                    {
+                        PROP_GROUPS: group_names,
+                        RESOURCE_PROPERTY_NAME: RN_APPLICATION,
+                    }
+                ),
             )
 
         elif ydid_type == YDIDType.USER:

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/submit.py

@@ -23,8 +23,6 @@ from yellowdog_client.model import (
     RunSpecification,
     Task,
     TaskData,
-    TaskErrorMatcher,
-    TaskErrorType,
     TaskGroup,
     TaskInput,
     TaskInputSource,
@@ -56,7 +54,6 @@ from yellowdog_cli.utils.printing import (
     print_log,
     print_numbered_strings,
     print_warning,
-    print_yd_object,
 )
 from yellowdog_cli.utils.property_names import *
 from yellowdog_cli.utils.settings import (

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/utils/args.py

@@ -107,7 +107,6 @@ class CLIParser:
             required=False,
             help="disable colouring and text wrapping in command output",
         )
-
         parser.add_argument(
             "--quiet",
             "-q",
@@ -115,6 +114,12 @@ class CLIParser:
             required=False,
             help="suppress (non-error, non-interactive) status and progress messages",
         )
+        parser.add_argument(
+            "--env-override",
+            action="store_true",
+            required=False,
+            help="values in '.env' file override values in the environment",
+        )
 
         # Module-specific argument sets
         if not any(module in sys.argv[0] for module in ["compare"]):
@@ -663,16 +668,6 @@ class CLIParser:
                 required=False,
                 help="show the full JSON representation of objects",
             )
-            parser.add_argument(
-                "--output-file",
-                type=str,
-                required=False,
-                help=(
-                    "if specified, the 'details' JSON resource listing will also be written "
-                    "to the nominated output file"
-                ),
-                metavar="<output-file>",
-            )
             parser.add_argument(
                 "--auto-select-all",
                 action="store_true",
@@ -1194,6 +1189,16 @@ class CLIParser:
                     "(implies '--details')"
                 ),
             )
+            parser.add_argument(
+                "--output-file",
+                type=str,
+                required=False,
+                help=(
+                    "if specified, the detailed JSON resource listing will also be written "
+                    "to the nominated output file"
+                ),
+                metavar="<output-file>",
+            )
 
         if "compare" in sys.argv[0]:
             parser.add_argument(
@@ -1265,6 +1270,11 @@ class CLIParser:
     def quiet(self) -> Optional[bool]:
         return self.args.quiet
 
+    @property
+    @allow_missing_attribute
+    def env_override(self) -> Optional[bool]:
+        return self.args.env_override
+
     @property
     @allow_missing_attribute
     def work_req_file(self) -> Optional[str]:

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/utils/csv_data.py

@@ -8,7 +8,7 @@ from ast import literal_eval
 from collections import OrderedDict
 from json import load as json_load
 from os.path import relpath
-from typing import Dict, List, Optional
+from typing import Dict, List, Optional, Tuple
 
 from toml import load as toml_load
 
@@ -290,7 +290,7 @@ USED_FILE_INDEXES = []
 
 def get_csv_file_index(
     csv_filename: str, task_groups: List[Dict]
-) -> [str, Optional[int]]:
+) -> Tuple[str, Optional[int]]:
     """
     Check if the CSV filename ends in an integer index (':<integer>'),
     or in a Task Group name (':<task_group_name>').
@@ -378,8 +378,6 @@ def csv_expand_toml_tasks(
         (config_wr.add_yd_env_vars, ADD_YD_ENV_VARS),
         (config_wr.always_upload, ALWAYS_UPLOAD),
         (config_wr.args, ARGS),
-        (config_wr.task_data_inputs, TASK_DATA_INPUTS),
-        (config_wr.task_data_outputs, TASK_DATA_OUTPUTS),
         (config_wr.docker_env, DOCKER_ENV),
         (config_wr.docker_options, DOCKER_OPTIONS),
         (config_wr.docker_password, DOCKER_PASSWORD),
@@ -393,8 +391,11 @@ def csv_expand_toml_tasks(
         (config_wr.outputs_optional, OUTPUTS_OPTIONAL),
         (config_wr.outputs_other, OUTPUTS_OTHER),
         (config_wr.outputs_required, OUTPUTS_REQUIRED),
+        (config_wr.set_task_names, SET_TASK_NAMES),
         (config_wr.task_data, TASK_DATA),
         (config_wr.task_data_file, TASK_DATA_FILE),
+        (config_wr.task_data_inputs, TASK_DATA_INPUTS),
+        (config_wr.task_data_outputs, TASK_DATA_OUTPUTS),
         (config_wr.task_group_name, TASK_GROUP_NAME),  # Note: oddity
         (config_wr.task_level_timeout, TASK_LEVEL_TIMEOUT),
         (config_wr.task_name, TASK_NAME),

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/utils/load_config.py

@@ -9,7 +9,6 @@ from pathlib import Path
 from sys import exit
 from typing import Dict
 
-from dotenv import dotenv_values, find_dotenv, load_dotenv
 from toml import TomlDecodeError
 
 from yellowdog_cli.utils.args import ARGS_PARSER
@@ -18,7 +17,14 @@ from yellowdog_cli.utils.config_types import (
     ConfigWorkerPool,
     ConfigWorkRequirement,
 )
-from yellowdog_cli.utils.misc_utils import pathname_relative_to_config_file
+from yellowdog_cli.utils.misc_utils import (
+    load_dotenv_file,
+    pathname_relative_to_config_file,
+)
+
+# Load additional environment variables as early as possible
+load_dotenv_file()
+
 from yellowdog_cli.utils.printing import print_error, print_log
 from yellowdog_cli.utils.property_names import *
 from yellowdog_cli.utils.settings import (
@@ -102,10 +108,6 @@ def load_config_common() -> ConfigCommon:
             common_section_imported.update(common_section)
             common_section = common_section_imported
 
-        # Load extra environment variables from a .env file if it exists;
-        # do not override existing variables (environment takes precedence)
-        _load_dotenv()
-
         # Replace common section properties with command line or
         # environment variable overrides. Precedence is:
         # command line > environment variable > config file
@@ -453,29 +455,3 @@ def load_config_worker_pool() -> ConfigWorkerPool:
     except ValueError as e:
         print_error(f"Invalid type for configuration: {e}")
         exit(1)
-
-
-def _load_dotenv():
-    """
-    Load extra environment variables from a .env file if it exists.
-    Do not override existing variables (environment takes precedence).
-    Report on YD vars that are taken from .env.
-    """
-    dotenv_file = find_dotenv()
-    if dotenv_file == "":
-        return
-
-    dotenv_yd_substitutions = [  # Find 'YD' variables
-        f"'{key}'"
-        for key in dotenv_values(dotenv_file).keys()
-        if key.startswith("YD") and os.environ.get(key) is None
-    ]
-
-    if len(dotenv_yd_substitutions) > 0:
-        print_log(
-            f"Adding 'YD' environment variables from '.env' file '{dotenv_file}': "
-            f"{', '.join(dotenv_yd_substitutions)}"
-        )
-
-    # Actually load the variables (including non-'YD' variables)
-    load_dotenv(dotenv_file, override=False)

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/utils/misc_utils.py

@@ -2,6 +2,7 @@
 General utility functions.
 """
 
+import os
 import re
 from dataclasses import dataclass
 from datetime import datetime, timezone
@@ -9,6 +10,7 @@ from os.path import join, normpath, relpath
 from typing import List, Optional
 from urllib.parse import urlparse
 
+from dotenv import dotenv_values, find_dotenv, load_dotenv
 from yellowdog_client.model import (
     ComputeRequirement,
     ConfiguredWorkerPool,
@@ -16,6 +18,8 @@ from yellowdog_client.model import (
     WorkRequirement,
 )
 
+from yellowdog_cli.utils.args import ARGS_PARSER
+from yellowdog_cli.utils.printing import print_log
 from yellowdog_cli.utils.settings import NAMESPACE_OBJECT_STORE_PREFIX_SEPARATOR
 
 UTCNOW = datetime.now(timezone.utc)
@@ -236,3 +240,31 @@ def format_yd_name(yd_name: str, add_prefix: bool = True) -> str:
 
     # Mustn't exceed 60 chars
     return new_yd_name[:60]
+
+
+def load_dotenv_file():
+    """
+    Load extra environment variables from a .env file if it exists.
+    Do not override existing variables (environment takes precedence)
+    unless --env-override option is set.
+    Report on YD vars that are taken from .env.
+    """
+    dotenv_file = find_dotenv()
+    if dotenv_file == "":
+        return
+
+    dotenv_yd_substitutions = [  # Find 'YD' variables
+        f"'{key}'"
+        for key in dotenv_values(dotenv_file).keys()
+        if key.startswith("YD")
+        and (os.environ.get(key) is None or ARGS_PARSER.env_override)
+    ]
+
+    if len(dotenv_yd_substitutions) > 0:
+        print_log(
+            f"Adding 'YD' environment variables from '.env' file '{dotenv_file}': "
+            f"{', '.join(dotenv_yd_substitutions)}"
+        )
+
+    # Actually load the variables (including non-'YD' variables)
+    load_dotenv(dotenv_file, override=ARGS_PARSER.env_override)

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/utils/printing.py

@@ -71,10 +71,12 @@ from yellowdog_cli.utils.settings import (
     MAX_LINES_COLOURED_FORMATTING,
     MAX_TABLE_DESCRIPTION,
     NAMESPACE_OBJECT_STORE_PREFIX_SEPARATOR,
+    PROP_ACCESS_DELEGATES,
+    PROP_ADMIN_GROUP,
+    PROP_CREATED_BY_USER_ID,
     PROP_CREATED_TIME,
     PROP_ID,
     PROP_INSTANCE_PRICING,
-    PROP_RESOURCE,
     PROP_SUPPORTING_RESOURCE_CREATED,
     PROP_TRAITS,
     WARNING_STYLE,
@@ -98,10 +100,13 @@ class PrintLogHighlighter(RegexHighlighter):
         r"(?P<date_time>[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"
         r" [0-9][0-9]:[0-9][0-9]:[0-9][0-9])",
         r"(?P<quoted>'[a-zA-Z0-9-._=;:\/\\\[\]{}+#@$£%\^&\*\(\)~`<>?]*')",
-        r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9abcdef-]*)",
         r"(?P<ydid>ydid:[a-z]*:[0-9abcdef-]*)",
+        r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9abcdef-]*)",
         r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9abcdef-]*:[0-9]*)",
         r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9abcdef-]*:[0-9]*:[0-9]*)",
+        r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9ABCDEF]*:[0-9abcdef-]*)",
+        r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9ABCDEF]*:[0-9abcdef-]*:[0-9]*)",
+        r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9ABCDEF]*:[0-9abcdef-]*:[0-9]*:[0-9]*)",
         r"(?P<url>(https?):((//)|(\\\\))+[\w\d:#@%/;$~_?\+-=\\\.&]*)",
     ] + HIGHLIGHTED_STATES
 
@@ -120,6 +125,9 @@ class PrintTableHighlighter(RegexHighlighter):
         r"(?P<ydid>ydid:[a-z]*:[0-9abcdef-]*)",
         r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9abcdef-]*:[0-9]*)",
         r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9abcdef-]*:[0-9]*:[0-9]*)",
+        r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9ABCDEF]*:[0-9abcdef-]*)",
+        r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9ABCDEF]*:[0-9abcdef-]*:[0-9]*)",
+        r"(?P<ydid>ydid:[a-z]*:[0-9ABCDEF]*:[0-9ABCDEF]*:[0-9abcdef-]*:[0-9]*:[0-9]*)",
     ] + HIGHLIGHTED_STATES
 
 
@@ -227,10 +235,12 @@ def print_warning(
 TYPE_MAP = {
     AWSAvailabilityZone: "AWS Availability Zones",
     Allowance: "Allowance",
+    Application: "Application",
     ComputeRequirement: "Compute Requirement",
     ComputeRequirementTemplateSummary: "Compute Requirement Template",
     ComputeSourceTemplateSummary: "Compute Source Template",
     ConfiguredWorkerPool: "Configured Worker Pool",
+    Group: "Group",
     KeyringSummary: "Keyring",
     MachineImageFamilySummary: "Machine Image Family",
     NamespacePolicy: "Namespace Policy",
@@ -238,8 +248,10 @@ TYPE_MAP = {
     ObjectPath: "Object Path",
     PermissionDetail: "Permission",
     ProvisionedWorkerPool: "Provisioned Worker Pool",
+    Role: "Role",
     Task: "Task",
     TaskGroup: "Task Group",
+    User: "User",
     WorkRequirementSummary: "Work Requirement",
     Worker: "Worker",
     WorkerPoolSummary: "Worker Pool",
@@ -1048,7 +1060,7 @@ def print_json(
             CONSOLE_JSON.print(json_string, soft_wrap=True)
 
     if ARGS_PARSER.output_file is not None:  # Also output to a nominated file
-        _print_to_file(
+        print_to_file(
             json_string=json_string,
             output_file=ARGS_PARSER.output_file,
             with_final_comma=with_final_comma,
@@ -1081,6 +1093,9 @@ def print_yd_object(
                 if k
                 not in [
                     PROP_ID,
+                    PROP_ACCESS_DELEGATES,
+                    PROP_ADMIN_GROUP,
+                    PROP_CREATED_BY_USER_ID,
                     PROP_CREATED_TIME,
                     PROP_INSTANCE_PRICING,
                     PROP_SUPPORTING_RESOURCE_CREATED,
@@ -1111,7 +1126,7 @@ def print_yd_object(
 
 
 def print_yd_object_list(
-    objects: List[Tuple[Any, Optional[str]]],  # Tuples are (object, resource_type_name)
+    objects: List[Tuple[Any, Optional[Dict]]],
 ):
     """
     Print a JSON list of objects.
@@ -1123,24 +1138,20 @@ def print_yd_object_list(
     if len(objects) > 1:
         print("[")
         if ARGS_PARSER.output_file is not None:
-            _print_to_file("[", ARGS_PARSER.output_file)
+            print_to_file("[", ARGS_PARSER.output_file)
 
-    for index, (object_, resource_type_name) in enumerate(objects):
+    for index, (object_, add_fields) in enumerate(objects):
         print_yd_object(
             object_,
             initial_indent=2 if len(objects) > 1 else 0,
             with_final_comma=(True if index < len(objects) - 1 else False),
-            add_fields=(
-                {}
-                if resource_type_name is None
-                else {PROP_RESOURCE: resource_type_name}
-            ),
+            add_fields=add_fields,
         )
 
     if len(objects) > 1:
         print("]")
         if ARGS_PARSER.output_file is not None:
-            _print_to_file("]", ARGS_PARSER.output_file)
+            print_to_file("]", ARGS_PARSER.output_file)
 
 
 def print_worker_pool(
@@ -1501,7 +1512,7 @@ def print_event(event: str, id_type: YDIDType):
 FIRST_OUTPUT_TO_FILE = True  # Determine whether to 'write' or 'append'
 
 
-def _print_to_file(json_string: str, output_file: str, with_final_comma: bool = False):
+def print_to_file(json_string: str, output_file: str, with_final_comma: bool = False):
     """
     Dump details output to a file.
     """

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_cli/utils/settings.py

@@ -133,7 +133,10 @@ RN_UPDATE_GROUP_REQUEST = "UpdateGroupRequest"
 RN_USER = "User"
 
 # Property Names
+PROP_ACCESS_DELEGATES = "accessDelegates"
+PROP_ADMIN_GROUP = "adminGroup"
 PROP_AUTOSCALING_MAX_NODES = "autoscalingMaxNodes"
+PROP_CREATED_BY_USER_ID = "createdByUserId"
 PROP_CREATED_TIME = "createdTime"
 PROP_CREDENTIAL = "credential"
 PROP_CST_ID = "sourceTemplateId"

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6/yellowdog_python_examples.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: yellowdog-python-examples
-Version: 7.18.4
+Version: 7.18.6
 Summary: Python CLI commands using the YellowDog Python SDK
 Author-email: YellowDog Limited <support@yellowdog.co>
 Project-URL: Homepage, https://github.com/yellowdog/python-examples
@@ -20,7 +20,7 @@ Requires-Dist: requests
 Requires-Dist: rich==13.9.4
 Requires-Dist: tabulate>=0.9.0
 Requires-Dist: toml
-Requires-Dist: yellowdog-sdk>=9.3.3
+Requires-Dist: yellowdog-sdk>=9.4.0
 Provides-Extra: jsonnet
 Requires-Dist: jsonnet; extra == "jsonnet"
 Provides-Extra: cloudwizard

{yellowdog_python_examples-7.18.4 → yellowdog_python_examples-7.18.6}/yellowdog_python_examples.egg-info/requires.txt

@@ -5,7 +5,7 @@ requests
 rich==13.9.4
 tabulate>=0.9.0
 toml
-yellowdog-sdk>=9.3.3
+yellowdog-sdk>=9.4.0
 
 [cloudwizard]
 boto3

yellowdog_python_examples-7.18.4/yellowdog_cli/__init__.py

@@ -1 +0,0 @@
-__version__ = "7.18.4"