yellowdog-python-examples 7.18.1__tar.gz → 7.18.3__tar.gz

{yellowdog_python_examples-7.18.1/yellowdog_python_examples.egg-info → yellowdog_python_examples-7.18.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: yellowdog-python-examples
-Version: 7.18.1
+Version: 7.18.3
 Summary: Python CLI commands using the YellowDog Python SDK
 Author-email: YellowDog Limited <support@yellowdog.co>
 Project-URL: Homepage, https://github.com/yellowdog/python-examples
@@ -15,11 +15,12 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: PyPAC>=0.16.4
 Requires-Dist: dateparser
+Requires-Dist: dotenv
 Requires-Dist: requests
 Requires-Dist: rich==13.9.4
 Requires-Dist: tabulate>=0.9.0
 Requires-Dist: toml
-Requires-Dist: yellowdog-sdk>=9.2.3
+Requires-Dist: yellowdog-sdk>=9.3.3
 Provides-Extra: jsonnet
 Requires-Dist: jsonnet; extra == "jsonnet"
 Provides-Extra: cloudwizard

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/README.md

@@ -10,9 +10,10 @@
 * [Configuration](#configuration)
 * [Naming Rules](#naming-rules)
 * [Common Properties](#common-properties)
-   * [Importing common properties](#importing-common-properties)
+   * [Importing common Properties](#importing-common-properties)
    * [HTTPS Proxy Support](#https-proxy-support)
    * [Specifying Common Properties using the Command Line or Environment Variables](#specifying-common-properties-using-the-command-line-or-environment-variables)
+   * [Support for .env Files](#support-for-env-files)
    * [Variable Substitutions in Common Properties](#variable-substitutions-in-common-properties)
 * [Variable Substitutions](#variable-substitutions)
    * [Default Variables](#default-variables)
@@ -99,7 +100,7 @@
    * [Namespace Policies](#namespace-policies)
    * [Groups](#groups)
    * [Applications](#applications)
-      * [Creating and Regenrating Application Keys](#creating-and-regenrating-application-keys)
+      * [Creating and Regenerating Application Keys](#creating-and-regenerating-application-keys)
    * [Users](#users)
 * [Jsonnet Support](#jsonnet-support)
    * [Jsonnet Installation](#jsonnet-installation)
@@ -130,7 +131,7 @@
    * [yd-compare](#yd-compare)
 
 <!-- Created by https://github.com/ekalinin/github-markdown-toc -->
-<!-- Added by: pwt, at: Fri May 23 08:51:46 BST 2025 -->
+<!-- Added by: pwt, at: Mon Jul 28 21:41:59 BST 2025 -->
 
 <!--te-->
 
@@ -161,7 +162,7 @@ The commands provide the following capabilities:
 - **Listing** YellowDog items using the **`yd-list`** command
 - **Resizing** Worker Pools and Compute Requirements with the **`yd-resize`** command
 - **Boosting** Allowances with the **`yd-boost`** command
-- **Creating, Updating and Removing** Source Templates, Compute Templates, Keyrings, Credentials, Namespace Storage Configurations, Image Families, Allowances, Configured Worker Pools, User Attributes, and Namespace Policies with the **`yd-create`** and **`yd-remove`** commands
+- **Creating, Updating and Removing** Compute Source Templates, Compute Requirement Templates, Keyrings, Credentials, Storage Configurations, Image Families, Allowances, Configured Worker Pools, User Attributes, Namespace Policies, Groups, and Applications with the **`yd-create`** and **`yd-remove`** commands
 - **Following Event Streams** for Work Requirements, Worker Pools and Compute Requirements with the **`yd-follow`** command
 - **Showing** the details of any YellowDog entity using its YellowDog ID with the **`yd-show`** command
 - **Comparing** whether worker pools are a match for task groups with the **`yd-compare`** command
@@ -328,7 +329,7 @@ An example `common` section is shown below:
 
 Indentation is optional in TOML files and is for readability only.
 
-## Importing `common` properties
+## Importing `common` Properties
 
 The `common` section can import properties from a separate TOML file, using the `importCommon` property. For example, the `key` and `secret` might be in a shared TOML file called `app_credentials.toml`, with the following contents:
 
@@ -383,6 +384,12 @@ When setting the value of the above properties, a property set on the command li
 
 If all the required common properties are set using the command line or environment variables, then the entire `common` section of the TOML file can be omitted.
 
+## Support for `.env` Files
+
+Environment variables can also be set in a `.env` file, typically in the user's home directory. Entries in the `.env` file will not overwrite existing environment variables -- i.e., environment variables take precedence over entries in the `.env` file.
+
+Environment variables sourced from a `.env` file whose names start with `YD` will be reported on the command line. Variables that do not start with `YD` will not be reported, but they will still be applied.
+
 ## Variable Substitutions in Common Properties
 
 Note the use of `{{username}}` in the value of the `tag` property example above: this is a **variable substitution** that can optionally be used to insert the login username of the user running the commands. So, for username `abc`, the `tag` would be set to `testing-abc`. This can be helpful to disambiguate multiple users running with the same configuration data.
@@ -464,7 +471,8 @@ The precedence order for setting variables is:
 1. Command line
 2. `YD_VAR` environment variables
 3. General environment variables
-4. TOML file
+4. Variables in a `.env` file
+5. TOML file
 
 This method can also be used to override the default variables, e.g., setting `-v username="other-user"` will override the default `{{username}}` variable.
 
@@ -824,6 +832,9 @@ Here's an example of the `workRequirement` section of a TOML configuration file,
     providers = ["AWS"]
     ram = [0.5, 2.0]
     regions = ["eu-west-2"]
+    retryableErrors = [
+      {processExitCodes = [143], statusesAtFailure = ["FAILED"], errorTypes = ["ALLOCATION_LOST"]},
+    ]
     setTaskNames = false
     tag = "my_tag"
     taskBatchSize = 1000
@@ -844,9 +855,6 @@ Here's an example of the `workRequirement` section of a TOML configuration file,
     taskTimeout = 120.0
     taskType = "docker"
     tasksPerWorker = 1
-    retryableErrors = [
-        {processExitCodes = [143], statusesAtFailure = ["FAILED"], errorTypes = ["ALLOCATION_LOST"]},
-    ]
     uploadFiles = [{localPath = "file_1.txt", uploadPath = "file_1.txt"}]
     uploadTaskProcessOutput = true
     vcpus = [1, 4]
@@ -892,6 +900,13 @@ Showing all possible properties at the Work Requirement level:
   "providers": ["AWS"],
   "ram": [0.5, 2],
   "regions": ["eu-west-2"],
+  "retryableErrors": [
+    {
+      "processExitCodes": [143],
+      "statusesAtFailure" : ["FAILED"],
+      "errorTypes": ["ALLOCATION_LOST"]
+    }
+  ]
   "setTaskNames": false,
   "tag": "my_tag"
   "taskCount": 100,
@@ -906,13 +921,6 @@ Showing all possible properties at the Work Requirement level:
     {"alwaysUpload": false, "destination": "dest_path_2", "source": "out_src_path_2"}
   ],
   "taskGroupCount": 5,
-  "retryableErrors": [
-    {
-      "processExitCodes": [143],
-      "statusesAtFailure" : ["FAILED"],
-      "errorTypes": ["ALLOCATION_LOST"]
-    }
-  ]
   "taskTimeout": 120.0,
   "taskTypes": ["docker"],
   "tasksPerWorker": 1,
@@ -970,6 +978,13 @@ Showing all possible properties at the Task Group level:
       "providers": ["AWS"],
       "ram": [0.5, 2],
       "regions": ["eu-west-2"],
+      "retryableErrors": [
+        {
+          "processExitCodes": [143],
+          "statusesAtFailure" : ["FAILED"],
+          "errorTypes": ["ALLOCATION_LOST"]
+        }
+      ]
       "setTaskNames": false,
       "tag": "my_tag",
       "taskCount": 5,
@@ -983,13 +998,6 @@ Showing all possible properties at the Task Group level:
         {"alwaysUpload": true, "destination": "dest_path_1", "source": "out_src_path_1"},
         {"alwaysUpload": false, "destination": "dest_path_2", "source": "out_src_path_2"}
       ],
-      "retryableErrors": [
-        {
-          "processExitCodes": [143],
-          "statusesAtFailure" : ["FAILED"],
-          "errorTypes": ["ALLOCATION_LOST"]
-        }
-      ]
       "taskTimeout": 120.0,
       "taskTypes": ["docker"],
       "tasksPerWorker": 1,

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/requirements.txt

@@ -1,7 +1,8 @@
 PyPAC >= 0.16.4
 dateparser
+dotenv
 requests
 rich == 13.9.4
 tabulate >= 0.9.0
 toml
-yellowdog-sdk >= 9.2.3
+yellowdog-sdk >= 9.3.3

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/tests/test_demos.py

@@ -63,7 +63,7 @@ class TestDemos:
 
     def test_montecarlo_graviton(self):
         result = shell(
-            f"cd {DEMO_DIR}/montecarlo && yd-provision -c config_graviton.toml && yd-submit -f && yd-terminate -y && yd-delete -y"
+            f"cd {DEMO_DIR}/montecarlo && yd-provision -c config-graviton.toml && yd-submit -f && yd-terminate -y && yd-delete -y"
         )
         assert result.exit_code == 0
 

yellowdog_python_examples-7.18.3/yellowdog_cli/__init__.py

@@ -0,0 +1 @@
+__version__ = "7.18.3"

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_cli/list.py

@@ -93,6 +93,18 @@ def main():
     # Always use interactive mode for selections
     ARGS_PARSER.interactive = True
 
+    if (
+        ARGS_PARSER.auto_select_all
+        or ARGS_PARSER.strip_ids
+        or ARGS_PARSER.substitute_ids
+        or ARGS_PARSER.output_file
+    ) and not ARGS_PARSER.details:
+        print_log("Automatically setting the '--details' option")
+        ARGS_PARSER.details = True
+
+    if ARGS_PARSER.details and ARGS_PARSER.strip_ids:
+        print_log("Omitting YellowDog IDs from detailed JSON objects")
+
     if ARGS_PARSER.output_file and ARGS_PARSER.details:
         if exists(ARGS_PARSER.output_file):
             if not confirmed(
@@ -108,9 +120,9 @@ def main():
         list_worker_pools()
     elif ARGS_PARSER.compute_requirements or ARGS_PARSER.instances:
         list_compute_requirements()
-    elif ARGS_PARSER.compute_templates:
+    elif ARGS_PARSER.compute_requirement_templates:
         list_compute_requirement_templates()
-    elif ARGS_PARSER.source_templates:
+    elif ARGS_PARSER.compute_source_templates:
         list_compute_source_templates()
     elif ARGS_PARSER.keyrings:
         list_keyrings()
@@ -145,7 +157,7 @@ def check_for_valid_option() -> bool:
         ARGS_PARSER.applications,
         ARGS_PARSER.attribute_definitions,
         ARGS_PARSER.compute_requirements,
-        ARGS_PARSER.compute_templates,
+        ARGS_PARSER.compute_requirement_templates,
         ARGS_PARSER.groups,
         ARGS_PARSER.image_families,
         ARGS_PARSER.instances,
@@ -156,7 +168,7 @@ def check_for_valid_option() -> bool:
         ARGS_PARSER.object_paths,
         ARGS_PARSER.permissions,
         ARGS_PARSER.roles,
-        ARGS_PARSER.source_templates,
+        ARGS_PARSER.compute_source_templates,
         ARGS_PARSER.task_groups,
         ARGS_PARSER.tasks,
         ARGS_PARSER.users,

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_cli/show.py

@@ -33,6 +33,9 @@ def main():
     # and the 'quiet' option is enabled
     generate_json_list = len(ARGS_PARSER.yellowdog_ids) > 1 and ARGS_PARSER.quiet
 
+    if ARGS_PARSER.details and ARGS_PARSER.strip_ids:
+        print_log("Omitting YellowDog IDs from detailed JSON objects")
+
     if generate_json_list:
         print("[")
 

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_cli/utils/args.py

@@ -494,13 +494,6 @@ class CLIParser:
                     " level prefixes"
                 ),
             )
-            parser.add_argument(
-                "--details",
-                "-d",
-                action="store_true",
-                required=False,
-                help="show the full details of (interactively) selected objects",
-            )
             parser.add_argument(
                 "--work-requirements",
                 "-w",
@@ -547,14 +540,14 @@ class CLIParser:
                 ),
             )
             parser.add_argument(
-                "--compute-templates",
+                "--compute-requirement-templates",
                 "-C",
                 action="store_true",
                 required=False,
                 help="list compute requirement templates",
             )
             parser.add_argument(
-                "--source-templates",
+                "--compute-source-templates",
                 "-S",
                 action="store_true",
                 required=False,
@@ -657,16 +650,29 @@ class CLIParser:
                 required=False,
                 help="list permissions",
             )
+            parser.add_argument(
+                "--details",
+                "-d",
+                action="store_true",
+                required=False,
+                help="show the full JSON representation of objects",
+            )
             parser.add_argument(
                 "--output-file",
                 type=str,
                 required=False,
                 help=(
-                    "if specified, the 'details' resource listing will be appended "
+                    "if specified, the 'details' JSON resource listing will also be written "
                     "to the nominated output file"
                 ),
                 metavar="<output-file>",
             )
+            parser.add_argument(
+                "--auto-select-all",
+                action="store_true",
+                required=False,
+                help="use with '--details' to select all objects automatically",
+            )
 
         if any(module in sys.argv[0] for module in ["upload"]):
             parser.add_argument(
@@ -1165,8 +1171,17 @@ class CLIParser:
                 action="store_true",
                 required=False,
                 help=(
-                    "substitute compute source template IDs and image family IDs for "
-                    "names in detailed compute requirement templates"
+                    "use with '--details' to substitute compute source template IDs and "
+                    "image family IDs for names in detailed compute requirement templates"
+                ),
+            )
+            parser.add_argument(
+                "--strip-ids",
+                action="store_true",
+                required=False,
+                help=(
+                    "use with '--details' to omit the YellowDog IDs of objects from "
+                    "their JSON representations"
                 ),
             )
 
@@ -1334,6 +1349,10 @@ class CLIParser:
     def details(self) -> Optional[bool]:
         return self.args.details
 
+    @details.setter
+    def details(self, interactive: bool):
+        self.args.details = interactive
+
     @property
     @allow_missing_attribute
     def debug(self) -> Optional[bool]:
@@ -1491,13 +1510,13 @@ class CLIParser:
 
     @property
     @allow_missing_attribute
-    def compute_templates(self) -> Optional[bool]:
-        return self.args.compute_templates
+    def compute_requirement_templates(self) -> Optional[bool]:
+        return self.args.compute_requirement_templates
 
     @property
     @allow_missing_attribute
-    def source_templates(self) -> Optional[bool]:
-        return self.args.source_templates
+    def compute_source_templates(self) -> Optional[bool]:
+        return self.args.compute_source_templates
 
     @property
     @allow_missing_attribute
@@ -1734,6 +1753,16 @@ class CLIParser:
     def running_nodes_only(self) -> Optional[bool]:
         return self.args.running_nodes_only
 
+    @property
+    @allow_missing_attribute
+    def strip_ids(self) -> Optional[bool]:
+        return self.args.strip_ids
+
+    @property
+    @allow_missing_attribute
+    def auto_select_all(self) -> Optional[bool]:
+        return self.args.auto_select_all
+
 
 def lookup_module_description(module_name: str) -> Optional[str]:
     """

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_cli/utils/interactive.py

@@ -63,6 +63,10 @@ def select(
     if not ARGS_PARSER.interactive and force_interactive is False:
         return objects
 
+    if ARGS_PARSER.auto_select_all:
+        print_log("Automatically selecting all objects")
+        return objects
+
     return [
         objects[x - 1]
         for x in get_selected_list_items(

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_cli/utils/load_config.py

@@ -9,6 +9,7 @@ from pathlib import Path
 from sys import exit
 from typing import Dict
 
+from dotenv import dotenv_values, find_dotenv, load_dotenv
 from toml import TomlDecodeError
 
 from yellowdog_cli.utils.args import ARGS_PARSER
@@ -101,6 +102,10 @@ def load_config_common() -> ConfigCommon:
             common_section_imported.update(common_section)
             common_section = common_section_imported
 
+        # Load extra environment variables from a .env file if it exists;
+        # do not override existing variables (environment takes precedence)
+        _load_dotenv()
+
         # Replace common section properties with command line or
         # environment variable overrides. Precedence is:
         # command line > environment variable > config file
@@ -445,3 +450,29 @@ def load_config_worker_pool() -> ConfigWorkerPool:
     except ValueError as e:
         print_error(f"Invalid type for configuration: {e}")
         exit(1)
+
+
+def _load_dotenv():
+    """
+    Load extra environment variables from a .env file if it exists.
+    Do not override existing variables (environment takes precedence).
+    Report on YD vars that are taken from .env.
+    """
+    dotenv_file = find_dotenv()
+    if dotenv_file == "":
+        return
+
+    dotenv_yd_substitutions = [  # Find 'YD' variables
+        f"'{key}'"
+        for key in dotenv_values(dotenv_file).keys()
+        if key.startswith("YD") and os.environ.get(key) is None
+    ]
+
+    if len(dotenv_yd_substitutions) > 0:
+        print_log(
+            f"Adding 'YD' environment variables from '.env' file '{dotenv_file}': "
+            f"{', '.join(dotenv_yd_substitutions)}"
+        )
+
+    # Actually load the variables (including non-'YD' variables)
+    load_dotenv(dotenv_file, override=False)

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_cli/utils/printing.py

@@ -71,6 +71,8 @@ from yellowdog_cli.utils.settings import (
     MAX_LINES_COLOURED_FORMATTING,
     MAX_TABLE_DESCRIPTION,
     NAMESPACE_OBJECT_STORE_PREFIX_SEPARATOR,
+    PROP_CREATED_TIME,
+    PROP_ID,
     PROP_RESOURCE,
     WARNING_STYLE,
 )
@@ -1062,6 +1064,30 @@ def print_yd_object(
     using the compact JSON encoder.
     """
     object_data: object = Json.dump(yd_object)
+
+    def remove_unused_props(d):
+        """
+        Helper function to remove the 'id' and 'createdTime' properties
+        recursively.
+        """
+        if isinstance(d, dict):
+            # Create a new dictionary omitting the 'id' and 'createdTime'
+            # keys
+            return {
+                k: remove_unused_props(v)
+                for k, v in d.items()
+                if k not in [PROP_ID, PROP_CREATED_TIME]
+            }
+        elif isinstance(d, list):
+            # Recursively process each item in the list
+            return [remove_unused_props(item) for item in d]
+        else:
+            # Return non-dict/list values unchanged
+            return d
+
+    if ARGS_PARSER.strip_ids:
+        object_data = remove_unused_props(object_data)
+
     if add_fields is not None:
         # Requires a copy of the 'object' datatype to be made,
         # in order to insert additional fields
@@ -1071,6 +1097,7 @@ def print_yd_object(
         for key, value in add_fields.items():
             object_data_new[key] = value
         object_data = object_data_new
+
     print_json(object_data, initial_indent, drop_first_line, with_final_comma)
 
 

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_cli/utils/settings.py

@@ -134,6 +134,7 @@ RN_USER = "User"
 
 # Property Names
 PROP_AUTOSCALING_MAX_NODES = "autoscalingMaxNodes"
+PROP_CREATED_TIME = "createdTime"
 PROP_CREDENTIAL = "credential"
 PROP_CST_ID = "sourceTemplateId"
 PROP_DEFAULT_RANK_ORDER = "defaultRankOrder"

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_cli/utils/variables.py

@@ -397,6 +397,8 @@ def process_typed_variable_substitution(
                 f"(Use JSON syntax?) ({e})"
             )
 
+    return None
+
 
 def resolve_filename(files_directory: str, filename: str) -> str:
     """

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3/yellowdog_python_examples.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: yellowdog-python-examples
-Version: 7.18.1
+Version: 7.18.3
 Summary: Python CLI commands using the YellowDog Python SDK
 Author-email: YellowDog Limited <support@yellowdog.co>
 Project-URL: Homepage, https://github.com/yellowdog/python-examples
@@ -15,11 +15,12 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: PyPAC>=0.16.4
 Requires-Dist: dateparser
+Requires-Dist: dotenv
 Requires-Dist: requests
 Requires-Dist: rich==13.9.4
 Requires-Dist: tabulate>=0.9.0
 Requires-Dist: toml
-Requires-Dist: yellowdog-sdk>=9.2.3
+Requires-Dist: yellowdog-sdk>=9.3.3
 Provides-Extra: jsonnet
 Requires-Dist: jsonnet; extra == "jsonnet"
 Provides-Extra: cloudwizard

{yellowdog_python_examples-7.18.1 → yellowdog_python_examples-7.18.3}/yellowdog_python_examples.egg-info/requires.txt

@@ -1,10 +1,11 @@
 PyPAC>=0.16.4
 dateparser
+dotenv
 requests
 rich==13.9.4
 tabulate>=0.9.0
 toml
-yellowdog-sdk>=9.2.3
+yellowdog-sdk>=9.3.3
 
 [cloudwizard]
 boto3

yellowdog_python_examples-7.18.1/yellowdog_cli/__init__.py

@@ -1 +0,0 @@
-__version__ = "7.18.1"