PyPI - ciocore - Versions diffs - 8.1.1rc1__py2.py3-none-any.whl → 8.1.1rc3__py2.py3-none-any.whl - Mend - Supply Chain Defender

ciocore 8.1.1rc1py2.py3-none-any.whl → 8.1.1rc3py2.py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.

This version of ciocore might be problematic. Click here for more details.

Files changed (10) hide show

ciocore/docsite/search/search_index.json CHANGED Viewed

	@@ -1 +1 @@
1	- {"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Overview","text":"<p>This site contains the documentation for the Conductor Core Python package <code>ciocore</code>. </p> <p>The package provides two main interfaces for interacting with Conductor.</p> <ol> <li>A command-line interface.</li> <li>A Python API.</li> </ol>"},{"location":"#installation","title":"Installation","text":"<p>Since you are reading this, you already have Conductor Core installed. If not, you'll find it is available through the Companion app or directly from PyPi.</p>"},{"location":"#command-line-interface","title":"Command-line interface","text":"<p>The command-line interface is the easiest way to interact with Conductor Core. It provides a simple way to upload and download data, query available software packages, and view this documentation. Below are quick liks to the most commonly used commands.</p> <ul> <li>Uploader</li> <li>Downloader</li> <li>Supported Software</li> <li>Documentation</li> </ul>"},{"location":"#python-api","title":"Python API","text":"<p>The Python API is a more flexible way to interact with Conductor. It can be used to build custom submission tools, and to integrate Conductor into existing workflows. </p> <ul> <li>The API is documented here.</li> </ul> <p>Note</p> <p>This documentation does not cover the wider Conductor ecosystem, which consists of the web dashboard, the Companion app, the back-end render services, and the DCC submission plugins. For information on these components, and to get a conceptual overview of Conductor, please see the Conductor documentation online.</p>"},{"location":"how-to-guides/","title":"How-To Guides","text":"<p>This part of the documentation includes recipes and tutorials for common tasks and workflows using Conductor Core. If you have a suggestion for a new guide, feel free to open an issue</p>"},{"location":"how-to-guides/#how-to-create-a-submission","title":"How to create a submission?","text":"<pre><code># your_script.py\nfrom ciocore import data as coredata\n\n...\n</code></pre>"},{"location":"how-to-guides/#how-to-authenticate-with-an-api-key","title":"How to authenticate with an API key?","text":"<pre><code># your_script.py\nfrom ciocore import api_client\n\n...\n</code></pre>"},{"location":"apidoc/api_client/","title":"ciocore.api_client","text":"<p>The api_client module is used to make requests to the Conductor API.</p>"},{"location":"apidoc/api_client/#ciocore.api_client.ApiClient","title":"<code> ApiClient </code>","text":"<p>The ApiClient class is a wrapper around the requests library that handles authentication and retries.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>class ApiClient:\n\"\"\"\n The ApiClient class is a wrapper around the requests library that handles authentication and retries.\n \"\"\"\n\n http_verbs = [\"PUT\", \"POST\", \"GET\", \"DELETE\", \"HEAD\", \"PATCH\"]\n\n USER_AGENT_TEMPLATE = \"client {client_name}/{client_version} (ciocore {ciocore_version}; {runtime} {runtime_version}; {platform} {platform_details}; {hostname} {pid}; {python_path})\"\n USER_AGENT_MAX_PATH_LENGTH = 1024\n\n user_agent_header = None\n\n def __init__(self):\n logger.debug(\"\")\n\n def _make_request(self, verb, conductor_url, headers, params, data, raise_on_error=True):\n response = requests.request(\n method=verb, url=conductor_url, headers=headers, params=params, data=data\n )\n\n logger.debug(f\"verb: {verb}\")\n logger.debug(f\"conductor_url: {conductor_url}\")\n logger.debug(f\"headers: {headers}\")\n logger.debug(f\"params: {params}\")\n logger.debug(f\"data: {data}\")\n\n # If we get 300s/400s debug out the response. TODO(lws): REMOVE THIS\n if 300 <= response.status_code < 500:\n logger.debug(\"*** ERROR!! **\")\n logger.debug(f\"Reason: {response.reason}\")\n logger.debug(f\"Text: {response.text}\")\n\n # trigger an exception to be raised for 4XX or 5XX http responses\n if raise_on_error:\n response.raise_for_status()\n\n return response\n\n def make_prepared_request(\n self,\n verb,\n url,\n headers=None,\n params=None,\n json_payload=None,\n data=None,\n stream=False,\n remove_headers_list=None,\n raise_on_error=True,\n tries=5,\n ):\n\"\"\"\n Make a request to the Conductor API.\n\n Deprecated:\n Primarily used to removed enforced headers by requests.Request. Requests 2.x will add\n Transfer-Encoding: chunked with file like object that is 0 bytes, causing s3 failures (501)\n - https://github.com/psf/requests/issues/4215#issuecomment-319521235\n\n To get around this bug make_prepared_request has functionality to remove the enforced header\n that would occur when using requests.request(...). Requests 3.x resolves this issue, when\n client is built to use Requests 3.x this function can be removed.\n\n Returns:\n requests.Response: The response object.\n\n Args:\n verb (str): The HTTP verb to use.\n url (str): The URL to make the request to.\n headers (dict): A dictionary of headers to send with the request.\n params (dict): A dictionary of query parameters to send with the request.\n json (dict): A JSON payload to send with the request.\n stream (bool): Whether or not to stream the response.\n remove_headers_list (list): A list of headers to remove from the request.\n raise_on_error (bool): Whether or not to raise an exception if the request fails.\n tries (int): The number of times to retry the request.\n \"\"\"\n\n req = requests.Request(\n method=verb,\n url=url,\n headers=headers,\n params=params,\n json=json_payload,\n data=data,\n )\n prepped = req.prepare()\n\n if remove_headers_list:\n for header in remove_headers_list:\n prepped.headers.pop(header, None)\n\n # Create a retry wrapper function\n retry_wrapper = common.DecRetry(\n retry_exceptions=CONNECTION_EXCEPTIONS, tries=tries\n )\n\n # requests sessions potentially not thread-safe, but need to removed enforced\n # headers by using a prepared request.create which can only be done through an\n # request.Session object. Create Session object per call of make_prepared_request, it will\n # not benefit from connection pooling reuse. https://github.com/psf/requests/issues/1871\n session = requests.Session()\n\n # wrap the request function with the retry wrapper\n wrapped_func = retry_wrapper(session.send)\n\n # call the wrapped request function\n response = wrapped_func(prepped, stream=stream)\n\n logger.debug(\"verb: %s\", prepped.method)\n logger.debug(\"url: %s\", prepped.url)\n logger.debug(\"headers: %s\", prepped.headers)\n logger.debug(\"params: %s\", req.params)\n logger.debug(\"response: %s\", response)\n\n # trigger an exception to be raised for 4XX or 5XX http responses\n if raise_on_error:\n response.raise_for_status()\n\n return response\n\n def make_request(\n self,\n uri_path=\"/\",\n headers=None,\n params=None,\n data=None,\n verb=None,\n conductor_url=None,\n raise_on_error=True,\n tries=5,\n use_api_key=False,\n ):\n\"\"\"\n Make a request to the Conductor API.\n\n Args:\n uri_path (str): The path to the resource to request.\n headers (dict): A dictionary of headers to send with the request.\n params (dict): A dictionary of query parameters to send with the request.\n data (dict): A dictionary of data to send with the request.\n verb (str): The HTTP verb to use.\n conductor_url (str): The Conductor URL.\n raise_on_error (bool): Whether or not to raise an exception if the request fails.\n tries (int): The number of times to retry the request.\n use`_api_key (bool): Whether or not to use the API key for authentication.\n\n Returns:\n tuple(str, int): The response text and status code.\n \"\"\"\n cfg = config.get()\n # TODO: set Content Content-Type to json if data arg\n if not headers:\n headers = {\"Content-Type\": \"application/json\", \"Accept\": \"application/json\"}\n\n logger.debug(\"read_conductor_credentials({})\".format(use_api_key))\n bearer_token = read_conductor_credentials(use_api_key)\n if not bearer_token:\n raise Exception(\"Error: Could not get conductor credentials!\")\n\n headers[\"Authorization\"] = \"Bearer %s\" % bearer_token\n\n if not ApiClient.user_agent_header:\n self.register_client(\"ciocore\")\n\n headers[\"User-Agent\"] = ApiClient.user_agent_header\n\n # Construct URL\n if not conductor_url:\n conductor_url = parse.urljoin(cfg[\"url\"], uri_path)\n\n if not verb:\n if data:\n verb = \"POST\"\n else:\n verb = \"GET\"\n\n assert verb in self.http_verbs, \"Invalid http verb: %s\" % verb\n\n # Create a retry wrapper function\n retry_wrapper = common.DecRetry(\n retry_exceptions=CONNECTION_EXCEPTIONS, tries=tries\n )\n\n # wrap the request function with the retry wrapper\n wrapped_func = retry_wrapper(self._make_request)\n\n # call the wrapped request function\n response = wrapped_func(\n verb, conductor_url, headers, params, data, raise_on_error=raise_on_error\n )\n\n return response.text, response.status_code\n\n @classmethod\n def register_client(cls, client_name, client_version=None):\n\"\"\"\n Generates the http User Agent header that includes helpful debug info.\n \"\"\"\n\n # Use the provided client_version.\n if not client_version:\n client_version = 'unknown'\n\n python_version = platform.python_version()\n system_info = platform.system()\n release_info = platform.release()\n\n\n # Get the MD5 hex digest of the path to the python executable\n python_executable_path = truncate_middle(sys.executable.encode('utf-8'), cls.USER_AGENT_MAX_PATH_LENGTH)\n md5_hash = hashlib.md5(python_executable_path).hexdigest()\n\n user_agent = (\n f\"{client_name}/{client_version} \"\n f\"(python {python_version}; {system_info} {release_info}; {md5_hash})\"\n )\n cls.user_agent_header = user_agent\n\n return user_agent\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.ApiClient.make_prepared_request","title":"<code>make_prepared_request(self, verb, url, headers=None, params=None, json_payload=None, data=None, stream=False, remove_headers_list=None, raise_on_error=True, tries=5)</code>","text":"<p>Make a request to the Conductor API.</p> <p>Deprecated</p> <p>Primarily used to removed enforced headers by requests.Request. Requests 2.x will add Transfer-Encoding: chunked with file like object that is 0 bytes, causing s3 failures (501) - https://github.com/psf/requests/issues/4215#issuecomment-319521235</p> <p>To get around this bug make_prepared_request has functionality to remove the enforced header that would occur when using requests.request(...). Requests 3.x resolves this issue, when client is built to use Requests 3.x this function can be removed.</p> <p>Returns:</p> Type Description <code>requests.Response</code> <p>The response object.</p> <p>Parameters:</p> Name Type Description Default <code>verb</code> <code>str</code> <p>The HTTP verb to use.</p> required <code>url</code> <code>str</code> <p>The URL to make the request to.</p> required <code>headers</code> <code>dict</code> <p>A dictionary of headers to send with the request.</p> <code>None</code> <code>params</code> <code>dict</code> <p>A dictionary of query parameters to send with the request.</p> <code>None</code> <code>json</code> <code>dict</code> <p>A JSON payload to send with the request.</p> required <code>stream</code> <code>bool</code> <p>Whether or not to stream the response.</p> <code>False</code> <code>remove_headers_list</code> <code>list</code> <p>A list of headers to remove from the request.</p> <code>None</code> <code>raise_on_error</code> <code>bool</code> <p>Whether or not to raise an exception if the request fails.</p> <code>True</code> <code>tries</code> <code>int</code> <p>The number of times to retry the request.</p> <code>5</code> Source code in <code>ciocore/api_client.py</code> <pre><code>def make_prepared_request(\n self,\n verb,\n url,\n headers=None,\n params=None,\n json_payload=None,\n data=None,\n stream=False,\n remove_headers_list=None,\n raise_on_error=True,\n tries=5,\n):\n\"\"\"\n Make a request to the Conductor API.\n\n Deprecated:\n Primarily used to removed enforced headers by requests.Request. Requests 2.x will add\n Transfer-Encoding: chunked with file like object that is 0 bytes, causing s3 failures (501)\n - https://github.com/psf/requests/issues/4215#issuecomment-319521235\n\n To get around this bug make_prepared_request has functionality to remove the enforced header\n that would occur when using requests.request(...). Requests 3.x resolves this issue, when\n client is built to use Requests 3.x this function can be removed.\n\n Returns:\n requests.Response: The response object.\n\n Args:\n verb (str): The HTTP verb to use.\n url (str): The URL to make the request to.\n headers (dict): A dictionary of headers to send with the request.\n params (dict): A dictionary of query parameters to send with the request.\n json (dict): A JSON payload to send with the request.\n stream (bool): Whether or not to stream the response.\n remove_headers_list (list): A list of headers to remove from the request.\n raise_on_error (bool): Whether or not to raise an exception if the request fails.\n tries (int): The number of times to retry the request.\n \"\"\"\n\n req = requests.Request(\n method=verb,\n url=url,\n headers=headers,\n params=params,\n json=json_payload,\n data=data,\n )\n prepped = req.prepare()\n\n if remove_headers_list:\n for header in remove_headers_list:\n prepped.headers.pop(header, None)\n\n # Create a retry wrapper function\n retry_wrapper = common.DecRetry(\n retry_exceptions=CONNECTION_EXCEPTIONS, tries=tries\n )\n\n # requests sessions potentially not thread-safe, but need to removed enforced\n # headers by using a prepared request.create which can only be done through an\n # request.Session object. Create Session object per call of make_prepared_request, it will\n # not benefit from connection pooling reuse. https://github.com/psf/requests/issues/1871\n session = requests.Session()\n\n # wrap the request function with the retry wrapper\n wrapped_func = retry_wrapper(session.send)\n\n # call the wrapped request function\n response = wrapped_func(prepped, stream=stream)\n\n logger.debug(\"verb: %s\", prepped.method)\n logger.debug(\"url: %s\", prepped.url)\n logger.debug(\"headers: %s\", prepped.headers)\n logger.debug(\"params: %s\", req.params)\n logger.debug(\"response: %s\", response)\n\n # trigger an exception to be raised for 4XX or 5XX http responses\n if raise_on_error:\n response.raise_for_status()\n\n return response\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.ApiClient.make_request","title":"<code>make_request(self, uri_path='/', headers=None, params=None, data=None, verb=None, conductor_url=None, raise_on_error=True, tries=5, use_api_key=False)</code>","text":"<p>Make a request to the Conductor API.</p> <p>Parameters:</p> Name Type Description Default <code>uri_path</code> <code>str</code> <p>The path to the resource to request.</p> <code>'/'</code> <code>headers</code> <code>dict</code> <p>A dictionary of headers to send with the request.</p> <code>None</code> <code>params</code> <code>dict</code> <p>A dictionary of query parameters to send with the request.</p> <code>None</code> <code>data</code> <code>dict</code> <p>A dictionary of data to send with the request.</p> <code>None</code> <code>verb</code> <code>str</code> <p>The HTTP verb to use.</p> <code>None</code> <code>conductor_url</code> <code>str</code> <p>The Conductor URL.</p> <code>None</code> <code>raise_on_error</code> <code>bool</code> <p>Whether or not to raise an exception if the request fails.</p> <code>True</code> <code>tries</code> <code>int</code> <p>The number of times to retry the request.</p> <code>5</code> <code>use`_api_key</code> <code>bool</code> <p>Whether or not to use the API key for authentication.</p> required <p>Returns:</p> Type Description <code>tuple(str, int)</code> <p>The response text and status code.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def make_request(\n self,\n uri_path=\"/\",\n headers=None,\n params=None,\n data=None,\n verb=None,\n conductor_url=None,\n raise_on_error=True,\n tries=5,\n use_api_key=False,\n):\n\"\"\"\n Make a request to the Conductor API.\n\n Args:\n uri_path (str): The path to the resource to request.\n headers (dict): A dictionary of headers to send with the request.\n params (dict): A dictionary of query parameters to send with the request.\n data (dict): A dictionary of data to send with the request.\n verb (str): The HTTP verb to use.\n conductor_url (str): The Conductor URL.\n raise_on_error (bool): Whether or not to raise an exception if the request fails.\n tries (int): The number of times to retry the request.\n use`_api_key (bool): Whether or not to use the API key for authentication.\n\n Returns:\n tuple(str, int): The response text and status code.\n \"\"\"\n cfg = config.get()\n # TODO: set Content Content-Type to json if data arg\n if not headers:\n headers = {\"Content-Type\": \"application/json\", \"Accept\": \"application/json\"}\n\n logger.debug(\"read_conductor_credentials({})\".format(use_api_key))\n bearer_token = read_conductor_credentials(use_api_key)\n if not bearer_token:\n raise Exception(\"Error: Could not get conductor credentials!\")\n\n headers[\"Authorization\"] = \"Bearer %s\" % bearer_token\n\n if not ApiClient.user_agent_header:\n self.register_client(\"ciocore\")\n\n headers[\"User-Agent\"] = ApiClient.user_agent_header\n\n # Construct URL\n if not conductor_url:\n conductor_url = parse.urljoin(cfg[\"url\"], uri_path)\n\n if not verb:\n if data:\n verb = \"POST\"\n else:\n verb = \"GET\"\n\n assert verb in self.http_verbs, \"Invalid http verb: %s\" % verb\n\n # Create a retry wrapper function\n retry_wrapper = common.DecRetry(\n retry_exceptions=CONNECTION_EXCEPTIONS, tries=tries\n )\n\n # wrap the request function with the retry wrapper\n wrapped_func = retry_wrapper(self._make_request)\n\n # call the wrapped request function\n response = wrapped_func(\n verb, conductor_url, headers, params, data, raise_on_error=raise_on_error\n )\n\n return response.text, response.status_code\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.ApiClient.register_client","title":"<code>register_client(client_name, client_version=None)</code> <code>classmethod</code>","text":"<p>Generates the http User Agent header that includes helpful debug info.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>@classmethod\ndef register_client(cls, client_name, client_version=None):\n\"\"\"\n Generates the http User Agent header that includes helpful debug info.\n \"\"\"\n\n # Use the provided client_version.\n if not client_version:\n client_version = 'unknown'\n\n python_version = platform.python_version()\n system_info = platform.system()\n release_info = platform.release()\n\n\n # Get the MD5 hex digest of the path to the python executable\n python_executable_path = truncate_middle(sys.executable.encode('utf-8'), cls.USER_AGENT_MAX_PATH_LENGTH)\n md5_hash = hashlib.md5(python_executable_path).hexdigest()\n\n user_agent = (\n f\"{client_name}/{client_version} \"\n f\"(python {python_version}; {system_info} {release_info}; {md5_hash})\"\n )\n cls.user_agent_header = user_agent\n\n return user_agent\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.truncate_middle","title":"<code>truncate_middle(s, max_length)</code>","text":"<p>Truncate the string <code>s</code> to <code>max_length</code> by removing characters from the middle.</p> <p>:param s: The original string to be truncated. :type s: str :param max_length: The maximum allowed length of the string after truncation. :type max_length: int :return: The truncated string. :rtype: str</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def truncate_middle(s, max_length):\n\"\"\"\n Truncate the string `s` to `max_length` by removing characters from the middle.\n\n :param s: The original string to be truncated.\n :type s: str\n :param max_length: The maximum allowed length of the string after truncation.\n :type max_length: int\n :return: The truncated string.\n :rtype: str\n \"\"\"\n\n if len(s) <= max_length:\n # String is already at or below the maximum length, return it as is\n return s\n\n # Calculate the number of characters to keep from the start and end of the string\n num_keep_front = (max_length // 2)\n num_keep_end = max_length - num_keep_front - 1 # -1 for the ellipsis\n\n # Construct the truncated string\n return s[:num_keep_front] + '~' + s[-num_keep_end:]\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.read_conductor_credentials","title":"<code>read_conductor_credentials(use_api_key=False)</code>","text":"<p>Read the conductor credentials file.</p> <p>If the credentials file exists, it will contain a bearer token from either the user or the API key.</p> <p>If the credentials file doesn't exist, or is expired, or is from a different domain, we try to fetch a new one in the API key scenario or prompt the user to log in. </p> <p>Args: use_api_key (bool): Whether or not to try to use the API key</p> <p>Returns: A Bearer token in the event of a success or None</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def read_conductor_credentials(use_api_key=False):\n\"\"\"\n Read the conductor credentials file.\n\n If the credentials file exists, it will contain a bearer token from either\n the user or the API key.\n\n If the credentials file doesn't exist, or is\n expired, or is from a different domain, we try to fetch a new one in the API key scenario or\n prompt the user to log in. \n\n Args: \n use_api_key (bool): Whether or not to try to use the API key\n\n Returns: \n A Bearer token in the event of a success or None\n\n \"\"\"\n\n cfg = config.get()\n\n logger.debug(\"Reading conductor credentials...\")\n if use_api_key:\n if not cfg.get(\"api_key\"):\n use_api_key = False\n if use_api_key and not cfg[\"api_key\"].get(\"client_id\"):\n use_api_key = False\n logger.debug(\"use_api_key = %s\" % use_api_key)\n creds_file = get_creds_path(use_api_key)\n\n logger.debug(\"Creds file is %s\" % creds_file)\n logger.debug(\"Auth url is %s\" % cfg[\"url\"])\n if not os.path.exists(creds_file):\n if use_api_key:\n if not cfg[\"api_key\"]:\n logger.debug(\"Attempted to use API key, but no api key in in config!\")\n return None\n\n # Exchange the API key for a bearer token\n logger.debug(\"Attempting to get API key bearer token\")\n get_api_key_bearer_token(creds_file)\n\n else:\n auth.run(creds_file, cfg[\"url\"])\n if not os.path.exists(creds_file):\n return None\n\n logger.debug(\"Reading credentials file...\")\n with open(creds_file, \"r\", encoding=\"utf-8\") as fp:\n file_contents = json.loads(fp.read())\n expiration = file_contents.get(\"expiration\")\n same_domain = creds_same_domain(file_contents)\n if same_domain and expiration and expiration >= int(time.time()):\n return file_contents[\"access_token\"]\n logger.debug(\"Credentials have expired or are from a different domain\")\n if use_api_key:\n logger.debug(\"Refreshing API key bearer token!\")\n get_api_key_bearer_token(creds_file)\n else:\n logger.debug(\"Sending to auth page...\")\n auth.run(creds_file, cfg[\"url\"])\n # Re-read the creds file, since it has been re-upped\n with open(creds_file, \"r\", encoding=\"utf-8\") as fp:\n file_contents = json.loads(fp.read())\n return file_contents[\"access_token\"]\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_api_key_bearer_token","title":"<code>get_api_key_bearer_token(creds_file=None)</code>","text":"<p>Get a bearer token from the API key.</p> <p>Parameters:</p> Name Type Description Default <code>creds_file</code> <code>str</code> <p>The path to the credentials file. If not provided, the bearer token will not be written to disk.</p> <code>None</code> <p>Returns:</p> Type Description <p>A dictionary containing the bearer token and other information.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_api_key_bearer_token(creds_file=None):\n\"\"\"\n Get a bearer token from the API key.\n\n Args:\n creds_file (str): The path to the credentials file. If not provided, the bearer token will not be written to disk.\n\n Returns:\n A dictionary containing the bearer token and other information.\n \"\"\"\n cfg = config.get()\n url = \"{}/api/oauth_jwt\".format(cfg[\"url\"])\n response = requests.get(\n url,\n params={\n \"grant_type\": \"client_credentials\",\n \"scope\": \"owner admin user\",\n \"client_id\": cfg[\"api_key\"][\"client_id\"],\n \"client_secret\": cfg[\"api_key\"][\"private_key\"],\n },\n )\n if response.status_code == 200:\n response_dict = json.loads(response.text)\n credentials_dict = {\n \"access_token\": response_dict[\"access_token\"],\n \"token_type\": \"Bearer\",\n \"expiration\": int(time.time()) + int(response_dict[\"expires_in\"]),\n \"scope\": \"user admin owner\",\n }\n\n if not creds_file:\n return credentials_dict\n\n if not os.path.exists(os.path.dirname(creds_file)):\n os.makedirs(os.path.dirname(creds_file))\n\n with open(creds_file, \"w\") as fp:\n fp.write(json.dumps(credentials_dict))\n return\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_creds_path","title":"<code>get_creds_path(api_key=False)</code>","text":"<p>Get the path to the credentials file.</p> <p>Parameters:</p> Name Type Description Default <code>api_key</code> <code>bool</code> <p>Whether or not to use the API key.</p> <code>False</code> <p>Returns:</p> Type Description <code>str</code> <p>The path to the credentials file.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_creds_path(api_key=False):\n\"\"\"\n Get the path to the credentials file.\n\n Args:\n api_key (bool): Whether or not to use the API key.\n\n Returns:\n str: The path to the credentials file.\n \"\"\"\n creds_dir = os.path.join(os.path.expanduser(\"~\"), \".config\", \"conductor\")\n if api_key:\n creds_file = os.path.join(creds_dir, \"api_key_credentials\")\n else:\n creds_file = os.path.join(creds_dir, \"credentials\")\n return creds_file\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_bearer_token","title":"<code>get_bearer_token(refresh=False)</code>","text":"<p>Return the bearer token.</p> <p>Parameters:</p> Name Type Description Default <code>refresh</code> <code>bool</code> <p>Whether or not to refresh the token.</p> <code>False</code> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_bearer_token(refresh=False):\n\"\"\"\n Return the bearer token.\n\n Args:\n refresh (bool): Whether or not to refresh the token.\n\n TODO: Thread safe multiproc caching, like it used to be pre-python3.7.\n \"\"\"\n return read_conductor_credentials(True)\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.creds_same_domain","title":"<code>creds_same_domain(creds)</code>","text":"<p>Check if the creds are for the same domain as the config.</p> <p>Parameters:</p> Name Type Description Default <code>creds</code> <code>dict</code> <p>The credentials dictionary.</p> required <p>Returns:</p> Type Description <code>bool</code> <p>Whether or not the creds are for the same domain as the config.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def creds_same_domain(creds):\n\"\"\"\n Check if the creds are for the same domain as the config.\n\n Args:\n creds (dict): The credentials dictionary.\n\n Returns:\n bool: Whether or not the creds are for the same domain as the config.\n \"\"\"\n cfg = config.get()\n\"\"\"Ensure the creds file refers to the domain in config\"\"\"\n token = creds.get(\"access_token\")\n if not token:\n return False\n\n decoded = jwt.decode(creds[\"access_token\"], verify=False)\n audience_domain = decoded.get(\"aud\")\n return (\n audience_domain\n and audience_domain.rpartition(\"/\")[-1] == cfg[\"api_url\"].rpartition(\"/\")[-1]\n )\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.account_id_from_jwt","title":"<code>account_id_from_jwt(token)</code>","text":"<p>Fetch the accounts id from a jwt token value.</p> <p>Parameters:</p> Name Type Description Default <code>token</code> <code>str</code> <p>The jwt token.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The account id.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def account_id_from_jwt(token):\n\"\"\"\n Fetch the accounts id from a jwt token value.\n\n Args:\n token (str): The jwt token.\n\n Returns:\n str: The account id.\n \"\"\"\n payload = jwt.decode(token, verify=False)\n return payload.get(\"account\")\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.account_name_from_jwt","title":"<code>account_name_from_jwt(token)</code>","text":"<p>Fetch the accounts name from a jwt token value.</p> <p>Parameters:</p> Name Type Description Default <code>token</code> <code>str</code> <p>The jwt token.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The account name.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def account_name_from_jwt(token):\n\"\"\"\n Fetch the accounts name from a jwt token value.\n\n Args:\n token (str): The jwt token.\n\n Returns:\n str: The account name.\n \"\"\"\n account_id = account_id_from_jwt(token)\n cfg = config.get()\n if account_id:\n url = \"%s/api/v1/accounts/%s\" % (cfg[\"api_url\"], account_id)\n response = requests.get(url, headers={\"authorization\": \"Bearer %s\" % token})\n if response.status_code == 200:\n response_dict = json.loads(response.text)\n return response_dict[\"data\"][\"name\"]\n return None\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.request_instance_types","title":"<code>request_instance_types(as_dict=False)</code>","text":"<p>Get the list of available instances types.</p> <p>Parameters:</p> Name Type Description Default <code>as_dict</code> <code>bool</code> <p>Whether or not to return the instance types as a dictionary.</p> <code>False</code> <p>Returns:</p> Type Description <code>list</code> <p>The list of instance types.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def request_instance_types(as_dict=False):\n\"\"\"\n Get the list of available instances types.\n\n Args:\n as_dict (bool): Whether or not to return the instance types as a dictionary.\n\n Returns:\n list: The list of instance types.\n \"\"\"\n api = ApiClient()\n response, response_code = api.make_request(\n \"api/v1/instance-types\", use_api_key=True, raise_on_error=False\n )\n if response_code not in (200,):\n msg = \"Failed to get instance types\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n instance_types = json.loads(response).get(\"data\", [])\n logger.debug(\"Found available instance types: %s\", instance_types)\n\n if as_dict:\n return dict(\n [(instance[\"description\"], instance) for instance in instance_types]\n )\n return instance_types\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.request_projects","title":"<code>request_projects(statuses=('active',))</code>","text":"<p>Query Conductor for all client Projects that are in the given status(es).</p> <p>Parameters:</p> Name Type Description Default <code>statuses</code> <code>tuple</code> <p>The statuses to filter for.</p> <code>('active',)</code> <p>Returns:</p> Type Description <code>list</code> <p>The list of project names.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def request_projects(statuses=(\"active\",)):\n\"\"\"\n Query Conductor for all client Projects that are in the given status(es).\n\n Args:\n statuses (tuple): The statuses to filter for.\n\n Returns:\n list: The list of project names.\n \"\"\"\n api = ApiClient()\n\n logger.debug(\"statuses: %s\", statuses)\n\n uri = \"api/v1/projects/\"\n\n response, response_code = api.make_request(\n uri_path=uri, verb=\"GET\", raise_on_error=False, use_api_key=True\n )\n logger.debug(\"response: %s\", response)\n logger.debug(\"response: %s\", response_code)\n if response_code not in [200]:\n msg = \"Failed to get available projects from Conductor\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n projects = []\n\n # Filter for only projects of the proper status\n for project in json.loads(response).get(\"data\") or []:\n if not statuses or project.get(\"status\") in statuses:\n projects.append(project[\"name\"])\n return projects\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.request_software_packages","title":"<code>request_software_packages()</code>","text":"<p>Query Conductor for all software packages for the currently available sidecar.</p> <p>Returns:</p> Type Description <code>list</code> <p>The list of software packages.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def request_software_packages():\n\"\"\"\n Query Conductor for all software packages for the currently available sidecar.\n\n Returns:\n list: The list of software packages.\n \"\"\"\n api = ApiClient()\n\n uri = \"api/v1/ee/packages?all=true,\"\n response, response_code = api.make_request(\n uri_path=uri, verb=\"GET\", raise_on_error=False, use_api_key=True\n )\n\n if response_code not in [200]:\n msg = \"Failed to get software packages for latest sidecar\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n software = json.loads(response).get(\"data\", [])\n software = [sw for sw in software if not (\"3dsmax\" in sw[\"product\"] and sw[\"platform\"] == \"linux\")]\n return software\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.request_extra_environment","title":"<code>request_extra_environment()</code>","text":"<p>Query Conductor for extra environment.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def request_extra_environment():\n\"\"\"\n Query Conductor for extra environment.\n \"\"\"\n api = ApiClient()\n\n uri = \"api/v1/integrations/env-vars-configs\"\n response, response_code = api.make_request(\n uri_path=uri, verb=\"GET\", raise_on_error=False, use_api_key=True\n )\n\n if response_code not in [200]:\n msg = \"Failed to get extra environment\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n all_accounts = json.loads(response).get(\"data\", [])\n\n token = read_conductor_credentials(True)\n if not token:\n raise Exception(\"Error: Could not get conductor credentials!\")\n account_id = str(account_id_from_jwt(token))\n\n if not account_id:\n raise Exception(\"Error: Could not get account id from jwt!\")\n account_env = next((account for account in all_accounts if account[\"account_id\"] == account_id), None)\n if not account_env:\n raise Exception(\"Error: Could not get account environment!\")\n return account_env.get(\"env\", [])\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_jobs","title":"<code>get_jobs(first_jid, last_jid=None)</code>","text":"<p>Query Conductor for all jobs between the given job ids.</p> <p>Returns:</p> Type Description <code>list</code> <p>The list of jobs.</p> <p>Exceptions:</p> Type Description <code>Exception</code> <p>If the request fails.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> jobs = api_client.get_jobs(1959)\n>>> len(jobs)\n1\n>>> jobs[0][\"jid\"]\n'01959'\n>>> jobs = api_client.get_jobs(1959, 1961)\n>>> len(jobs)\n3\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_jobs(first_jid, last_jid=None):\n\"\"\"\n Query Conductor for all jobs between the given job ids.\n\n Returns:\n list: The list of jobs.\n\n Raises:\n Exception: If the request fails.\n\n Examples:\n >>> from ciocore import api_client\n >>> jobs = api_client.get_jobs(1959)\n >>> len(jobs)\n 1\n >>> jobs[0][\"jid\"]\n '01959'\n >>> jobs = api_client.get_jobs(1959, 1961)\n >>> len(jobs)\n 3\n \"\"\"\n if last_jid is None:\n last_jid = first_jid\n low = str(int(first_jid) - 1).zfill(5)\n high = str(int(last_jid) + 1).zfill(5)\n api = ApiClient()\n uri = \"api/v1/jobs\"\n\n response, response_code = api.make_request(\n uri_path=uri,\n verb=\"GET\",\n raise_on_error=False,\n use_api_key=True,\n params={\"filter\": f\"jid_gt_{low},jid_lt_{high}\"},\n )\n\n if response_code not in [200]:\n msg = f\"Failed to get jobs {first_jid}-{last_jid}\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n jobs = json.loads(response).get(\"data\")\n return jobs\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_log","title":"<code>get_log(job_id, task_id)</code>","text":"<p>Get the log for the given job and task.</p> <p>Parameters:</p> Name Type Description Default <code>job_id</code> <code>str</code> <p>The job id.</p> required <code>task_id</code> <code>str</code> <p>The task id.</p> required <p>Returns:</p> Type Description <code>list</code> <p>A list of logs.</p> <p>Exceptions:</p> Type Description <code>Exception</code> <p>If the request fails.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> logs = api_client.get_log(1959, 0)\n{\n \"logs\": [\n {\n \"container_id\": \"j-5669544198668288-5619559933149184-5095331660038144-stde\",\n \"instance_name\": \"renderer-5669544198668288-170062309438-62994\",\n \"log\": [\n \"Blender 2.93.0 (hash 84da05a8b806 built 2021-06-02 11:29:24)\",\n ...\n ...\n \"Saved: '/var/folders/8r/46lmjdmj50x_0swd9klwptzm0000gq/T/blender_bmw/renders/render_0001.png'\",\n \" Time: 00:29.22 (Saving: 00:00.32)\",\n \"\",\n \"\",\n \"Blender quit\"\n ],\n \"timestamp\": \"1.700623521101516E9\"\n }\n ],\n \"new_num_lines\": [\n 144\n ],\n \"status_description\": \"\",\n \"task_status\": \"success\"\n}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_log(job_id, task_id):\n\"\"\"\n Get the log for the given job and task.\n\n Args:\n job_id (str): The job id.\n task_id (str): The task id.\n\n Returns:\n list: A list of logs.\n\n Raises:\n Exception: If the request fails.\n\n Examples:\n >>> from ciocore import api_client\n >>> logs = api_client.get_log(1959, 0)\n {\n \"logs\": [\n {\n \"container_id\": \"j-5669544198668288-5619559933149184-5095331660038144-stde\",\n \"instance_name\": \"renderer-5669544198668288-170062309438-62994\",\n \"log\": [\n \"Blender 2.93.0 (hash 84da05a8b806 built 2021-06-02 11:29:24)\",\n ...\n ...\n \"Saved: '/var/folders/8r/46lmjdmj50x_0swd9klwptzm0000gq/T/blender_bmw/renders/render_0001.png'\",\n \" Time: 00:29.22 (Saving: 00:00.32)\",\n \"\",\n \"\",\n \"Blender quit\"\n ],\n \"timestamp\": \"1.700623521101516E9\"\n }\n ],\n \"new_num_lines\": [\n 144\n ],\n \"status_description\": \"\",\n \"task_status\": \"success\"\n }\n \"\"\"\n job_id = str(job_id).zfill(5)\n task_id = str(task_id).zfill(3)\n\n api = ApiClient()\n uri = f\"get_log_file?job={job_id}&task={task_id}&num_lines[]=0\"\n\n response, response_code = api.make_request(\n uri_path=uri, verb=\"GET\", raise_on_error=False, use_api_key=True\n )\n\n if response_code not in [200]:\n msg = f\"Failed to get log for job {job_id} task {task_id}\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n return response\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.kill_jobs","title":"<code>kill_jobs(job_ids)</code>","text":"<p>Kill the given jobs.</p> <p>Parameters:</p> Name Type Description Default <code>job_ids</code> <code>list</code> <p>The list of job ids.</p> <code>()</code> <p>Returns:</p> Type Description <code>dict</code> <p>The response.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> api_client.kill_jobs(\"03095\",\"03094\")\n{'body': 'success', 'message': \"Jobs [u'03095', u'03094'] have been kill.\"}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def kill_jobs(job_ids):\n\"\"\"\n Kill the given jobs.\n\n Args:\n job_ids (list): The list of job ids.\n\n Returns:\n dict: The response.\n\n Examples:\n >>> from ciocore import api_client\n >>> api_client.kill_jobs(\"03095\",\"03094\")\n {'body': 'success', 'message': \"Jobs [u'03095', u'03094'] have been kill.\"}\n\n \"\"\"\n job_ids = [str(job_id).zfill(5) for job_id in job_ids]\n api = ApiClient()\n payload = {\n \"action\": \"kill\",\n \"jobids\": job_ids,\n }\n response, response_code = api.make_request(\n uri_path=\"jobs_multi\", \n verb=\"PUT\", \n raise_on_error=False, \n use_api_key=True, \n data=json.dumps(payload)\n )\n\n if response_code not in [200]:\n msg = f\"Failed to kill jobs {job_ids}\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n return json.loads(response)\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.kill_tasks","title":"<code>kill_tasks(job_id, task_ids)</code>","text":"<p>Kill the given tasks.</p> <p>Parameters:</p> Name Type Description Default <code>job_id</code> <code>str</code> <p>The job id.</p> required <code>task_ids</code> <code>list</code> <p>The list of task ids.</p> <code>()</code> <p>Returns:</p> Type Description <code>dict</code> <p>The response.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> api_client.kill_tasks(\"03096\", range(50,56))\n{'body': 'success', 'message': ' 6 Tasks set to \"kill\"\n050\n051\n052\n053\n054\n055'}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def kill_tasks(job_id, task_ids):\n\"\"\"\n Kill the given tasks.\n\n Args:\n job_id (str): The job id.\n task_ids (list): The list of task ids.\n\n Returns:\n dict: The response.\n\n Examples:\n >>> from ciocore import api_client\n >>> api_client.kill_tasks(\"03096\", range(50,56))\n {'body': 'success', 'message': ' 6 Tasks set to \"kill\"\\n\\t050\\n\\t051\\n\\t052\\n\\t053\\n\\t054\\n\\t055'}\n \"\"\"\n\n job_id = str(job_id).zfill(5)\n task_ids = [str(task_id).zfill(3) for task_id in task_ids]\n api = ApiClient()\n payload = {\n \"action\": \"kill\",\n \"jobid\": job_id,\n \"taskids\": task_ids,\n }\n response, response_code = api.make_request(\n uri_path=\"tasks_multi\", \n verb=\"PUT\", \n raise_on_error=False, \n use_api_key=True, \n data=json.dumps(payload)\n )\n\n if response_code not in [200]:\n msg = f\"Failed to kill tasks {task_ids} of job {job_id}\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n return json.loads(response)\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_compute_usage","title":"<code>get_compute_usage(start_time, end_time)</code>","text":"<p>Query the compute usage for an account.</p> <p>Compute includes licenses, instances and Conductor cost. Everything involved with running a job. </p> <p>Parameters:</p> Name Type Description Default <code>start_time</code> <code>datetime.datetime</code> <p>The first day to include in the report. Only the date is considered and it's assumed to be in UTC.</p> required <code>end_time</code> <code>datetime.datetime</code> <p>The last day to include in the report. Only the date is considered and it's assumed to be in UTC.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>Each key is a date (UTC). The value is a dict with values for: - cost: The total accumulated compute cost for the day - corehours: The total accumulated core hours for the day - walltime: The number of minutes that instances (regardless of type) were running</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> api_client.get_compute_usage(start_time, end_time)\n{ '2024-01-09': { 'cost': 0.08,\n 'corehours': 0.9, \n 'walltime': 13.93},\n '2024-01-16': { 'cost': 0.12,\n 'corehours': 0.9613, \n 'walltime': 7.21}}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_compute_usage(start_time, end_time):\n'''\n Query the compute usage for an account.\n\n Compute includes licenses, instances and Conductor cost. Everything involved\n with running a job. \n\n Args:\n start_time (datetime.datetime): The first day to include in the report. Only the date is considered and it's assumed to be in UTC.\n end_time (datetime.datetime): The last day to include in the report. Only the date is considered and it's assumed to be in UTC.\n\n Returns:\n dict: Each key is a date (UTC). The value is a dict with values for:\n - cost: The total accumulated compute cost for the day\n - corehours: The total accumulated core hours for the day\n - walltime: The number of minutes that instances (regardless of type) were running\n\n Examples:\n >>> from ciocore import api_client\n >>> api_client.get_compute_usage(start_time, end_time)\n { '2024-01-09': { 'cost': 0.08,\n 'corehours': 0.9, \n 'walltime': 13.93},\n '2024-01-16': { 'cost': 0.12,\n 'corehours': 0.9613, \n 'walltime': 7.21}}\n '''\n date_format = \"%a, %d %b %Y %H:%M:%S %Z\"\n data = _get_compute_usage(start_time, end_time)\n\n # Create a nested default dictionary with initial float values of 0.0\n results = collections.defaultdict(lambda: collections.defaultdict(float))\n\n for entry in data:\n entry_start_date = datetime.datetime.strptime(entry['start_time'], date_format).date().isoformat()\n\n results[entry_start_date]['walltime'] += entry['minutes']\n results[entry_start_date]['corehours'] += entry['cores']\n results[entry_start_date]['cost'] += entry['license_cost'] + entry['instance_cost']\n\n # Round the data to avoid FP errors\n results[entry_start_date]['walltime'] = round(results[entry_start_date]['walltime'], 4)\n results[entry_start_date]['corehours'] = round(results[entry_start_date]['corehours'], 4)\n results[entry_start_date]['cost'] = round(results[entry_start_date]['cost'], 4)\n\n return results\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_storage_usage","title":"<code>get_storage_usage(start_time, end_time)</code>","text":"<p>Query the storage usage for an account.</p> <p>Storage is calculated twice a day (UTC) and the average is used.</p> <p>Parameters:</p> Name Type Description Default <code>start_time</code> <code>datetime.datetime</code> <p>The first day to include in the report. Only the date is considered and it's assumed to be in UTC.</p> required <code>end_time</code> <code>datetime.datetime</code> <p>The last day to include in the report. Only the date is considered and it's assumed to be in UTC.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>Each key is a date (UTC). The value is a dict with values for: - cost: The cost of accumulated storage for that one day - GiB: The total amount of storage used on that day</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> api_client.get_storage_usage(start_time, end_time)\n{ '2024-01-01': {'cost': 4.022, 'GiB': 679.714},\n '2024-01-02': {'cost': 4.502, 'GiB': 750.34},\n '2024-01-03': {'cost': 4.502, 'GiB': 750.34}}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_storage_usage(start_time, end_time):\n'''\n Query the storage usage for an account.\n\n Storage is calculated twice a day (UTC) and the average is used.\n\n Args:\n start_time (datetime.datetime): The first day to include in the report. Only the date is considered and it's assumed to be in UTC.\n end_time (datetime.datetime): The last day to include in the report. Only the date is considered and it's assumed to be in UTC.\n\n Returns:\n dict: Each key is a date (UTC). The value is a dict with values for:\n - cost: The cost of accumulated storage for that one day\n - GiB: The total amount of storage used on that day\n\n Examples:\n >>> from ciocore import api_client\n >>> api_client.get_storage_usage(start_time, end_time)\n { '2024-01-01': {'cost': 4.022, 'GiB': 679.714},\n '2024-01-02': {'cost': 4.502, 'GiB': 750.34},\n '2024-01-03': {'cost': 4.502, 'GiB': 750.34}}\n '''\n one_day = datetime.timedelta(days=1)\n\n data = _get_storage_usage(start_time, end_time)\n\n results = {}\n\n entry_date = datetime.date.fromisoformat(data['start_date'])\n\n for cnt, entry in enumerate(data[\"cost_per_day\"]):\n\n entry_start_date = entry_date.isoformat()\n results[entry_start_date] = {'cost': float(entry), 'GiB': float(data['gibs_per_day'][cnt])}\n entry_date += one_day\n\n return results\n</code></pre>"},{"location":"apidoc/apidoc/","title":"Api Overview","text":"<p>Conductor's Python API is intended to be used by developers to build custom submission tools, and to integrate Conductor into existing workflows. In order to use the API, you will need to have a Conductor account, and we recommend that you use an API key to authenticate.</p> <p>The goal of all Conductor submitters is to create a payload to be posted in an HTTP request. The payload is a JSON object that describes the submission, and includes a list of files to be uploaded to Conductor's storage. Let's take a look at a Conductor submission payload.</p>"},{"location":"apidoc/apidoc/#anatomy-of-a-conductor-submission","title":"Anatomy of a Conductor Submission","text":"<p>A Conductor submission consists of a JSON payload that describes the submission, and a collection of files that are uploaded to Conductor's storage.</p> <pre><code>{\n\"job_title\": \"Blender example bmw_half_turn_low.blend\",\n\"project\": \"default\",\n\"instance_type\": \"n1-standard-4\",\n\"software_package_ids\": [\n\"7be1b2410b3f93b2a2889f6ce191d4e1\"\n],\n\"force\": false,\n\"local_upload\": true,\n\"preemptible\": true,\n\"autoretry_policy\": {\n\"preempted\": {\n\"max_retries\": 3\n}\n},\n\"output_path\": \"projects/blender/renders\",\n\"environment\": {\n\"PATH\": \"/opt/blenderfoundation/blender/2/blender-2.93.0-glibc217\",\n\"CONDUCTOR_PATHHELPER\": \"0\"\n},\n\"upload_paths\": [\n\"/projects/blender/bmw_half_turn_low.blend\",\n\"/projects/blender/sourceimages/img_0001.png\",\n],\n\"scout_frames\": \"1,3\",\n\"tasks_data\": [\n{\n\"command\": \"blender -b \\\"/projects/blender/bmw_half_turn_low.blend\\\" -E CYCLES --render-output \\\"/projects/blender/renders/img_\\\" --render-frame 1..1 \",\n\"frames\": \"1\"\n},\n{\n\"command\": \"blender -b \\\"/projects/blender/bmw_half_turn_low.blend\\\" -E CYCLES --render-output \\\"/projects/blender/renders/img_\\\" --render-frame 2..2 \",\n\"frames\": \"2\"\n},\n{\n\"command\": \"blender -b \\\"/projects/blender/bmw_half_turn_low.blend\\\" -E CYCLES --render-output \\\"/projects/blender/renders/img_\\\" --render-frame 3..3 \",\n\"frames\": \"3\"\n},\n]\n}\n</code></pre>"},{"location":"apidoc/apidoc/#authentication","title":"Authentication","text":"<p>WIP</p>"},{"location":"apidoc/config/","title":"ciocore.config","text":"<p>Config is a configuration object implemented as a module-level singleton.</p> <p>Configuration variables can be shared by importing the module. If there are changes in environment variables or other sources, the config can be refreshed.</p>"},{"location":"apidoc/config/#ciocore.config.Config","title":"<code> Config </code>","text":"Source code in <code>ciocore/config.py</code> <pre><code>class Config(object):\n def __init__(self):\n\"\"\"\n Initialize the config object.\n\n A config object is a dictionary containing configuration values. It is a singleton, so there is only one instance of it. It is instantiated the first time it is needed. It can be refreshed by calling get() with the `force` keyword argument set to `True`.\n\n A Config object has the following properties:\n\n `thread_count` The number of threads to use for downloading files. Defaults to the number of CPUs on the system times 2. It can be overridden by the `CONDUCTOR_THREAD_COUNT` environment variable.\n * `priority` Set the priority for submissions. Defaults to 5. It can be overridden by the `CONDUCTOR_PRIORITY` environment variable.\n * `md5_caching` Whether to cache MD5s. Defaults to `True`. It can be overridden by the `CONDUCTOR_MD5_CACHING` environment variable. Cachine MD5s significantly improves submission performance, but on rare occasions it can cause submissions to fail. If you experience this, set `md5_caching` to `False`.\n * `log_level` The logging level. Defaults to `INFO`. It can be overridden by the `CONDUCTOR_LOG_LEVEL` environment variable.\n * `url` The URL of the Conductor dashboard. Defaults to `https://dashboard.conductortech.com`. It can be overridden by the `CONDUCTOR_URL` environment variable.\n * `auth_url` The URL of the Conductor dashboard. Defaults to `https://dashboard.conductortech.com`. It can be overridden by the `CONDUCTOR_AUTH_URL` environment variable. This is deprecated. Use `url` instead.\n * `api_url` The URL of the Conductor API. Defaults to `https://api.conductortech.com`. It can be overridden by the `CONDUCTOR_API_URL` environment variable.\n * `api_key` The API key. The API key can be acquired from the Conductor dashboard, and can be stored in an environment variable or a file. In both cases the API KEY can be a JSON object or a base64 encoded JSON object. If it is base64 encoded, it can be a string or bytes. If it is a string, it will be decoded as ASCII. If it is bytes, it will be decoded as UTF-8.\n * Environment variable: The `CONDUCTOR_API_KEY` variable can hold the API KEY directly.\n * File: The `CONDUCTOR_API_KEY_PATH` variable can hold the path to a file containing the API KEY.\n * `downloader_page_size` The number of files to request from the Conductor API at a time. Defaults to 50. It can be overridden by the `CONDUCTOR_DOWNLOADER_PAGE_SIZE` environment variable.\n\n Returns:\n Config: A config object.\n\n Raises:\n ValueError -- Invalid inputs, such as badly formed URLs.\n \"\"\"\n default_downloader_page_size = 50\n\n\n\n try:\n default_thread_count = min( os.cpu_count() - 1, 15)\n except NotImplementedError:\n default_thread_count = 15\n\n url = os.environ.get(\"CONDUCTOR_URL\", \"https://dashboard.conductortech.com\")\n\n if not URL_REGEX.match(url):\n raise ValueError(\"CONDUCTOR_URL is not valid '{}'\".format(url))\n\n api_url = os.environ.get(\"CONDUCTOR_API_URL\", url.replace(\"dashboard\", \"api\"))\n if not URL_REGEX.match(api_url):\n raise ValueError(\"CONDUCTOR_API_URL is not valid '{}'\".format(api_url))\n\n falsy = [\"false\", \"no\", \"off\", \"0\"]\n\n log_level = os.environ.get(\"CONDUCTOR_LOG_LEVEL\", \"INFO\")\n if log_level not in [\"CRITICAL\", \"ERROR\", \"WARNING\", \"INFO\", \"DEBUG\"]:\n log_level = \"INFO\"\n\n self.config = {\n \"thread_count\": int(\n os.environ.get(\"CONDUCTOR_THREAD_COUNT\", default_thread_count)\n ),\n \"downloader_page_size\": int(\n os.environ.get(\n \"CONDUCTOR_DOWNLOADER_PAGE_SIZE\", default_downloader_page_size\n )\n ),\n \"priority\": int(os.environ.get(\"CONDUCTOR_PRIORITY\", 5)),\n \"md5_caching\": False\n if os.environ.get(\"CONDUCTOR_MD5_CACHING\", \"True\").lower() in falsy\n else True,\n \"log_level\": log_level,\n \"url\": url,\n # Keep \"auth_url\" for backwwards compatibillity only.\n # Clients should use \"url\" moving forward.\n # Remove \"auth_url\" on the next major version bump.\n \"auth_url\": url,\n \"api_url\": api_url,\n \"api_key\": self.get_api_key_from_variable() or self.get_api_key_from_file(),\n \"user_dir\": os.environ.get('CONDUCTOR_USER_DIR', DEFAULT_USER_DIR)\n }\n\n @staticmethod\n def get_api_key_from_variable():\n\"\"\"\n Attempt to get an API key from the `CONDUCTOR_API_KEY` environment variable.\n\n Raises:\n ValueError: An error occurred while reading or loading the key into JSON.\n\n Returns:\n str: JSON object containing the key - base 64 decoded if necessary.\n\n \"\"\"\n api_key = os.environ.get(\"CONDUCTOR_API_KEY\")\n if not api_key:\n return\n logger.info(\"Attempting to read API key from CONDUCTOR_API_KEY\")\n try:\n return json.loads(api_key.replace(\"\\n\", \"\").replace(\"\\r\", \"\"))\n except ValueError:\n try:\n result = base64.b64decode(api_key)\n return Config._to_json(result)\n except BaseException:\n result = base64.b64decode(api_key.encode()).decode(\"ascii\")\n return Config._to_json(result)\n except BaseException:\n message = \"An error occurred reading the API key from the CONDUCTOR_API_KEY variable\"\n logger.error(message)\n raise ValueError(message)\n\n @staticmethod\n def get_api_key_from_file():\n\"\"\"\n Attempt to get an API key from the file in the CONDUCTOR_API_KEY_PATH environment variable.\n\n Raises:\n ValueError: An error occurred while reading or loading the key into JSON.\n\n Returns:\n str: JSON object containing the key - base 64 decoded if necessary.\n\n \"\"\"\n api_key_path = os.environ.get(\"CONDUCTOR_API_KEY_PATH\")\n if not api_key_path:\n return\n logger.info(\"Attempting to read API key from CONDUCTOR_API_KEY_PATH\")\n try:\n with open(api_key_path, \"r\") as fp:\n return Config._to_json(fp.read())\n except BaseException:\n message = \"An error occurred reading the API key from the path described in the CONDUCTOR_API_KEY_PATH variable\"\n logger.error(message)\n raise ValueError(message)\n\n @staticmethod\n def _to_json(content):\n return json.loads(content.replace(\"\\n\", \"\").replace(\"\\r\", \"\"))\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.Config.__init__","title":"<code>__init__(self)</code> <code>special</code>","text":"<p>Initialize the config object.</p> <p>A config object is a dictionary containing configuration values. It is a singleton, so there is only one instance of it. It is instantiated the first time it is needed. It can be refreshed by calling get() with the <code>force</code> keyword argument set to <code>True</code>.</p> <p>A Config object has the following properties:</p> <ul> <li><code>thread_count</code> The number of threads to use for downloading files. Defaults to the number of CPUs on the system times 2. It can be overridden by the <code>CONDUCTOR_THREAD_COUNT</code> environment variable.</li> <li><code>priority</code> Set the priority for submissions. Defaults to 5. It can be overridden by the <code>CONDUCTOR_PRIORITY</code> environment variable.</li> <li><code>md5_caching</code> Whether to cache MD5s. Defaults to <code>True</code>. It can be overridden by the <code>CONDUCTOR_MD5_CACHING</code> environment variable. Cachine MD5s significantly improves submission performance, but on rare occasions it can cause submissions to fail. If you experience this, set <code>md5_caching</code> to <code>False</code>.</li> <li><code>log_level</code> The logging level. Defaults to <code>INFO</code>. It can be overridden by the <code>CONDUCTOR_LOG_LEVEL</code> environment variable.</li> <li><code>url</code> The URL of the Conductor dashboard. Defaults to <code>https://dashboard.conductortech.com</code>. It can be overridden by the <code>CONDUCTOR_URL</code> environment variable.</li> <li><code>auth_url</code> The URL of the Conductor dashboard. Defaults to <code>https://dashboard.conductortech.com</code>. It can be overridden by the <code>CONDUCTOR_AUTH_URL</code> environment variable. This is deprecated. Use <code>url</code> instead.</li> <li><code>api_url</code> The URL of the Conductor API. Defaults to <code>https://api.conductortech.com</code>. It can be overridden by the <code>CONDUCTOR_API_URL</code> environment variable.</li> <li><code>api_key</code> The API key. The API key can be acquired from the Conductor dashboard, and can be stored in an environment variable or a file. In both cases the API KEY can be a JSON object or a base64 encoded JSON object. If it is base64 encoded, it can be a string or bytes. If it is a string, it will be decoded as ASCII. If it is bytes, it will be decoded as UTF-8.<ul> <li>Environment variable: The <code>CONDUCTOR_API_KEY</code> variable can hold the API KEY directly.</li> <li>File: The <code>CONDUCTOR_API_KEY_PATH</code> variable can hold the path to a file containing the API KEY.</li> </ul> </li> <li><code>downloader_page_size</code> The number of files to request from the Conductor API at a time. Defaults to 50. It can be overridden by the <code>CONDUCTOR_DOWNLOADER_PAGE_SIZE</code> environment variable.</li> </ul> <p>Returns:</p> Type Description <code>Config</code> <p>A config object.</p> Source code in <code>ciocore/config.py</code> <pre><code>def __init__(self):\n\"\"\"\n Initialize the config object.\n\n A config object is a dictionary containing configuration values. It is a singleton, so there is only one instance of it. It is instantiated the first time it is needed. It can be refreshed by calling get() with the `force` keyword argument set to `True`.\n\n A Config object has the following properties:\n\n * `thread_count` The number of threads to use for downloading files. Defaults to the number of CPUs on the system times 2. It can be overridden by the `CONDUCTOR_THREAD_COUNT` environment variable.\n * `priority` Set the priority for submissions. Defaults to 5. It can be overridden by the `CONDUCTOR_PRIORITY` environment variable.\n * `md5_caching` Whether to cache MD5s. Defaults to `True`. It can be overridden by the `CONDUCTOR_MD5_CACHING` environment variable. Cachine MD5s significantly improves submission performance, but on rare occasions it can cause submissions to fail. If you experience this, set `md5_caching` to `False`.\n * `log_level` The logging level. Defaults to `INFO`. It can be overridden by the `CONDUCTOR_LOG_LEVEL` environment variable.\n * `url` The URL of the Conductor dashboard. Defaults to `https://dashboard.conductortech.com`. It can be overridden by the `CONDUCTOR_URL` environment variable.\n * `auth_url` The URL of the Conductor dashboard. Defaults to `https://dashboard.conductortech.com`. It can be overridden by the `CONDUCTOR_AUTH_URL` environment variable. This is deprecated. Use `url` instead.\n * `api_url` The URL of the Conductor API. Defaults to `https://api.conductortech.com`. It can be overridden by the `CONDUCTOR_API_URL` environment variable.\n * `api_key` The API key. The API key can be acquired from the Conductor dashboard, and can be stored in an environment variable or a file. In both cases the API KEY can be a JSON object or a base64 encoded JSON object. If it is base64 encoded, it can be a string or bytes. If it is a string, it will be decoded as ASCII. If it is bytes, it will be decoded as UTF-8.\n * Environment variable: The `CONDUCTOR_API_KEY` variable can hold the API KEY directly.\n * File: The `CONDUCTOR_API_KEY_PATH` variable can hold the path to a file containing the API KEY.\n * `downloader_page_size` The number of files to request from the Conductor API at a time. Defaults to 50. It can be overridden by the `CONDUCTOR_DOWNLOADER_PAGE_SIZE` environment variable.\n\n Returns:\n Config: A config object.\n\n Raises:\n ValueError -- Invalid inputs, such as badly formed URLs.\n \"\"\"\n default_downloader_page_size = 50\n\n\n\n try:\n default_thread_count = min( os.cpu_count() - 1, 15)\n except NotImplementedError:\n default_thread_count = 15\n\n url = os.environ.get(\"CONDUCTOR_URL\", \"https://dashboard.conductortech.com\")\n\n if not URL_REGEX.match(url):\n raise ValueError(\"CONDUCTOR_URL is not valid '{}'\".format(url))\n\n api_url = os.environ.get(\"CONDUCTOR_API_URL\", url.replace(\"dashboard\", \"api\"))\n if not URL_REGEX.match(api_url):\n raise ValueError(\"CONDUCTOR_API_URL is not valid '{}'\".format(api_url))\n\n falsy = [\"false\", \"no\", \"off\", \"0\"]\n\n log_level = os.environ.get(\"CONDUCTOR_LOG_LEVEL\", \"INFO\")\n if log_level not in [\"CRITICAL\", \"ERROR\", \"WARNING\", \"INFO\", \"DEBUG\"]:\n log_level = \"INFO\"\n\n self.config = {\n \"thread_count\": int(\n os.environ.get(\"CONDUCTOR_THREAD_COUNT\", default_thread_count)\n ),\n \"downloader_page_size\": int(\n os.environ.get(\n \"CONDUCTOR_DOWNLOADER_PAGE_SIZE\", default_downloader_page_size\n )\n ),\n \"priority\": int(os.environ.get(\"CONDUCTOR_PRIORITY\", 5)),\n \"md5_caching\": False\n if os.environ.get(\"CONDUCTOR_MD5_CACHING\", \"True\").lower() in falsy\n else True,\n \"log_level\": log_level,\n \"url\": url,\n # Keep \"auth_url\" for backwwards compatibillity only.\n # Clients should use \"url\" moving forward.\n # Remove \"auth_url\" on the next major version bump.\n \"auth_url\": url,\n \"api_url\": api_url,\n \"api_key\": self.get_api_key_from_variable() or self.get_api_key_from_file(),\n \"user_dir\": os.environ.get('CONDUCTOR_USER_DIR', DEFAULT_USER_DIR)\n }\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.Config.get_api_key_from_variable","title":"<code>get_api_key_from_variable()</code> <code>staticmethod</code>","text":"<p>Attempt to get an API key from the <code>CONDUCTOR_API_KEY</code> environment variable.</p> <p>Exceptions:</p> Type Description <code>ValueError</code> <p>An error occurred while reading or loading the key into JSON.</p> <p>Returns:</p> Type Description <code>str</code> <p>JSON object containing the key - base 64 decoded if necessary.</p> Source code in <code>ciocore/config.py</code> <pre><code>@staticmethod\ndef get_api_key_from_variable():\n\"\"\"\n Attempt to get an API key from the `CONDUCTOR_API_KEY` environment variable.\n\n Raises:\n ValueError: An error occurred while reading or loading the key into JSON.\n\n Returns:\n str: JSON object containing the key - base 64 decoded if necessary.\n\n \"\"\"\n api_key = os.environ.get(\"CONDUCTOR_API_KEY\")\n if not api_key:\n return\n logger.info(\"Attempting to read API key from CONDUCTOR_API_KEY\")\n try:\n return json.loads(api_key.replace(\"\\n\", \"\").replace(\"\\r\", \"\"))\n except ValueError:\n try:\n result = base64.b64decode(api_key)\n return Config._to_json(result)\n except BaseException:\n result = base64.b64decode(api_key.encode()).decode(\"ascii\")\n return Config._to_json(result)\n except BaseException:\n message = \"An error occurred reading the API key from the CONDUCTOR_API_KEY variable\"\n logger.error(message)\n raise ValueError(message)\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.Config.get_api_key_from_file","title":"<code>get_api_key_from_file()</code> <code>staticmethod</code>","text":"<p>Attempt to get an API key from the file in the CONDUCTOR_API_KEY_PATH environment variable.</p> <p>Exceptions:</p> Type Description <code>ValueError</code> <p>An error occurred while reading or loading the key into JSON.</p> <p>Returns:</p> Type Description <code>str</code> <p>JSON object containing the key - base 64 decoded if necessary.</p> Source code in <code>ciocore/config.py</code> <pre><code>@staticmethod\ndef get_api_key_from_file():\n\"\"\"\n Attempt to get an API key from the file in the CONDUCTOR_API_KEY_PATH environment variable.\n\n Raises:\n ValueError: An error occurred while reading or loading the key into JSON.\n\n Returns:\n str: JSON object containing the key - base 64 decoded if necessary.\n\n \"\"\"\n api_key_path = os.environ.get(\"CONDUCTOR_API_KEY_PATH\")\n if not api_key_path:\n return\n logger.info(\"Attempting to read API key from CONDUCTOR_API_KEY_PATH\")\n try:\n with open(api_key_path, \"r\") as fp:\n return Config._to_json(fp.read())\n except BaseException:\n message = \"An error occurred reading the API key from the path described in the CONDUCTOR_API_KEY_PATH variable\"\n logger.error(message)\n raise ValueError(message)\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.config","title":"<code>config(force=False)</code>","text":"<p>Instantiate a config object if necessary.</p> <p>Deprecated</p> <p>Use get() instead.</p> <p>Parameters:</p> Name Type Description Default <code>force</code> <code>bool</code> <p>Discards any existing config object and instantiate a new one -- Defaults to <code>False</code>.</p> <code>False</code> <p>Returns:</p> Type Description <code>dict</code> <p>A dictionary containing configuration values.</p> Source code in <code>ciocore/config.py</code> <pre><code>def config(force=False):\n\"\"\"\n Instantiate a config object if necessary.\n\n Deprecated:\n Use [get()](#get) instead.\n\n Args:\n force (bool): Discards any existing config object and instantiate a new one -- Defaults to `False`.\n\n Returns:\n dict: A dictionary containing configuration values.\n \"\"\"\n\n global __config__\n if force or not __config__:\n __config__ = Config()\n return __config__\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.get","title":"<code>get(force=False)</code>","text":"<p>Instantiate a config object if necessary and return the dictionary.</p> <p>Parameters:</p> Name Type Description Default <code>force</code> <code>bool</code> <p>Discards any existing config object and instantiate a new one -- Defaults to <code>False</code>.</p> <code>False</code> <p>Returns:</p> Type Description <code>dict</code> <p>A dictionary containing configuration values.</p> <p>Examples:</p> <pre><code>>>> from ciocore import config\n>>> config.get()\n{\n 'thread_count': 16,\n 'priority': 5,\n 'md5_caching': True,\n 'log_level': 'INFO',\n 'url': 'https://dashboard.conductortech.com',\n 'auth_url': 'https://dashboard.conductortech.com',\n 'api_url': 'https://api.conductortech.com',\n 'api_key': None,\n 'downloader_page_size': 50,\n}\n</code></pre> Source code in <code>ciocore/config.py</code> <pre><code>def get(force=False):\n\"\"\"\n Instantiate a config object if necessary and return the dictionary.\n\n Args:\n force (bool): Discards any existing config object and instantiate a new one -- Defaults to `False`.\n\n Returns:\n dict: A dictionary containing configuration values.\n\n Example:\n >>> from ciocore import config\n >>> config.get()\n {\n 'thread_count': 16,\n 'priority': 5,\n 'md5_caching': True,\n 'log_level': 'INFO',\n 'url': 'https://dashboard.conductortech.com',\n 'auth_url': 'https://dashboard.conductortech.com',\n 'api_url': 'https://api.conductortech.com',\n 'api_key': None,\n 'downloader_page_size': 50,\n }\n \"\"\"\n global __config__\n if force or not __config__:\n __config__ = Config()\n return __config__.config\n</code></pre>"},{"location":"apidoc/data/","title":"ciocore.data","text":"<p>This module is a singleton that provides the data from Conductor endpoints. Specifically, it provides projects, instance types, and software package data.</p> <p>Since the data is stored at the module level, you can access it from anywhere in your code without the need to pass it around.</p>"},{"location":"apidoc/data/#ciocore.data.init","title":"<code>init(products, kwargs)</code>","text":"<p>Initialize the module and let it know what host products to provide.</p> <p>Parameters:</p> Name Type Description Default <code>products</code> <code>str</code> <p>Provide a list of products for which to get software packages. If no products are given, the software data contains all products from the packages endpoint. If you provide more than one product, they must all be host level products.</p> <code>()</code> <p>Keyword arguments:</p> Name Type Description <code>product</code> <code>str</code> <p><code>DEPRECATED</code> Provide one product for which to get software packages.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n# OR\n>>> coredata.init(\"maya-io\")\n# OR LEGACY\n>>> coredata.init(product=\"all\")\n# OR\n>>> coredata.init(product=\"maya-io\")\n</code></pre> Source code in <code>ciocore/data.py</code> <pre><code>def init(products, *kwargs):\n\"\"\"\n Initialize the module and let it know what host products to provide.\n\n Args:\n products (str): Provide a list of products for which to get software packages. If no products are given, the software data contains all products from the packages endpoint. If you provide more than one product, they must all be host level products.\n\n Keyword Args:\n product (str): `DEPRECATED` Provide one product for which to get software packages.\n\n Examples:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n # OR\n >>> coredata.init(\"maya-io\")\n # OR LEGACY\n >>> coredata.init(product=\"all\")\n # OR\n >>> coredata.init(product=\"maya-io\")\n \"\"\"\n global __products__\n global __platforms__\n if products:\n if kwargs.get(\"product\"):\n raise ValueError(\n \"Arguments: `products` and `product` specified. Please don't use both together. The `product` arg is deprecated.\"\n )\n __products__ = list(products)\n elif kwargs.get(\"product\"):\n if kwargs.get(\"product\") == \"all\":\n __products__ = []\n else:\n __products__ = [kwargs.get(\"product\")]\n else:\n __products__ = []\n\n __platforms__ = set(kwargs.get(\"platforms\", [\"windows\", \"linux\"]))\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.data","title":"<code>data(force=False)</code>","text":"<p>Provide projects, instance types, and software package data.</p> <p>Keyword arguments:</p> Name Type Description <code>force</code> <p>(bool) If <code>True</code>, then force the system to fetch fresh data -- Defaults to <code>False</code>.</p> <p>Exceptions:</p> Type Description <code>ValueError</code> <p>Module was not initialized with init().</p> <p>Returns:</p> Type Description <code>dict</code> <p>Keys are <code>projects</code>, <code>instance_types</code>, <code>software</code>.</p> <p>When you access the data, if it has already been fetched, it will be returned. Otherwise, requests will be made to fetch the data. You may need to authenticate in order to access the data.</p> <p>The set of instance types and software can be pruned to match the available platforms represented by each other. For example, if the instance types come from an orchestrator that provides both Windows and Linux machines, and the software product(s) are available on both platforms, no pruning occurs. However, if there are no Windows machines available, any Windows software will be removed from the package tree. Similarly, if a product is chosen that only runs on Windows, Linux instance types will not appearin the list of available hardware.</p> <p>Here is a breakdown of each key in the dictionary:</p> <ul> <li> <p>projects is a list of project names for your authenticated account.</p> </li> <li> <p>instance_types is an instance of HardwareSet, providing you with access to the list of available machines configurations.</p> </li> <li> <p>software is a PackageTree object containing either all the software available at Conductor, or a subset based on specified products.</p> </li> </ul> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init(product=\"maya-io\")\n</code></pre> <pre><code>>>> coredata.data()[\"software\"]\n<ciocore.package_tree.PackageTree object at 0x10e9a4040>\n</code></pre> <pre><code>>>> coredata.data()[\"projects\"][0]\nATestForScott\n</code></pre> <pre><code>>>> coredata.data()[\"instance_types\"]\n<ciocore.hardware_set.HardwareSet object at 0x0000028941CD9DC0>\n</code></pre> Source code in <code>ciocore/data.py</code> <pre><code>def data(force=False):\n\"\"\"\n Provide projects, instance types, and software package data.\n\n Keyword Args:\n force: (bool) If `True`, then force the system to fetch fresh data -- Defaults to `False`.\n\n Raises:\n ValueError: Module was not initialized with [init()](/data/#ciocore.data.init).\n\n Returns:\n dict: Keys are `projects`, `instance_types`, `software`.\n\n When you access the data, if it has already been fetched, it will be returned. Otherwise,\n requests will be made to fetch the data. You may need to authenticate in order to access the\n data.\n\n The set of instance types and software can be pruned to match the available platforms\n represented by each other. For example, if the instance types come from an orchestrator that\n provides both Windows and Linux machines, and the software product(s) are available on both\n platforms, no pruning occurs. However, if there are no Windows machines available, any Windows\n software will be removed from the package tree. Similarly, if a product is chosen that only runs\n on Windows, Linux instance types will not appearin the list of available hardware.\n\n Here is a breakdown of each key in the dictionary:\n\n projects is a list of project names for your authenticated account.\n\n * instance_types is an instance of HardwareSet, providing you with access to the list of\n available machines configurations.\n\n * software is a PackageTree object containing either all\n the software available at Conductor, or a subset based on specified products.\n\n\n Examples:\n >>> from ciocore import data as coredata\n >>> coredata.init(product=\"maya-io\")\n\n >>> coredata.data()[\"software\"]\n <ciocore.package_tree.PackageTree object at 0x10e9a4040>\n\n >>> coredata.data()[\"projects\"][0]\n ATestForScott\n\n >>> coredata.data()[\"instance_types\"]\n <ciocore.hardware_set.HardwareSet object at 0x0000028941CD9DC0>\n \"\"\"\n\n global __data__\n global __products__\n global __platforms__\n\n if __products__ is None:\n raise ValueError(\n 'Data must be initialized before use, e.g. data.init(\"maya-io\") or data.init().'\n )\n products_copy = copy.copy(__products__)\n\n if force:\n clear()\n init(products_copy)\n\n if __data__ == {}:\n # PROJECTS\n __data__[\"projects\"] = sorted(api_client.request_projects())\n # INST_TYPES\n instance_types = api_client.request_instance_types()\n # SOFTWARE\n software = api_client.request_software_packages()\n\n # EXTRA ENV VARS\n extra_env_vars = []\n try:\n extra_env_vars = api_client.request_extra_environment()\n except Exception:\n pass \n __data__[\"extra_environment\"] = extra_env_vars\n\n # PLATFORMS\n it_platforms = set([it[\"operating_system\"] for it in instance_types])\n valid_platforms = it_platforms.intersection(__platforms__)\n kwargs = {\"platforms\": valid_platforms}\n\n # If there's only one product, it's possible to initialize the software tree with a plugin.\n # So we set the product kwarg. Otherwise, we set the host_products kwarg\n host_products = __products__\n if len(__products__) == 1:\n host_products = []\n kwargs[\"product\"] = __products__[0]\n\n software_tree = PackageTree(software, host_products, kwargs)\n\n if software_tree:\n __data__[\"software\"] = software_tree\n # Revisit instance types to filter out any that are not needed for any software package.\n sw_platforms = software_tree.platforms()\n\n instance_types = [\n it for it in instance_types if it[\"operating_system\"] in sw_platforms\n ]\n # Then adjust __platforms__ to match the instance types that are represented.\n __platforms__ = set([it[\"operating_system\"] for it in instance_types])\n\n __data__[\"instance_types\"] = HardwareSet(instance_types)\n\n return __data__\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.valid","title":"<code>valid()</code>","text":"<p>Check validity.</p> <p>Returns:</p> Type Description <code>bool</code> <p>True if <code>projects</code>, <code>instance_types</code>, and <code>software</code> are valid.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.valid()\nTrue\n</code></pre> Source code in <code>ciocore/data.py</code> <pre><code>def valid():\n\"\"\"\n Check validity.\n\n Returns:\n bool: True if `projects`, `instance_types`, and `software` are valid.\n\n Examples:\n >>> from ciocore import data as coredata\n >>> coredata.valid()\n True\n \"\"\"\n\n if not __data__.get(\"projects\"):\n return False\n if not __data__.get(\"instance_types\"):\n return False\n if not __data__.get(\"software\"):\n return False\n return True\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.clear","title":"<code>clear()</code>","text":"<p>Clear out data.</p> <p>valid() returns False after clear().</p> Source code in <code>ciocore/data.py</code> <pre><code>def clear():\n\"\"\"\n Clear out data.\n\n [valid()](/data/#ciocore.data.valid) returns False after clear().\n \"\"\"\n global __data__\n global __products__\n global __platforms__\n __data__ = {}\n __products__ = None\n __platforms__ = None\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.products","title":"<code>products()</code>","text":"<p>Returns:</p> Type Description <code>list(str)</code> <p>The product names. An empty list signifies all products.</p> Source code in <code>ciocore/data.py</code> <pre><code>def products():\n\"\"\"\n\n Returns:\n list(str): The product names. An empty list signifies all products.\n \"\"\"\n return __products__\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.set_fixtures_dir","title":"<code>set_fixtures_dir(_)</code>","text":"<p>Deprecated. </p> Source code in <code>ciocore/data.py</code> <pre><code>def set_fixtures_dir(_):\n\"\"\"Deprecated. \"\"\"\n pass\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.platforms","title":"<code>platforms()</code>","text":"<p>The set of platforms that both software and instance types are valid on.</p> <p>Returns:</p> Type Description <code>set</code> <p>A set containing platforms: windows and/or linux.</p> Source code in <code>ciocore/data.py</code> <pre><code>def platforms():\n\"\"\"\n The set of platforms that both software and instance types are valid on.\n\n Returns:\n set: A set containing platforms: windows and/or linux.\n \"\"\"\n return __platforms__\n</code></pre>"},{"location":"apidoc/hardware_set/","title":"ciocore.hardware_set","text":"<p>This module contains the HardwareSet class. </p> <p>It is designed to allow submission tools and other clients that consume instance types to be able to display them in categories. This is particularly useful for UIs that utilize combo-boxes, since they can be orgnaized into a nested structure and displayed in a tree-like fashion.</p>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet","title":"<code> HardwareSet </code>","text":"<p>A class to manage categorized instance types.</p> <p>A HardwareSet encapsulates the instance types available to an account. It accepts a flat list of instance types and builds a nested structure where those instance types exist in categories.</p> <p>It keeps a dictionary of instance types (<code>instance_types</code>) with the name field as key. This allows easy lookup by name.</p> <p>In addition, it keeps the nested structure of categories (<code>categories</code>) that contain the instance types. Each category is a dictionary with keys: <code>label</code>, <code>content</code>, and <code>order</code>.</p> <p><code>content</code> is a list of instance types in the category. The order is used to sort the categories. The order of the instance types within a category is determined by the number of cores and memory.</p> <p>If all instance_types have not been assigned any categories, then the structure is built with two default categories: CPU and GPU.</p> Source code in <code>ciocore/hardware_set.py</code> <pre><code>class HardwareSet(object):\n\"\"\"A class to manage categorized instance types.\n\n A HardwareSet encapsulates the instance types available to an account. It accepts a flat list of instance types and builds a nested structure where those instance types exist in categories.\n\n It keeps a dictionary of instance types (`instance_types`) with the name field as key. This allows easy lookup by name.\n\n In addition, it keeps the nested structure of categories (`categories`) that contain the instance types. Each category is a dictionary with keys: `label`, `content`, and `order`.\n\n `content` is a list of instance types in the category. The order is used to sort the categories. The order of the instance types within a category is determined by the number of cores and memory.\n\n If all instance_types have not been assigned any categories, then the structure is built with two default categories: CPU and GPU.\n \"\"\"\n\n def __init__(self, instance_types):\n\"\"\"Initialize the HardwareSet with a list of instance types.\n Typically, you would access the HardwareSet through the ciocore.data.data() function, which initializes it for you. However, you can also initialize it directly with a list of instance types straight from ciocore.api_client. The difference being that the latter contains all instance types, whereas the former contains only the instance types compatible with the products you have specified, as well as being cached.\n\n Args:\n instance_types (list): A list of instance types.\n\n Returns:\n HardwareSet: The initialized HardwareSet.\n\n Examples:\n ### Initialize with a list of instance types\n >>> from ciocore import api_client\n >>> from ciocore.hardware_set import HardwareSet\n >>> instance_types = api_client.request_instance_types()\n >>> hardware_set = HardwareSet(instance_types)\n <ciocore.hardware_set.HardwareSet object at 0x104c43d30>\n\n ### Initialize implicitly with a list of instance types from ciocore.data (recommended).\n >>> from ciocore import data as coredata\n >>> coredata.init(\"cinema4d\")\n >>> hardware_set = coredata.data()[\"instance_types\"]\n <ciocore.hardware_set.HardwareSet object at 0x104c43ee0>\n\n !!! note\n To avoid repetition, we use the implicit initialization for the examples below.\n \"\"\"\n\n self.instance_types = self._build_unique(instance_types)\n self.categories = self._build_categories()\n self.provider = self._get_provider()\n\n def labels(self):\n\"\"\"Get the list of category labels.\n\n Returns:\n list: A list of category labels.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.labels()\n ['CPU', 'GPU']\n\n \"\"\"\n return [c[\"label\"] for c in self.categories]\n\n def number_of_categories(self):\n\"\"\"Get the number of categories in the data.\n\n Returns:\n int: The number of categories.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.number_of_categories()\n 2\n\n \"\"\"\n return len(self.categories)\n\n def recategorize(self, partitioner):\n\"\"\"Recategorize the instance types.\n\n Args:\n partitioner (function): A function that takes an instance type and returns a list of categories to assign to it. The function should return an empty list if the instance type should not be categorized.\n\n Example:\n # Confirm current categories\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> print(hardware_set.labels()\n ['CPU', 'GPU']\n\n # Recategorize\n >>> hardware_set.recategorize(lambda x: [{'label': 'Low cores', 'order': 10}] if x[\"cores\"] < 16 else [{'label': 'High cores', 'order': 20}])\n >>> print(hardware_set.labels()\n ['Low cores', 'High cores']\n \"\"\"\n for key in self.instance_types:\n self.instance_types[key][\"categories\"] = partitioner(\n self.instance_types[key]\n )\n self.categories = self._build_categories()\n\n def find(self, name, category=None):\n\"\"\"Find an instance type by its name (sku).\n\n Args:\n name (str): The name of the instance type.\n\n Returns:\n dict: The instance type or None if not found.\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find(\"n2-highmem-80\")\n {\n 'cores': 80,\n 'description': '80 core, 640GB Mem',\n 'gpu': None,\n 'memory': '640',\n 'name': 'n2-highmem-80',\n 'operating_system': 'linux',\n 'categories': [\n {'label': 'High cores', 'order': 20}\n ]\n }\n\n \"\"\"\n if not category:\n return self.instance_types.get(name)\n\n return self.find_first(\n lambda x: x[\"name\"] == name\n and category in [c[\"label\"] for c in x[\"categories\"]]\n )\n\n def find_category(self, label):\n\"\"\"Find a category by label.\n\n Args:\n label (str): The label of the category.\n\n Returns:\n dict: The category or None if not found.\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find_category(\"High cores\")\n {\n \"label\": \"Low cores\",\n \"content\": [\n {\n \"cores\": 8,\n \"description\": \"8 core, 52GB Mem\",\n \"gpu\": None,\n \"memory\": \"52\",\n \"name\": \"n1-highmem-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 7.2GB Mem\",\n \"gpu\": None,\n \"memory\": \"7.2\",\n \"name\": \"n1-highcpu-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ],\n \"order\": 10\n }\n \"\"\"\n return next((c for c in self.categories if c[\"label\"] == label), None)\n\n def find_all(self, condition):\n\"\"\"Find all instance types that match a condition.\n\n Args:\n condition (function): A function that takes an instance type and returns True or False.\n\n Returns:\n list: A list of instance types that match the condition.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find_all(lambda x: x[\"gpu\"])\n [\n {\n \"cores\": 4,\n \"description\": \"4 core, 15GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"15\",\n \"name\": \"n1-standard-4-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 30GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"30\",\n \"name\": \"n1-standard-8-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ]\n \"\"\"\n result = []\n for key in self.instance_types:\n if condition(self.instance_types[key]):\n result.append(self.instance_types[key])\n return result\n\n def find_first(self, condition):\n\"\"\"Find the first instance type that matches a condition.\n\n Please see find_all() above for more details. This method is just a convenience wrapper around find_all() that returns the first result or None if not found.\n\n Args:\n condition (function): A function that takes an instance type and returns True or False.\n\n Returns:\n dict: The first instance type that matches the condition or None if not found.\n \"\"\"\n return next(iter(self.find_all(condition)), None)\n\n # DEPRECATED\n def get_model(self, with_misc=False):\n\"\"\"Get the categories structure with renaming ready for some specific widget, such as a Qt Combobox.\n\n Deprecated:\n The get_model() method is deprecated. The `with_misc` parameter is no longer used, which means that this function only serves to rename a few keys. What's more, the init function ensures that every instance type has a category. This function is no longer needed. Submitters that use it will work but should be updated to use the categories structure directly as it minimizes the levels of indirection necessary to work with it.\n \"\"\"\n if with_misc:\n logger.warning(\"with_misc is no longer used\")\n result = []\n for category in self.categories:\n result.append(\n {\n \"label\": category[\"label\"],\n \"content\": [\n {\"label\": k[\"description\"], \"value\": k[\"name\"]}\n for k in category[\"content\"]\n ],\n }\n )\n\n return result\n\n # PRIVATE METHODS\n @classmethod\n def _build_unique(cls, instance_types):\n\"\"\"Build a dictionary of instance types using the name field as key. This allows fast lookup by name.\n\n Args:\n instance_types (list): A list of instance types.\n\n Returns:\n dict: A dictionary of instance types with the name field as key.\n \"\"\"\n\n instance_types = cls._rewrite_descriptions(instance_types)\n categories = [\n category\n for it in instance_types\n for category in (it.get(\"categories\") or [])\n ]\n result = {}\n for it in instance_types:\n is_gpu = it.get(\"gpu\", False)\n if categories:\n if it.get(\"categories\") in [[], None]:\n continue\n else:\n # make our own categories GPU/CPU\n it[\"categories\"] = (\n [{\"label\": \"GPU\", \"order\": 2}]\n if is_gpu\n else [{\"label\": \"CPU\", \"order\": 1}]\n )\n result[it[\"name\"]] = it\n\n return result\n\n\n\n\n @classmethod\n def _rewrite_descriptions(cls, instance_types):\n\"\"\"Rewrite the descriptions of the instance types.\n\n If there are both OS types, then the descriptions are prefixed with the OS type.\n\n Args:\n instance_types (list): A list of instance types.\n\n Returns:\n list: A list of instance types with rewritten descriptions.\n \"\"\"\n if not instance_types:\n return instance_types\n\n first_os = instance_types[0][\"operating_system\"]\n dual_platforms = next((it for it in instance_types if it[\"operating_system\"] != first_os), False)\n\n if dual_platforms:\n for it in instance_types:\n flat_dict = flatten_dict(it)\n is_gpu = \"gpu_count\" in flat_dict\n if is_gpu:\n it[\"description\"] = DESCRIPTION_TEMPLATE_OS[\"gpu\"].format(flat_dict)\n else:\n it[\"description\"] = DESCRIPTION_TEMPLATE_OS[\"cpu\"].format(flat_dict)\n else:\n for it in instance_types:\n flat_dict = flatten_dict(it)\n is_gpu = \"gpu_count\" in flat_dict\n if is_gpu:\n it[\"description\"] = DESCRIPTION_TEMPLATE[\"gpu\"].format(flat_dict)\n else:\n it[\"description\"] = DESCRIPTION_TEMPLATE[\"cpu\"].format(*flat_dict)\n\n return instance_types\n\n def _build_categories(self):\n\"\"\"Build a sorted list of categories where each category contains a sorted list of machines.\n\n Returns:\n list: A list of categories where each category is a dictionary with keys: `label`, `content`, and `order`.\n \"\"\"\n\n dikt = {}\n for key in self.instance_types:\n it = self.instance_types[key]\n categories = it[\"categories\"]\n for category in categories:\n label = category[\"label\"]\n if label not in dikt:\n dikt[label] = {\n \"label\": label,\n \"content\": [],\n \"order\": category[\"order\"],\n }\n dikt[label][\"content\"].append(it)\n\n result = []\n for label in dikt:\n category = dikt[label]\n category[\"content\"].sort(key=lambda k: (k[\"cores\"], k[\"memory\"]))\n result.append(category)\n return sorted(result, key=lambda k: k[\"order\"])\n\n def _get_provider(self):\n\"\"\"Get the provider from the first instance type.\n\n Returns:\n str: The provider.\n \"\"\"\n first_name = next(iter(self.instance_types))\n if not first_name:\n return None\n if first_name.startswith(\"cw-\"):\n return \"cw\"\n if \".\" in first_name:\n return \"aws\"\n return \"gcp\"\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.__init__","title":"<code>__init__(self, instance_types)</code> <code>special</code>","text":"<p>Initialize the HardwareSet with a list of instance types. Typically, you would access the HardwareSet through the ciocore.data.data() function, which initializes it for you. However, you can also initialize it directly with a list of instance types straight from ciocore.api_client. The difference being that the latter contains all instance types, whereas the former contains only the instance types compatible with the products you have specified, as well as being cached.</p> <p>Parameters:</p> Name Type Description Default <code>instance_types</code> <code>list</code> <p>A list of instance types.</p> required <p>Returns:</p> Type Description <code>HardwareSet</code> <p>The initialized HardwareSet.</p> <p>Examples:</p>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.__init__--initialize-with-a-list-of-instance-types","title":"Initialize with a list of instance types","text":"<pre><code>>>> from ciocore import api_client\n>>> from ciocore.hardware_set import HardwareSet\n>>> instance_types = api_client.request_instance_types()\n>>> hardware_set = HardwareSet(instance_types)\n<ciocore.hardware_set.HardwareSet object at 0x104c43d30>\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.__init__--initialize-implicitly-with-a-list-of-instance-types-from-ciocoredata-recommended","title":"Initialize implicitly with a list of instance types from ciocore.data (recommended).","text":"<pre><code>>>> from ciocore import data as coredata\n>>> coredata.init(\"cinema4d\")\n>>> hardware_set = coredata.data()[\"instance_types\"]\n<ciocore.hardware_set.HardwareSet object at 0x104c43ee0>\n</code></pre> <p>Note</p> <p>To avoid repetition, we use the implicit initialization for the examples below.</p> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def __init__(self, instance_types):\n\"\"\"Initialize the HardwareSet with a list of instance types.\n Typically, you would access the HardwareSet through the ciocore.data.data() function, which initializes it for you. However, you can also initialize it directly with a list of instance types straight from ciocore.api_client. The difference being that the latter contains all instance types, whereas the former contains only the instance types compatible with the products you have specified, as well as being cached.\n\n Args:\n instance_types (list): A list of instance types.\n\n Returns:\n HardwareSet: The initialized HardwareSet.\n\n Examples:\n ### Initialize with a list of instance types\n >>> from ciocore import api_client\n >>> from ciocore.hardware_set import HardwareSet\n >>> instance_types = api_client.request_instance_types()\n >>> hardware_set = HardwareSet(instance_types)\n <ciocore.hardware_set.HardwareSet object at 0x104c43d30>\n\n ### Initialize implicitly with a list of instance types from ciocore.data (recommended).\n >>> from ciocore import data as coredata\n >>> coredata.init(\"cinema4d\")\n >>> hardware_set = coredata.data()[\"instance_types\"]\n <ciocore.hardware_set.HardwareSet object at 0x104c43ee0>\n\n !!! note\n To avoid repetition, we use the implicit initialization for the examples below.\n \"\"\"\n\n self.instance_types = self._build_unique(instance_types)\n self.categories = self._build_categories()\n self.provider = self._get_provider()\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.labels","title":"<code>labels(self)</code>","text":"<p>Get the list of category labels.</p> <p>Returns:</p> Type Description <code>list</code> <p>A list of category labels.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.labels()\n['CPU', 'GPU']\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def labels(self):\n\"\"\"Get the list of category labels.\n\n Returns:\n list: A list of category labels.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.labels()\n ['CPU', 'GPU']\n\n \"\"\"\n return [c[\"label\"] for c in self.categories]\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.number_of_categories","title":"<code>number_of_categories(self)</code>","text":"<p>Get the number of categories in the data.</p> <p>Returns:</p> Type Description <code>int</code> <p>The number of categories.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.number_of_categories()\n2\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def number_of_categories(self):\n\"\"\"Get the number of categories in the data.\n\n Returns:\n int: The number of categories.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.number_of_categories()\n 2\n\n \"\"\"\n return len(self.categories)\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.recategorize","title":"<code>recategorize(self, partitioner)</code>","text":"<p>Recategorize the instance types.</p> <p>Parameters:</p> Name Type Description Default <code>partitioner</code> <code>function</code> <p>A function that takes an instance type and returns a list of categories to assign to it. The function should return an empty list if the instance type should not be categorized.</p> required <p>Examples:</p>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.recategorize--confirm-current-categories","title":"Confirm current categories","text":"<pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> print(hardware_set.labels()\n['CPU', 'GPU']\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.recategorize--recategorize","title":"Recategorize","text":"<pre><code>>>> hardware_set.recategorize(lambda x: [{'label': 'Low cores', 'order': 10}] if x[\"cores\"] < 16 else [{'label': 'High cores', 'order': 20}])\n>>> print(hardware_set.labels()\n['Low cores', 'High cores']\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def recategorize(self, partitioner):\n\"\"\"Recategorize the instance types.\n\n Args:\n partitioner (function): A function that takes an instance type and returns a list of categories to assign to it. The function should return an empty list if the instance type should not be categorized.\n\n Example:\n # Confirm current categories\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> print(hardware_set.labels()\n ['CPU', 'GPU']\n\n # Recategorize\n >>> hardware_set.recategorize(lambda x: [{'label': 'Low cores', 'order': 10}] if x[\"cores\"] < 16 else [{'label': 'High cores', 'order': 20}])\n >>> print(hardware_set.labels()\n ['Low cores', 'High cores']\n \"\"\"\n for key in self.instance_types:\n self.instance_types[key][\"categories\"] = partitioner(\n self.instance_types[key]\n )\n self.categories = self._build_categories()\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.find","title":"<code>find(self, name, category=None)</code>","text":"<p>Find an instance type by its name (sku).</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>The name of the instance type.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>The instance type or None if not found.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.find(\"n2-highmem-80\")\n{\n 'cores': 80,\n 'description': '80 core, 640GB Mem',\n 'gpu': None,\n 'memory': '640',\n 'name': 'n2-highmem-80',\n 'operating_system': 'linux',\n 'categories': [\n {'label': 'High cores', 'order': 20}\n ]\n}\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def find(self, name, category=None):\n\"\"\"Find an instance type by its name (sku).\n\n Args:\n name (str): The name of the instance type.\n\n Returns:\n dict: The instance type or None if not found.\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find(\"n2-highmem-80\")\n {\n 'cores': 80,\n 'description': '80 core, 640GB Mem',\n 'gpu': None,\n 'memory': '640',\n 'name': 'n2-highmem-80',\n 'operating_system': 'linux',\n 'categories': [\n {'label': 'High cores', 'order': 20}\n ]\n }\n\n \"\"\"\n if not category:\n return self.instance_types.get(name)\n\n return self.find_first(\n lambda x: x[\"name\"] == name\n and category in [c[\"label\"] for c in x[\"categories\"]]\n )\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.find_category","title":"<code>find_category(self, label)</code>","text":"<p>Find a category by label.</p> <p>Parameters:</p> Name Type Description Default <code>label</code> <code>str</code> <p>The label of the category.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>The category or None if not found.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.find_category(\"High cores\")\n{\n \"label\": \"Low cores\",\n \"content\": [\n {\n \"cores\": 8,\n \"description\": \"8 core, 52GB Mem\",\n \"gpu\": None,\n \"memory\": \"52\",\n \"name\": \"n1-highmem-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 7.2GB Mem\",\n \"gpu\": None,\n \"memory\": \"7.2\",\n \"name\": \"n1-highcpu-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ],\n \"order\": 10\n}\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def find_category(self, label):\n\"\"\"Find a category by label.\n\n Args:\n label (str): The label of the category.\n\n Returns:\n dict: The category or None if not found.\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find_category(\"High cores\")\n {\n \"label\": \"Low cores\",\n \"content\": [\n {\n \"cores\": 8,\n \"description\": \"8 core, 52GB Mem\",\n \"gpu\": None,\n \"memory\": \"52\",\n \"name\": \"n1-highmem-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 7.2GB Mem\",\n \"gpu\": None,\n \"memory\": \"7.2\",\n \"name\": \"n1-highcpu-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ],\n \"order\": 10\n }\n \"\"\"\n return next((c for c in self.categories if c[\"label\"] == label), None)\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.find_all","title":"<code>find_all(self, condition)</code>","text":"<p>Find all instance types that match a condition.</p> <p>Parameters:</p> Name Type Description Default <code>condition</code> <code>function</code> <p>A function that takes an instance type and returns True or False.</p> required <p>Returns:</p> Type Description <code>list</code> <p>A list of instance types that match the condition.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.find_all(lambda x: x[\"gpu\"])\n[\n {\n \"cores\": 4,\n \"description\": \"4 core, 15GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"15\",\n \"name\": \"n1-standard-4-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 30GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"30\",\n \"name\": \"n1-standard-8-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n]\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def find_all(self, condition):\n\"\"\"Find all instance types that match a condition.\n\n Args:\n condition (function): A function that takes an instance type and returns True or False.\n\n Returns:\n list: A list of instance types that match the condition.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find_all(lambda x: x[\"gpu\"])\n [\n {\n \"cores\": 4,\n \"description\": \"4 core, 15GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"15\",\n \"name\": \"n1-standard-4-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 30GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"30\",\n \"name\": \"n1-standard-8-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ]\n \"\"\"\n result = []\n for key in self.instance_types:\n if condition(self.instance_types[key]):\n result.append(self.instance_types[key])\n return result\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.find_first","title":"<code>find_first(self, condition)</code>","text":"<p>Find the first instance type that matches a condition.</p> <p>Please see find_all() above for more details. This method is just a convenience wrapper around find_all() that returns the first result or None if not found.</p> <p>Parameters:</p> Name Type Description Default <code>condition</code> <code>function</code> <p>A function that takes an instance type and returns True or False.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>The first instance type that matches the condition or None if not found.</p> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def find_first(self, condition):\n\"\"\"Find the first instance type that matches a condition.\n\n Please see find_all() above for more details. This method is just a convenience wrapper around find_all() that returns the first result or None if not found.\n\n Args:\n condition (function): A function that takes an instance type and returns True or False.\n\n Returns:\n dict: The first instance type that matches the condition or None if not found.\n \"\"\"\n return next(iter(self.find_all(condition)), None)\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.get_model","title":"<code>get_model(self, with_misc=False)</code>","text":"<p>Get the categories structure with renaming ready for some specific widget, such as a Qt Combobox.</p> <p>Deprecated</p> <p>The get_model() method is deprecated. The <code>with_misc</code> parameter is no longer used, which means that this function only serves to rename a few keys. What's more, the init function ensures that every instance type has a category. This function is no longer needed. Submitters that use it will work but should be updated to use the categories structure directly as it minimizes the levels of indirection necessary to work with it.</p> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def get_model(self, with_misc=False):\n\"\"\"Get the categories structure with renaming ready for some specific widget, such as a Qt Combobox.\n\n Deprecated:\n The get_model() method is deprecated. The `with_misc` parameter is no longer used, which means that this function only serves to rename a few keys. What's more, the init function ensures that every instance type has a category. This function is no longer needed. Submitters that use it will work but should be updated to use the categories structure directly as it minimizes the levels of indirection necessary to work with it.\n \"\"\"\n if with_misc:\n logger.warning(\"with_misc is no longer used\")\n result = []\n for category in self.categories:\n result.append(\n {\n \"label\": category[\"label\"],\n \"content\": [\n {\"label\": k[\"description\"], \"value\": k[\"name\"]}\n for k in category[\"content\"]\n ],\n }\n )\n\n return result\n</code></pre>"},{"location":"apidoc/package_environment/","title":"ciocore.package_environment","text":"<p>Manage environment variables for both Windows and Linux render nodes.</p>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment","title":"<code> PackageEnvironment </code>","text":"Source code in <code>ciocore/package_environment.py</code> <pre><code>class PackageEnvironment(object):\n\n\n def __init__(self, env_list=None, platform=None):\n\"\"\"\n Encapsulate a list of environment variables.\n\n Typically, one would initialize a PackageEnvironment with a package, and then modify by adding more packages or lists of variables. Extra variables can be added by the customer, or programmatically such as during asset scraping.\n\n Args:\n env_list (object\|list): An object that provides a list of dictionaries with properties: `name`, `value`, and `merge_policy`.\n platform (str): If the env_list is a regular list, then this is required.\n\n Args are delegated to [extend()](/package_environment/#ciocore.package_environment.PackageEnvironment.extend).\n \"\"\"\n self.platform = None\n self._env = {}\n\n self.extend(env_list, platform)\n\n\n def extend(self, env_list, platform=None):\n\"\"\"\n Extend the Package environment with the given variable specifications.\n\n Args:\n env_list (object\|list): Either:\n A list of dictionaries with properties: `name`, `value`, and `merge_policy`.\n * An object with an `environment` key that contains a list of the dictionaries described above. The latter is the structure of a package. Therefore we can initialize or extend a PackageEnvironment with a package.\n platform (str): Defaults to `None`. If env_list is a package, then the platform is taken from the package and the `platform` keyword is ignored. If env_list is a list, then if this is the first add, a platform should be specified, otherwise it will default to linux.\n\n The first time data is added to a PackageEnvironment, the platform is set in stone. Subsequent `adds` that try to change the platform are considered an error.\n\n Each variable to be added specifies a merge_policy: `append`, `prepend`, or `exclusive` `append` and `prepend` can be thought of as lists= types. Once an individual variable has been initialized as a list, it can't be changed to `exclusive`. This means:\n\n 1. It's not possible to overwrite variables that have been added as `append` or `prepend`.\n 2. Exclusive variables are always overwritten by subsequent adds.\n\n Raises:\n ValueError: Either an attempt to change the platform once initialized, or an invalid merge policy.\n\n\n Example:\n >>> from ciocore import api_client, package_tree, package_environment\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> one_dcc_name = pt.supported_host_names()[0]\n cinema4d 21.209.RB305619 linux\n\n >>> pkg = pt.find_by_name(one_dcc_name)\n >>> pe = package_environment.PackageEnvironment(pkg)\n >>> print(dict(pe))\n {\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n }\n\n >>> extra_env = [\n {\"name\":\"PATH\", \"value\": \"/my/custom/scripts\", \"merge_policy\":\"append\"},\n {\"name\":\"DEBUG_MODE\", \"value\": \"1\", \"merge_policy\":\"exclusive\"}\n ]\n >>> pe.extend(extra_env)\n >>> print(dict(pe))\n {\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin:/my/custom/scripts\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n \"DEBUG_MODE\": \"1\",\n }\n \"\"\"\n\n if not env_list:\n return\n\n try:\n others = env_list[\"environment\"]\n requested_platform = env_list.get(\"platform\")\n except TypeError:\n others = env_list\n requested_platform = platform\n\n if not self.platform:\n self.platform = requested_platform or \"linux\"\n elif requested_platform and requested_platform != self.platform:\n raise ValueError(\"Can't change platform once initialized.\")\n\n for var in others:\n name = var[\"name\"]\n value = var[\"value\"]\n policy = var[\"merge_policy\"]\n if policy not in [\"append\", \"prepend\", \"exclusive\"]:\n raise ValueError(\"Unexpected merge policy: %s\" % policy)\n\n if policy == \"append\":\n self._append(name, value)\n elif policy == \"prepend\":\n self._prepend(name, value)\n else:\n self._set(name, value)\n\n\n def _set(self, name, value):\n\"\"\"Set the value of an exclusive variable.\n\n Can be overwritten by subsequent adds.\n\n It is an error if the variable has already been declared with policy=append.\n \"\"\"\n if self._env.get(name) and isinstance(self._env[name], list):\n raise ValueError(\n \"Can't change merge policy for '{}' from 'append' to 'exclusive'.\".format(name)\n )\n self._env[name] = value\n\n def _append(self, name, value):\n\"\"\"Set the value of an append variable.\n\n Can be appended to with subsequent adds.\n\n It is an error if the variable has already been declared with policy=exclusive.\n \"\"\"\n if self._env.get(name):\n if not isinstance(self._env[name], list):\n raise ValueError(\n \"Can't change merge policy for '{}' from 'exclusive' to 'append'.\".format(name)\n )\n else:\n self._env[name] = []\n self._env[name].append(value)\n\n def _prepend(self, name, value):\n\"\"\"Set the value of an append/prepend variable.\n\n Can be appended to with subsequent adds.\n\n It is an error if the variable has already been declared with policy=exclusive.\n \"\"\"\n if self._env.get(name):\n if not isinstance(self._env[name], list):\n raise ValueError(\n \"Can't change merge policy for '{}' from 'exclusive' to 'prepend'.\".format(name)\n )\n else:\n self._env[name] = []\n self._env[name].insert(0, value)\n\n def __iter__(self):\n\"\"\"Cast the object as a dict.\"\"\"\n sep = \";\" if self.platform == \"windows\" else \":\"\n for key in self._env:\n var = self._env[key]\n if isinstance(var, list):\n yield key, sep.join(var)\n else:\n yield key, var\n\n def __getitem__(self, key):\n\"\"\"Allow access by key.\"\"\"\n sep = \";\" if self.platform == \"windows\" else \":\"\n var = self._env.__getitem__(key)\n if isinstance(var, list):\n return sep.join(var)\n return var\n</code></pre>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment.__init__","title":"<code>__init__(self, env_list=None, platform=None)</code> <code>special</code>","text":"<p>Encapsulate a list of environment variables.</p> <p>Typically, one would initialize a PackageEnvironment with a package, and then modify by adding more packages or lists of variables. Extra variables can be added by the customer, or programmatically such as during asset scraping.</p> <p>Parameters:</p> Name Type Description Default <code>env_list</code> <code>object\|list</code> <p>An object that provides a list of dictionaries with properties: <code>name</code>, <code>value</code>, and <code>merge_policy</code>.</p> <code>None</code> <code>platform</code> <code>str</code> <p>If the env_list is a regular list, then this is required.</p> <code>None</code> <p>Args are delegated to extend().</p> Source code in <code>ciocore/package_environment.py</code> <pre><code>def __init__(self, env_list=None, platform=None):\n\"\"\"\n Encapsulate a list of environment variables.\n\n Typically, one would initialize a PackageEnvironment with a package, and then modify by adding more packages or lists of variables. Extra variables can be added by the customer, or programmatically such as during asset scraping.\n\n Args:\n env_list (object\|list): An object that provides a list of dictionaries with properties: `name`, `value`, and `merge_policy`.\n platform (str): If the env_list is a regular list, then this is required.\n\n Args are delegated to [extend()](/package_environment/#ciocore.package_environment.PackageEnvironment.extend).\n \"\"\"\n self.platform = None\n self._env = {}\n\n self.extend(env_list, platform)\n</code></pre>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment.extend","title":"<code>extend(self, env_list, platform=None)</code>","text":"<p>Extend the Package environment with the given variable specifications.</p> <p>Parameters:</p> Name Type Description Default <code>env_list</code> <code>object\|list</code> <ul> <li>A list of dictionaries with properties: <code>name</code>, <code>value</code>, and <code>merge_policy</code>.</li> <li>An object with an <code>environment</code> key that contains a list of the dictionaries described above. The latter is the structure of a package. Therefore we can initialize or extend a PackageEnvironment with a package.</li> </ul> required <code>platform</code> <code>str</code> <p>Defaults to <code>None</code>. If env_list is a package, then the platform is taken from the package and the <code>platform</code> keyword is ignored. If env_list is a list, then if this is the first add, a platform should be specified, otherwise it will default to linux.</p> <code>None</code> <p>The first time data is added to a PackageEnvironment, the platform is set in stone. Subsequent <code>adds</code> that try to change the platform are considered an error.</p> <p>Each variable to be added specifies a merge_policy: <code>append</code>, <code>prepend</code>, or <code>exclusive</code> <code>append</code> and <code>prepend</code> can be thought of as lists= types. Once an individual variable has been initialized as a list, it can't be changed to <code>exclusive</code>. This means:</p> <ol> <li>It's not possible to overwrite variables that have been added as <code>append</code> or <code>prepend</code>.</li> <li>Exclusive variables are always overwritten by subsequent adds.</li> </ol> <p>Exceptions:</p> Type Description <code>ValueError</code> <p>Either an attempt to change the platform once initialized, or an invalid merge policy.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree, package_environment\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> one_dcc_name = pt.supported_host_names()[0]\ncinema4d 21.209.RB305619 linux\n</code></pre> <pre><code>>>> pkg = pt.find_by_name(one_dcc_name)\n>>> pe = package_environment.PackageEnvironment(pkg)\n>>> print(dict(pe))\n{\n\"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin\",\n\"g_licenseServerRLM\": \"conductor-rlm:6112\",\n\"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n\"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n}\n</code></pre> <pre><code>>>> extra_env = [\n {\"name\":\"PATH\", \"value\": \"/my/custom/scripts\", \"merge_policy\":\"append\"},\n {\"name\":\"DEBUG_MODE\", \"value\": \"1\", \"merge_policy\":\"exclusive\"}\n]\n>>> pe.extend(extra_env)\n>>> print(dict(pe))\n{\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin:/my/custom/scripts\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n \"DEBUG_MODE\": \"1\",\n}\n</code></pre> Source code in <code>ciocore/package_environment.py</code> <pre><code>def extend(self, env_list, platform=None):\n\"\"\"\n Extend the Package environment with the given variable specifications.\n\n Args:\n env_list (object\|list): Either:\n * A list of dictionaries with properties: `name`, `value`, and `merge_policy`.\n * An object with an `environment` key that contains a list of the dictionaries described above. The latter is the structure of a package. Therefore we can initialize or extend a PackageEnvironment with a package.\n platform (str): Defaults to `None`. If env_list is a package, then the platform is taken from the package and the `platform` keyword is ignored. If env_list is a list, then if this is the first add, a platform should be specified, otherwise it will default to linux.\n\n The first time data is added to a PackageEnvironment, the platform is set in stone. Subsequent `adds` that try to change the platform are considered an error.\n\n Each variable to be added specifies a merge_policy: `append`, `prepend`, or `exclusive` `append` and `prepend` can be thought of as lists= types. Once an individual variable has been initialized as a list, it can't be changed to `exclusive`. This means:\n\n 1. It's not possible to overwrite variables that have been added as `append` or `prepend`.\n 2. Exclusive variables are always overwritten by subsequent adds.\n\n Raises:\n ValueError: Either an attempt to change the platform once initialized, or an invalid merge policy.\n\n\n Example:\n >>> from ciocore import api_client, package_tree, package_environment\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> one_dcc_name = pt.supported_host_names()[0]\n cinema4d 21.209.RB305619 linux\n\n >>> pkg = pt.find_by_name(one_dcc_name)\n >>> pe = package_environment.PackageEnvironment(pkg)\n >>> print(dict(pe))\n {\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n }\n\n >>> extra_env = [\n {\"name\":\"PATH\", \"value\": \"/my/custom/scripts\", \"merge_policy\":\"append\"},\n {\"name\":\"DEBUG_MODE\", \"value\": \"1\", \"merge_policy\":\"exclusive\"}\n ]\n >>> pe.extend(extra_env)\n >>> print(dict(pe))\n {\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin:/my/custom/scripts\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n \"DEBUG_MODE\": \"1\",\n }\n \"\"\"\n\n if not env_list:\n return\n\n try:\n others = env_list[\"environment\"]\n requested_platform = env_list.get(\"platform\")\n except TypeError:\n others = env_list\n requested_platform = platform\n\n if not self.platform:\n self.platform = requested_platform or \"linux\"\n elif requested_platform and requested_platform != self.platform:\n raise ValueError(\"Can't change platform once initialized.\")\n\n for var in others:\n name = var[\"name\"]\n value = var[\"value\"]\n policy = var[\"merge_policy\"]\n if policy not in [\"append\", \"prepend\", \"exclusive\"]:\n raise ValueError(\"Unexpected merge policy: %s\" % policy)\n\n if policy == \"append\":\n self._append(name, value)\n elif policy == \"prepend\":\n self._prepend(name, value)\n else:\n self._set(name, value)\n</code></pre>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment.__iter__","title":"<code>__iter__(self)</code> <code>special</code>","text":"<p>Cast the object as a dict.</p> Source code in <code>ciocore/package_environment.py</code> <pre><code>def __iter__(self):\n\"\"\"Cast the object as a dict.\"\"\"\n sep = \";\" if self.platform == \"windows\" else \":\"\n for key in self._env:\n var = self._env[key]\n if isinstance(var, list):\n yield key, sep.join(var)\n else:\n yield key, var\n</code></pre>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment.__getitem__","title":"<code>__getitem__(self, key)</code> <code>special</code>","text":"<p>Allow access by key.</p> Source code in <code>ciocore/package_environment.py</code> <pre><code>def __getitem__(self, key):\n\"\"\"Allow access by key.\"\"\"\n sep = \";\" if self.platform == \"windows\" else \":\"\n var = self._env.__getitem__(key)\n if isinstance(var, list):\n return sep.join(var)\n return var\n</code></pre>"},{"location":"apidoc/package_tree/","title":"ciocore.package_tree","text":"<p>A class to provide available packages as DAG structure. In reality however, the structure is just two levels deep: hosts and plugins. </p> <ul> <li>DCCs such as Maya and Cinema4D are top-level host packages. </li> <li>Renderers and other plugins are children of those hosts.</li> </ul> <p>Methods are provided to traverse the tree to find packages by name, version, platform and so on. If you are writing submission tools there's no need to create a Package tree directly. It is recommended to use the singleton module: ciocore.data.</p> <p>The only functions you should need from this module are:</p> <ul> <li>supported_host_names()</li> <li>supported_plugins()</li> </ul>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree","title":"<code> PackageTree </code>","text":"Source code in <code>ciocore/package_tree.py</code> <pre><code>class PackageTree(object):\n def __init__(self, packages, host_products, kwargs):\n\"\"\"Build the tree with a list of packages.\n\n Args:\n packages (list): List of packages direct from the [Conductor packages endpoint](https://dashboard.conductortech.com/api/v1/ee/packages).\n\n host_products: Filter the tree to contain only top-level host packages of products specified in this list and their plugins. If there are no host_products specified, and the product keyword is omitted, the tree contains all packages.\n\n Keyword Args:\n product (str): Build the tree from versions of a single product and its compatible plugins. Defaults to `None`, in which case the tree is built from all packages. It is an error to specify both host_products and product. If a nonexistent product is given, the PackageTree is empty. By specifying `product`, you can build the object based on a single plugin product.\n platforms (set): Build the tree from versions for a specific platform. Defaults to the set `{\"linux\", \"windows\"}`.\n\n Raises:\n KeyError: An invalid platform was provided.\n ValueError: Cannot choose both product and host_products.\n\n Example:\n >>> from ciocore import api_client, package_tree\n # Request packages as a flat list from Conductor.\n >>> packages = api_client.request_software_packages()\n # Build tree of dependencies from packages list\n >>> pt = package_tree.PackageTree(packages, \"cinema4d\", \"maya-io\")\n >>> for path in pt.to_path_list():\n >>> print(path)\n cinema4d 22.118.RB320081 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\n maya-io 2022.SP3 linux\n \"\"\"\n\n\n platforms = kwargs.get(\"platforms\",PLATFORMS)\n product=kwargs.get(\"product\")\n\n unknown_platforms = set(platforms) - PLATFORMS\n if unknown_platforms:\n raise KeyError(\"Unrecognized platform {}\".format(\" \".join(unknown_platforms)))\n\n if product and host_products:\n raise ValueError(\"You cannot choose both product and host_products.\")\n\n packages = [_clean_package(p) for p in packages if p[\"platform\"] in platforms]\n\n root_ids = [] \n if product:\n root_ids = [p[\"package_id\"] for p in packages if p[\"product\"] == product]\n else:\n for p in packages:\n if not p[\"plugin_host_product\"]:\n if p[\"product\"] in host_products or not host_products:\n root_ids.append(p[\"package_id\"])\n\n self._tree = _build_tree(packages, {\"children\": [], \"plugins\": root_ids})\n\n def supported_host_names(self):\n\"\"\"\n All host names from the software tree.\n\n These names can be used to populate a dropdown menu. Then a single selection from that menu\n can be used to retrieve the complete package in order to generate an environment dictionary\n and get package IDs.\n\n Returns:\n list(str): Fully qualified DCC hosts of the form: `product version platform`.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.supported_host_names()\n cinema4d 21.209.RB305619 linux\n cinema4d 22.116.RB316423 linux\n cinema4d 22.118.RB320081 linux\n cinema4d 23.110.RB330286 linux\n cinema4d 24.111 linux\n cinema4d 24.111 windows\n \"\"\"\n\n paths = []\n for pkg in self._tree[\"children\"]:\n paths.append(to_name(pkg))\n return sorted(paths)\n\n def supported_plugins(self, host):\n\"\"\"\n Find the plugins that are children of the given DCC host.\n\n The result does not contain platform information since we assume that plugins are compatible with the DCC host that was used to request them.\n\n Args:\n host (str): Name of the DCC host, typically one of the entries returned by [supported_host_names()](/package_tree/#ciocore.package_tree.PackageTree.supported_host_names).\n\n Returns:\n list(dict): Each entry contains a plugin product and a list of versions.\n\n Example:\n >>> from ciocore import api_client, package_tree \n >>> packages = api_client.request_software_packages() \n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\") \n >>> name = pt.supported_host_names()[0]\n >>> pt.supported_plugins(name)\n [\n {\n \"plugin\": \"arnold-cinema4d\",\n \"versions\": [\n \"3.3.2.100\",\n \"3.3.3.0\"\n ]\n },\n {\n \"plugin\": \"redshift-cinema4d\",\n \"versions\": [\n \"2.6.54\",\n \"2.6.56\",\n \"3.0.21\",\n \"3.0.22\",\n ],\n },\n ]\n \"\"\"\n\n try:\n subtree = self.find_by_name(host)\n plugin_versions = _to_path_list(subtree)\n except TypeError:\n return []\n\n if not plugin_versions:\n return []\n\n plugin_dict = {}\n for plugin, version, _ in [pv.split(\" \") for pv in plugin_versions]:\n if plugin not in plugin_dict:\n plugin_dict[plugin] = []\n plugin_dict[plugin].append(version)\n\n # convert to list so it can be sorted\n plugins = []\n for key in plugin_dict:\n plugins.append({\"plugin\": key, \"versions\": sorted(plugin_dict[key])})\n\n return sorted(plugins, key=lambda k: k[\"plugin\"])\n\n def find_by_name(self, name, limit=None):\n\"\"\"\n Search the tree for a product with the given name.\n\n Args:\n name (str): The name constructed from the package using to_name(). It must be an exact match with product, version, and platform. For example: `maya 2018.0 windows`\n\n Keyword Args:\n limit (int): Limit the search depth. Defaults to `None`.\n\n Returns:\n object: The package that matches.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.find_by_name(\"redshift-cinema4d 3.0.64 linux\")\n {\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '64',\n 'vendor': 'maxon',\n 'children': [],\n ...\n }\n \"\"\"\n\n return _find_by_name(self._tree, name, limit, 0)\n\n def find_by_path(self, path):\n\"\"\"\n Find the package uniquely described by the given path.\n\n The path is of the form returned by the to_path_list() method.\n\n Args:\n path (str): The path\n\n Returns:\n object: The package or None if no package exists with the given path.\n\n Example:\n >>> from ciocore import api_client, package_tree, package_environment\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.find_by_path(\"cinema4d 24.111 linux/redshift-cinema4d 3.0.62 linux\")\n {\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '62',\n 'vendor': 'maxon',\n 'children': [],\n 'plugin_host_version': \"24\",\n ...\n }\n \"\"\"\n return _find_by_path(self._tree, path)\n\n\n def to_path_list(self, name=None):\n\"\"\"\n Get paths to all nodes.\n\n This is useful for populating a chooser to choose packages fully qualified by path.\n Houdini's tree widget, for example, takes the below format unchanged and generates the\n appropriate UI.\n\n Args:\n name (str): Get paths below the tree represented by the name. Defaults to None (root node).\n\n Returns:\n list(str): Paths to all nodes in the tree.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.to_path_list()\n cinema4d 22.118.RB320081 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.22 linux\n cinema4d 22.118.RB320081 linux/arnold-cinema4d 3.3.2.100 linux\n ...\n\n >>> pt.to_path_list(name=\"cinema4d 24.111 linux\")\n redshift-cinema4d 3.0.57 linux\n redshift-cinema4d 3.0.62 linux\n redshift-cinema4d 3.0.45 linux\n redshift-cinema4d 3.0.64 linux\n \"\"\"\n if name:\n subtree = self.find_by_name(name)\n return _to_path_list(subtree)\n return _to_path_list(self._tree)\n\n def platforms(self):\n\"\"\"\n Get the platforms represented by packages in the tree.\n\n Returns:\n set: The set of platforms.\n \"\"\"\n\n # No need to recurse. Plugins are assumed to be compatible with the host.\n return set([host[\"platform\"] for host in self._tree[\"children\"]])\n\n def json(self):\n\"\"\"\n The whole tree of softwware as json.\n\n Returns:\n str: JSON.\n\n \"\"\"\n return json.dumps(self._tree)\n\n def __bool__(self):\n return True if self._tree[\"children\"] else False\n\n def __nonzero__(self):\n # Python 2.7\n return self.__bool__()\n\n def as_dict(self):\n\"\"\"\n Returns:\n dict: The underlying software dictionary.\n\n \"\"\"\n return self._tree\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.__init__","title":"<code>__init__(self, packages, host_products, kwargs)</code> <code>special</code>","text":"<p>Build the tree with a list of packages.</p> <p>Parameters:</p> Name Type Description Default <code>packages</code> <code>list</code> <p>List of packages direct from the Conductor packages endpoint.</p> required <code>host_products</code> <p>Filter the tree to contain only top-level host packages of products specified in this list and their plugins. If there are no host_products specified, and the product keyword is omitted, the tree contains all packages.</p> <code>()</code> <p>Keyword arguments:</p> Name Type Description <code>product</code> <code>str</code> <p>Build the tree from versions of a single product and its compatible plugins. Defaults to <code>None</code>, in which case the tree is built from all packages. It is an error to specify both host_products and product. If a nonexistent product is given, the PackageTree is empty. By specifying <code>product</code>, you can build the object based on a single plugin product.</p> <code>platforms</code> <code>set</code> <p>Build the tree from versions for a specific platform. Defaults to the set <code>{\"linux\", \"windows\"}</code>.</p> <p>Exceptions:</p> Type Description <code>KeyError</code> <p>An invalid platform was provided.</p> <code>ValueError</code> <p>Cannot choose both product and host_products.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n# Request packages as a flat list from Conductor.\n>>> packages = api_client.request_software_packages()\n# Build tree of dependencies from packages list\n>>> pt = package_tree.PackageTree(packages, \"cinema4d\", \"maya-io\")\n>>> for path in pt.to_path_list():\n>>> print(path)\ncinema4d 22.118.RB320081 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\nmaya-io 2022.SP3 linux\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def __init__(self, packages, host_products, kwargs):\n\"\"\"Build the tree with a list of packages.\n\n Args:\n packages (list): List of packages direct from the [Conductor packages endpoint](https://dashboard.conductortech.com/api/v1/ee/packages).\n\n host_products: Filter the tree to contain only top-level host packages of products specified in this list and their plugins. If there are no host_products specified, and the product keyword is omitted, the tree contains all packages.\n\n Keyword Args:\n product (str): Build the tree from versions of a single product and its compatible plugins. Defaults to `None`, in which case the tree is built from all packages. It is an error to specify both host_products and product. If a nonexistent product is given, the PackageTree is empty. By specifying `product`, you can build the object based on a single plugin product.\n platforms (set): Build the tree from versions for a specific platform. Defaults to the set `{\"linux\", \"windows\"}`.\n\n Raises:\n KeyError: An invalid platform was provided.\n ValueError: Cannot choose both product and host_products.\n\n Example:\n >>> from ciocore import api_client, package_tree\n # Request packages as a flat list from Conductor.\n >>> packages = api_client.request_software_packages()\n # Build tree of dependencies from packages list\n >>> pt = package_tree.PackageTree(packages, \"cinema4d\", \"maya-io\")\n >>> for path in pt.to_path_list():\n >>> print(path)\n cinema4d 22.118.RB320081 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\n maya-io 2022.SP3 linux\n \"\"\"\n\n\n platforms = kwargs.get(\"platforms\",PLATFORMS)\n product=kwargs.get(\"product\")\n\n unknown_platforms = set(platforms) - PLATFORMS\n if unknown_platforms:\n raise KeyError(\"Unrecognized platform {}\".format(\" \".join(unknown_platforms)))\n\n if product and host_products:\n raise ValueError(\"You cannot choose both product and host_products.\")\n\n packages = [_clean_package(p) for p in packages if p[\"platform\"] in platforms]\n\n root_ids = [] \n if product:\n root_ids = [p[\"package_id\"] for p in packages if p[\"product\"] == product]\n else:\n for p in packages:\n if not p[\"plugin_host_product\"]:\n if p[\"product\"] in host_products or not host_products:\n root_ids.append(p[\"package_id\"])\n\n self._tree = _build_tree(packages, {\"children\": [], \"plugins\": root_ids})\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.supported_host_names","title":"<code>supported_host_names(self)</code>","text":"<p>All host names from the software tree.</p> <p>These names can be used to populate a dropdown menu. Then a single selection from that menu can be used to retrieve the complete package in order to generate an environment dictionary and get package IDs.</p> <p>Returns:</p> Type Description <code>list(str)</code> <p>Fully qualified DCC hosts of the form: <code>product version platform</code>.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> pt.supported_host_names()\ncinema4d 21.209.RB305619 linux\ncinema4d 22.116.RB316423 linux\ncinema4d 22.118.RB320081 linux\ncinema4d 23.110.RB330286 linux\ncinema4d 24.111 linux\ncinema4d 24.111 windows\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def supported_host_names(self):\n\"\"\"\n All host names from the software tree.\n\n These names can be used to populate a dropdown menu. Then a single selection from that menu\n can be used to retrieve the complete package in order to generate an environment dictionary\n and get package IDs.\n\n Returns:\n list(str): Fully qualified DCC hosts of the form: `product version platform`.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.supported_host_names()\n cinema4d 21.209.RB305619 linux\n cinema4d 22.116.RB316423 linux\n cinema4d 22.118.RB320081 linux\n cinema4d 23.110.RB330286 linux\n cinema4d 24.111 linux\n cinema4d 24.111 windows\n \"\"\"\n\n paths = []\n for pkg in self._tree[\"children\"]:\n paths.append(to_name(pkg))\n return sorted(paths)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.supported_plugins","title":"<code>supported_plugins(self, host)</code>","text":"<p>Find the plugins that are children of the given DCC host.</p> <p>The result does not contain platform information since we assume that plugins are compatible with the DCC host that was used to request them.</p> <p>Parameters:</p> Name Type Description Default <code>host</code> <code>str</code> <p>Name of the DCC host, typically one of the entries returned by supported_host_names().</p> required <p>Returns:</p> Type Description <code>list(dict)</code> <p>Each entry contains a plugin product and a list of versions.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree \n>>> packages = api_client.request_software_packages() \n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\") \n>>> name = pt.supported_host_names()[0]\n>>> pt.supported_plugins(name)\n[\n {\n \"plugin\": \"arnold-cinema4d\",\n \"versions\": [\n \"3.3.2.100\",\n \"3.3.3.0\"\n ]\n },\n {\n \"plugin\": \"redshift-cinema4d\",\n \"versions\": [\n \"2.6.54\",\n \"2.6.56\",\n \"3.0.21\",\n \"3.0.22\",\n ],\n },\n]\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def supported_plugins(self, host):\n\"\"\"\n Find the plugins that are children of the given DCC host.\n\n The result does not contain platform information since we assume that plugins are compatible with the DCC host that was used to request them.\n\n Args:\n host (str): Name of the DCC host, typically one of the entries returned by [supported_host_names()](/package_tree/#ciocore.package_tree.PackageTree.supported_host_names).\n\n Returns:\n list(dict): Each entry contains a plugin product and a list of versions.\n\n Example:\n >>> from ciocore import api_client, package_tree \n >>> packages = api_client.request_software_packages() \n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\") \n >>> name = pt.supported_host_names()[0]\n >>> pt.supported_plugins(name)\n [\n {\n \"plugin\": \"arnold-cinema4d\",\n \"versions\": [\n \"3.3.2.100\",\n \"3.3.3.0\"\n ]\n },\n {\n \"plugin\": \"redshift-cinema4d\",\n \"versions\": [\n \"2.6.54\",\n \"2.6.56\",\n \"3.0.21\",\n \"3.0.22\",\n ],\n },\n ]\n \"\"\"\n\n try:\n subtree = self.find_by_name(host)\n plugin_versions = _to_path_list(subtree)\n except TypeError:\n return []\n\n if not plugin_versions:\n return []\n\n plugin_dict = {}\n for plugin, version, _ in [pv.split(\" \") for pv in plugin_versions]:\n if plugin not in plugin_dict:\n plugin_dict[plugin] = []\n plugin_dict[plugin].append(version)\n\n # convert to list so it can be sorted\n plugins = []\n for key in plugin_dict:\n plugins.append({\"plugin\": key, \"versions\": sorted(plugin_dict[key])})\n\n return sorted(plugins, key=lambda k: k[\"plugin\"])\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.find_by_name","title":"<code>find_by_name(self, name, limit=None)</code>","text":"<p>Search the tree for a product with the given name.</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>The name constructed from the package using to_name(). It must be an exact match with product, version, and platform. For example: <code>maya 2018.0 windows</code></p> required <p>Keyword arguments:</p> Name Type Description <code>limit</code> <code>int</code> <p>Limit the search depth. Defaults to <code>None</code>.</p> <p>Returns:</p> Type Description <code>object</code> <p>The package that matches.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> pt.find_by_name(\"redshift-cinema4d 3.0.64 linux\")\n{\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '64',\n 'vendor': 'maxon',\n 'children': [],\n ...\n}\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def find_by_name(self, name, limit=None):\n\"\"\"\n Search the tree for a product with the given name.\n\n Args:\n name (str): The name constructed from the package using to_name(). It must be an exact match with product, version, and platform. For example: `maya 2018.0 windows`\n\n Keyword Args:\n limit (int): Limit the search depth. Defaults to `None`.\n\n Returns:\n object: The package that matches.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.find_by_name(\"redshift-cinema4d 3.0.64 linux\")\n {\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '64',\n 'vendor': 'maxon',\n 'children': [],\n ...\n }\n \"\"\"\n\n return _find_by_name(self._tree, name, limit, 0)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.find_by_path","title":"<code>find_by_path(self, path)</code>","text":"<p>Find the package uniquely described by the given path.</p> <p>The path is of the form returned by the to_path_list() method.</p> <p>Parameters:</p> Name Type Description Default <code>path</code> <code>str</code> <p>The path</p> required <p>Returns:</p> Type Description <code>object</code> <p>The package or None if no package exists with the given path.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree, package_environment\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> pt.find_by_path(\"cinema4d 24.111 linux/redshift-cinema4d 3.0.62 linux\")\n{\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '62',\n 'vendor': 'maxon',\n 'children': [],\n 'plugin_host_version': \"24\",\n ...\n}\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def find_by_path(self, path):\n\"\"\"\n Find the package uniquely described by the given path.\n\n The path is of the form returned by the to_path_list() method.\n\n Args:\n path (str): The path\n\n Returns:\n object: The package or None if no package exists with the given path.\n\n Example:\n >>> from ciocore import api_client, package_tree, package_environment\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.find_by_path(\"cinema4d 24.111 linux/redshift-cinema4d 3.0.62 linux\")\n {\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '62',\n 'vendor': 'maxon',\n 'children': [],\n 'plugin_host_version': \"24\",\n ...\n }\n \"\"\"\n return _find_by_path(self._tree, path)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.to_path_list","title":"<code>to_path_list(self, name=None)</code>","text":"<p>Get paths to all nodes.</p> <p>This is useful for populating a chooser to choose packages fully qualified by path. Houdini's tree widget, for example, takes the below format unchanged and generates the appropriate UI.</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>Get paths below the tree represented by the name. Defaults to None (root node).</p> <code>None</code> <p>Returns:</p> Type Description <code>list(str)</code> <p>Paths to all nodes in the tree.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> pt.to_path_list()\ncinema4d 22.118.RB320081 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.22 linux\ncinema4d 22.118.RB320081 linux/arnold-cinema4d 3.3.2.100 linux\n...\n</code></pre> <pre><code>>>> pt.to_path_list(name=\"cinema4d 24.111 linux\")\nredshift-cinema4d 3.0.57 linux\nredshift-cinema4d 3.0.62 linux\nredshift-cinema4d 3.0.45 linux\nredshift-cinema4d 3.0.64 linux\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def to_path_list(self, name=None):\n\"\"\"\n Get paths to all nodes.\n\n This is useful for populating a chooser to choose packages fully qualified by path.\n Houdini's tree widget, for example, takes the below format unchanged and generates the\n appropriate UI.\n\n Args:\n name (str): Get paths below the tree represented by the name. Defaults to None (root node).\n\n Returns:\n list(str): Paths to all nodes in the tree.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.to_path_list()\n cinema4d 22.118.RB320081 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.22 linux\n cinema4d 22.118.RB320081 linux/arnold-cinema4d 3.3.2.100 linux\n ...\n\n >>> pt.to_path_list(name=\"cinema4d 24.111 linux\")\n redshift-cinema4d 3.0.57 linux\n redshift-cinema4d 3.0.62 linux\n redshift-cinema4d 3.0.45 linux\n redshift-cinema4d 3.0.64 linux\n \"\"\"\n if name:\n subtree = self.find_by_name(name)\n return _to_path_list(subtree)\n return _to_path_list(self._tree)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.platforms","title":"<code>platforms(self)</code>","text":"<p>Get the platforms represented by packages in the tree.</p> <p>Returns:</p> Type Description <code>set</code> <p>The set of platforms.</p> Source code in <code>ciocore/package_tree.py</code> <pre><code>def platforms(self):\n\"\"\"\n Get the platforms represented by packages in the tree.\n\n Returns:\n set: The set of platforms.\n \"\"\"\n\n # No need to recurse. Plugins are assumed to be compatible with the host.\n return set([host[\"platform\"] for host in self._tree[\"children\"]])\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.json","title":"<code>json(self)</code>","text":"<p>The whole tree of softwware as json.</p> <p>Returns:</p> Type Description <code>str</code> <p>JSON.</p> Source code in <code>ciocore/package_tree.py</code> <pre><code>def json(self):\n\"\"\"\n The whole tree of softwware as json.\n\n Returns:\n str: JSON.\n\n \"\"\"\n return json.dumps(self._tree)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.as_dict","title":"<code>as_dict(self)</code>","text":"<p>Returns:</p> Type Description <code>dict</code> <p>The underlying software dictionary.</p> Source code in <code>ciocore/package_tree.py</code> <pre><code>def as_dict(self):\n\"\"\"\n Returns:\n dict: The underlying software dictionary.\n\n \"\"\"\n return self._tree\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.to_name","title":"<code>to_name(pkg)</code>","text":"<p>Generate a name like <code>houdini 16.5.323 linux</code> or <code>maya 2016.SP3 linux</code>.</p> <p>This name is derived from the product and version fields in a package. Note: It is not necessarily possible to go the other way and extract version fields from the name.</p> <p>Parameters:</p> Name Type Description Default <code>pkg</code> <code>object</code> <p>An object with product, platform, and all version fields.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The package name.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n>>> packages = api_client.request_software_packages()\n>>> package_tree.to_name(packages[0])\nredshift-maya 3.0.64 linux\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def to_name(pkg):\n\"\"\"\n Generate a name like `houdini 16.5.323 linux` or `maya 2016.SP3 linux`.\n\n This name is derived from the product and version fields in a package. Note: It is not\n necessarily possible to go the other way and extract version fields from the name.\n\n Args:\n pkg (object): An object with product, platform, and all version fields.\n\n Returns:\n str: The package name.\n\n Examples:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> package_tree.to_name(packages[0])\n redshift-maya 3.0.64 linux\n\n \"\"\" \n\n version_parts = [\n pkg[\"major_version\"],\n pkg[\"minor_version\"],\n pkg[\"release_version\"],\n pkg[\"build_version\"],\n ]\n version_string = (\".\").join([p for p in version_parts if p])\n if pkg[\"platform\"] not in PLATFORMS:\n raise KeyError(\"Invalid platform: {}\".format(pkg[\"platform\"]))\n return \" \".join(filter(None, [pkg[\"product\"], version_string, pkg[\"platform\"]]))\n</code></pre>"},{"location":"cmdline/docs/","title":"Docs","text":"<p>Open the Conductor Core documentation in a web browser.</p> <p>To use this tool, you need to run the following command:</p> <pre><code>conductor docs [OPTIONS]\n</code></pre>"},{"location":"cmdline/docs/#options","title":"Options","text":"<p>The <code>docs</code> command provides an option to choose the port in the event of a conflict.</p> Option Type Default -p, --port INTEGER 8025 <p>The port on which to serve the documentation.</p> --help <p>Show this message and exit.</p>"},{"location":"cmdline/downloader/","title":"Downloader","text":"<p>The Conductor Download tool downloads renders and other output files from Conductor. You can give a list of job ids to download, or you can omit jobids and the downloader will run in daemon mode.</p> <p>If you provide jobids, the default behavior is to download all the files from completed tasks for those jobs. You can however specify an explicit set of tasks to download by providing a task range spec after each job id. To do so, append a colon to the job id and then a compact task specification. See the examples.</p> <p>In daemon mode, the downloader polls for new jobs to download. Examples:</p> <ul> <li><code>conductor download</code></li> <li><code>conductor download 1234 1235</code></li> <li><code>conductor download 1234:1-10</code></li> <li><code>conductor download 1234:1-5x2,10,12-14</code></li> <li><code>conductor download 1234:1-5 1235:5-10</code></li> </ul>"},{"location":"cmdline/downloader/#options","title":"Options:","text":"Option Type Default -d, --destination TEXT <p>Override the output directory</p> -lv, --log_level ENUM INFO <p>The logging level to display. Options are: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET.</p> -ld, --log_dir TEXT <p>Write a log file to the given directory. The log rotates, creating a new log file every day, while storing logs for the last 7 days.</p> <p>The log file is named conductor_ul.log</p> -tc, --thread_count INTEGER <p>The number of threads that should download simultaneously.</p> <p>Increasing the thread count may improve upload performance, but it can also consume more system resources.</p> -lc, --location TEXT <p>Specify a location tag to associate with uploads, downloads, and submissions.</p> <p>A location tag allows you to limit the scope of your uploads and downloads to jobs sharing the same location tag. This is useful while using the uploader or downloader in daemon mode.</p> --help <p>Show this message and exit.</p>"},{"location":"cmdline/packages/","title":"Packages","text":"<p>List the software packages available on the render nodes in the cloud.</p> <p>To use this tool, you need to run the following command:</p> <pre><code>conductor packages [OPTIONS]\n</code></pre>"},{"location":"cmdline/packages/#options","title":"Options","text":"<p>The <code>packages</code> command provides one option to format its output.</p> Option Type Default -f, --fmt, --format TEXT <ul> <li>text: The output is a simple list of software names and versions, with nesting to indicate plugin compatibility. Output is sent to stdout.</li> <li>markdown: Designed for use in other markdown documentation systems where it benefits from consistent styling. Output is sent to stdout and can be piped to a file.</li> <li>html: Opens a browser window and displays the output in an HTML page.</li> </ul> <p>Valid values are: [text\|markdown\|html]</p> --help <p>Show this message and exit.</p> <p>Feel free to explore all the software packages available on the render nodes with the <code>conductor packages</code> command. You have the flexibility to choose the format of the output as per your preference. Whether you want a simple text list, styled markdown for documentation, or an interactive HTML page, we've got you covered!</p>"},{"location":"cmdline/uploader/","title":"Uploader","text":"<p>The Conductor Upload tool allows you to upload files to Conductor. It provides multiple options for customization and flexibility.</p>"},{"location":"cmdline/uploader/#usage","title":"Usage","text":"<p>To use the Conductor Upload tool, you need to run the following command:</p> <pre><code>conductor upload [OPTIONS] [PATHS]...\n</code></pre> <p>By default, without any arguments, the uploader runs in daemon mode. In this mode, it continuously watches for files to upload for submitted jobs. If you want to manually specify a list of paths to upload, you can provide them as arguments after the command.</p> <p>For example, to upload multiple files named file1, file2, and file3, you can run:</p> <pre><code>conductor upload file1 file2 file3\n</code></pre>"},{"location":"cmdline/uploader/#options","title":"Options","text":"<p>The Conductor Upload tool provides several options to customize its behavior. Here are the available options:</p> Option Type Default -db, --database_filepath TEXT <p>This option allows you to specify a filepath to the local md5 caching database. The md5 caching feature improves uploading times by skipping md5 generation for previously uploaded files that haven't been modified since the last upload.</p> -md5, --md5_caching BOOLEAN <p>When set to true (default), the tool uses cached md5s to skip md5 checking for subsequent uploads of the same files. This dramatically improves uploading times. However, if there is concern that files may not be getting re-uploaded properly, you can set this flag to false.</p> -lv, --log_level ENUM <p>This option allows you to set the logging level to display. You can choose from the following levels: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET. By default, the log level is set to INFO.</p> <p>Options are: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET</p> -ld, --log_dir TEXT <p>If you want to write a log file, you can specify the directory using this option. The log file rotates, creating a new log file every day and storing logs for the last 7 days. The log file is named conductor_ul.log.</p> -tc, --thread_count INTEGER <p>This option sets the number of threads that should download simultaneously. Increasing the thread count may improve upload performance, but it can also consume more system resources.</p> -lc, --location TEXT <p>If you want to associate uploads, downloads, and submissions with a specific location tag, you can use this option. A location tag allows you to limit the scope of your uploads and downloads to jobs sharing the same location tag. This is particularly useful when using the uploader or downloader in daemon mode. </p> --help <p>Print help. </p>"},{"location":"cmdline/uploader/#conclusion","title":"Conclusion","text":"<p>The Conductor Upload tool provides a convenient way to upload files to Conductor. With its options and flexibility you can customize its behavior to suit your needs.</p>"}]}
1	+ {"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Overview","text":"<p>This site contains the documentation for the Conductor Core Python package <code>ciocore</code>. </p> <p>The package provides two main interfaces for interacting with Conductor.</p> <ol> <li>A command-line interface.</li> <li>A Python API.</li> </ol>"},{"location":"#installation","title":"Installation","text":"<p>Since you are reading this, you already have Conductor Core installed. If not, you'll find it is available through the Companion app or directly from PyPi.</p>"},{"location":"#command-line-interface","title":"Command-line interface","text":"<p>The command-line interface is the easiest way to interact with Conductor Core. It provides a simple way to upload and download data, query available software packages, and view this documentation. Below are quick liks to the most commonly used commands.</p> <ul> <li>Uploader</li> <li>Downloader</li> <li>Supported Software</li> <li>Documentation</li> </ul>"},{"location":"#python-api","title":"Python API","text":"<p>The Python API is a more flexible way to interact with Conductor. It can be used to build custom submission tools, and to integrate Conductor into existing workflows. </p> <ul> <li>The API is documented here.</li> </ul> <p>Note</p> <p>This documentation does not cover the wider Conductor ecosystem, which consists of the web dashboard, the Companion app, the back-end render services, and the DCC submission plugins. For information on these components, and to get a conceptual overview of Conductor, please see the Conductor documentation online.</p>"},{"location":"how-to-guides/","title":"How-To Guides","text":"<p>This part of the documentation includes recipes and tutorials for common tasks and workflows using Conductor Core. If you have a suggestion for a new guide, feel free to open an issue</p>"},{"location":"how-to-guides/#how-to-create-a-submission","title":"How to create a submission?","text":"<pre><code># your_script.py\nfrom ciocore import data as coredata\n\n...\n</code></pre>"},{"location":"how-to-guides/#how-to-authenticate-with-an-api-key","title":"How to authenticate with an API key?","text":"<pre><code># your_script.py\nfrom ciocore import api_client\n\n...\n</code></pre>"},{"location":"apidoc/api_client/","title":"ciocore.api_client","text":"<p>The api_client module is used to make requests to the Conductor API.</p>"},{"location":"apidoc/api_client/#ciocore.api_client.ApiClient","title":"<code> ApiClient </code>","text":"<p>The ApiClient class is a wrapper around the requests library that handles authentication and retries.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>class ApiClient:\n\"\"\"\n The ApiClient class is a wrapper around the requests library that handles authentication and retries.\n \"\"\"\n\n http_verbs = [\"PUT\", \"POST\", \"GET\", \"DELETE\", \"HEAD\", \"PATCH\"]\n\n USER_AGENT_TEMPLATE = \"client {client_name}/{client_version} (ciocore {ciocore_version}; {runtime} {runtime_version}; {platform} {platform_details}; {hostname} {pid}; {python_path})\"\n USER_AGENT_MAX_PATH_LENGTH = 1024\n\n user_agent_header = None\n\n def __init__(self):\n logger.debug(\"\")\n\n def _make_request(self, verb, conductor_url, headers, params, data, raise_on_error=True):\n response = requests.request(\n method=verb, url=conductor_url, headers=headers, params=params, data=data\n )\n\n logger.debug(f\"verb: {verb}\")\n logger.debug(f\"conductor_url: {conductor_url}\")\n logger.debug(f\"headers: {headers}\")\n logger.debug(f\"params: {params}\")\n logger.debug(f\"data: {data}\")\n\n # If we get 300s/400s debug out the response. TODO(lws): REMOVE THIS\n if 300 <= response.status_code < 500:\n logger.debug(\"*** ERROR!! **\")\n logger.debug(f\"Reason: {response.reason}\")\n logger.debug(f\"Text: {response.text}\")\n\n # trigger an exception to be raised for 4XX or 5XX http responses\n if raise_on_error:\n response.raise_for_status()\n\n return response\n\n def make_prepared_request(\n self,\n verb,\n url,\n headers=None,\n params=None,\n json_payload=None,\n data=None,\n stream=False,\n remove_headers_list=None,\n raise_on_error=True,\n tries=5,\n ):\n\"\"\"\n Make a request to the Conductor API.\n\n Deprecated:\n Primarily used to removed enforced headers by requests.Request. Requests 2.x will add\n Transfer-Encoding: chunked with file like object that is 0 bytes, causing s3 failures (501)\n - https://github.com/psf/requests/issues/4215#issuecomment-319521235\n\n To get around this bug make_prepared_request has functionality to remove the enforced header\n that would occur when using requests.request(...). Requests 3.x resolves this issue, when\n client is built to use Requests 3.x this function can be removed.\n\n Returns:\n requests.Response: The response object.\n\n Args:\n verb (str): The HTTP verb to use.\n url (str): The URL to make the request to.\n headers (dict): A dictionary of headers to send with the request.\n params (dict): A dictionary of query parameters to send with the request.\n json (dict): A JSON payload to send with the request.\n stream (bool): Whether or not to stream the response.\n remove_headers_list (list): A list of headers to remove from the request.\n raise_on_error (bool): Whether or not to raise an exception if the request fails.\n tries (int): The number of times to retry the request.\n \"\"\"\n\n req = requests.Request(\n method=verb,\n url=url,\n headers=headers,\n params=params,\n json=json_payload,\n data=data,\n )\n prepped = req.prepare()\n\n if remove_headers_list:\n for header in remove_headers_list:\n prepped.headers.pop(header, None)\n\n # Create a retry wrapper function\n retry_wrapper = common.DecRetry(\n retry_exceptions=CONNECTION_EXCEPTIONS, tries=tries\n )\n\n # requests sessions potentially not thread-safe, but need to removed enforced\n # headers by using a prepared request.create which can only be done through an\n # request.Session object. Create Session object per call of make_prepared_request, it will\n # not benefit from connection pooling reuse. https://github.com/psf/requests/issues/1871\n session = requests.Session()\n\n # wrap the request function with the retry wrapper\n wrapped_func = retry_wrapper(session.send)\n\n # call the wrapped request function\n response = wrapped_func(prepped, stream=stream)\n\n logger.debug(\"verb: %s\", prepped.method)\n logger.debug(\"url: %s\", prepped.url)\n logger.debug(\"headers: %s\", prepped.headers)\n logger.debug(\"params: %s\", req.params)\n logger.debug(\"response: %s\", response)\n\n # trigger an exception to be raised for 4XX or 5XX http responses\n if raise_on_error:\n response.raise_for_status()\n\n return response\n\n def make_request(\n self,\n uri_path=\"/\",\n headers=None,\n params=None,\n data=None,\n verb=None,\n conductor_url=None,\n raise_on_error=True,\n tries=5,\n use_api_key=False,\n ):\n\"\"\"\n Make a request to the Conductor API.\n\n Args:\n uri_path (str): The path to the resource to request.\n headers (dict): A dictionary of headers to send with the request.\n params (dict): A dictionary of query parameters to send with the request.\n data (dict): A dictionary of data to send with the request.\n verb (str): The HTTP verb to use.\n conductor_url (str): The Conductor URL.\n raise_on_error (bool): Whether or not to raise an exception if the request fails.\n tries (int): The number of times to retry the request.\n use`_api_key (bool): Whether or not to use the API key for authentication.\n\n Returns:\n tuple(str, int): The response text and status code.\n \"\"\"\n cfg = config.get()\n # TODO: set Content Content-Type to json if data arg\n if not headers:\n headers = {\"Content-Type\": \"application/json\", \"Accept\": \"application/json\"}\n\n logger.debug(\"read_conductor_credentials({})\".format(use_api_key))\n bearer_token = read_conductor_credentials(use_api_key)\n if not bearer_token:\n raise Exception(\"Error: Could not get conductor credentials!\")\n\n headers[\"Authorization\"] = \"Bearer %s\" % bearer_token\n\n if not ApiClient.user_agent_header:\n self.register_client(\"ciocore\")\n\n headers[\"User-Agent\"] = ApiClient.user_agent_header\n\n # Construct URL\n if not conductor_url:\n conductor_url = parse.urljoin(cfg[\"url\"], uri_path)\n\n if not verb:\n if data:\n verb = \"POST\"\n else:\n verb = \"GET\"\n\n assert verb in self.http_verbs, \"Invalid http verb: %s\" % verb\n\n # Create a retry wrapper function\n retry_wrapper = common.DecRetry(\n retry_exceptions=CONNECTION_EXCEPTIONS, tries=tries\n )\n\n # wrap the request function with the retry wrapper\n wrapped_func = retry_wrapper(self._make_request)\n\n # call the wrapped request function\n response = wrapped_func(\n verb, conductor_url, headers, params, data, raise_on_error=raise_on_error\n )\n\n return response.text, response.status_code\n\n @classmethod\n def register_client(cls, client_name, client_version=None):\n\"\"\"\n Generates the http User Agent header that includes helpful debug info.\n \"\"\"\n\n # Use the provided client_version.\n if not client_version:\n client_version = 'unknown'\n\n python_version = platform.python_version()\n system_info = platform.system()\n release_info = platform.release()\n\n\n # Get the MD5 hex digest of the path to the python executable\n python_executable_path = truncate_middle(sys.executable.encode('utf-8'), cls.USER_AGENT_MAX_PATH_LENGTH)\n md5_hash = hashlib.md5(python_executable_path).hexdigest()\n\n user_agent = (\n f\"{client_name}/{client_version} \"\n f\"(python {python_version}; {system_info} {release_info}; {md5_hash})\"\n )\n cls.user_agent_header = user_agent\n\n return user_agent\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.ApiClient.make_prepared_request","title":"<code>make_prepared_request(self, verb, url, headers=None, params=None, json_payload=None, data=None, stream=False, remove_headers_list=None, raise_on_error=True, tries=5)</code>","text":"<p>Make a request to the Conductor API.</p> <p>Deprecated</p> <p>Primarily used to removed enforced headers by requests.Request. Requests 2.x will add Transfer-Encoding: chunked with file like object that is 0 bytes, causing s3 failures (501) - https://github.com/psf/requests/issues/4215#issuecomment-319521235</p> <p>To get around this bug make_prepared_request has functionality to remove the enforced header that would occur when using requests.request(...). Requests 3.x resolves this issue, when client is built to use Requests 3.x this function can be removed.</p> <p>Returns:</p> Type Description <code>requests.Response</code> <p>The response object.</p> <p>Parameters:</p> Name Type Description Default <code>verb</code> <code>str</code> <p>The HTTP verb to use.</p> required <code>url</code> <code>str</code> <p>The URL to make the request to.</p> required <code>headers</code> <code>dict</code> <p>A dictionary of headers to send with the request.</p> <code>None</code> <code>params</code> <code>dict</code> <p>A dictionary of query parameters to send with the request.</p> <code>None</code> <code>json</code> <code>dict</code> <p>A JSON payload to send with the request.</p> required <code>stream</code> <code>bool</code> <p>Whether or not to stream the response.</p> <code>False</code> <code>remove_headers_list</code> <code>list</code> <p>A list of headers to remove from the request.</p> <code>None</code> <code>raise_on_error</code> <code>bool</code> <p>Whether or not to raise an exception if the request fails.</p> <code>True</code> <code>tries</code> <code>int</code> <p>The number of times to retry the request.</p> <code>5</code> Source code in <code>ciocore/api_client.py</code> <pre><code>def make_prepared_request(\n self,\n verb,\n url,\n headers=None,\n params=None,\n json_payload=None,\n data=None,\n stream=False,\n remove_headers_list=None,\n raise_on_error=True,\n tries=5,\n):\n\"\"\"\n Make a request to the Conductor API.\n\n Deprecated:\n Primarily used to removed enforced headers by requests.Request. Requests 2.x will add\n Transfer-Encoding: chunked with file like object that is 0 bytes, causing s3 failures (501)\n - https://github.com/psf/requests/issues/4215#issuecomment-319521235\n\n To get around this bug make_prepared_request has functionality to remove the enforced header\n that would occur when using requests.request(...). Requests 3.x resolves this issue, when\n client is built to use Requests 3.x this function can be removed.\n\n Returns:\n requests.Response: The response object.\n\n Args:\n verb (str): The HTTP verb to use.\n url (str): The URL to make the request to.\n headers (dict): A dictionary of headers to send with the request.\n params (dict): A dictionary of query parameters to send with the request.\n json (dict): A JSON payload to send with the request.\n stream (bool): Whether or not to stream the response.\n remove_headers_list (list): A list of headers to remove from the request.\n raise_on_error (bool): Whether or not to raise an exception if the request fails.\n tries (int): The number of times to retry the request.\n \"\"\"\n\n req = requests.Request(\n method=verb,\n url=url,\n headers=headers,\n params=params,\n json=json_payload,\n data=data,\n )\n prepped = req.prepare()\n\n if remove_headers_list:\n for header in remove_headers_list:\n prepped.headers.pop(header, None)\n\n # Create a retry wrapper function\n retry_wrapper = common.DecRetry(\n retry_exceptions=CONNECTION_EXCEPTIONS, tries=tries\n )\n\n # requests sessions potentially not thread-safe, but need to removed enforced\n # headers by using a prepared request.create which can only be done through an\n # request.Session object. Create Session object per call of make_prepared_request, it will\n # not benefit from connection pooling reuse. https://github.com/psf/requests/issues/1871\n session = requests.Session()\n\n # wrap the request function with the retry wrapper\n wrapped_func = retry_wrapper(session.send)\n\n # call the wrapped request function\n response = wrapped_func(prepped, stream=stream)\n\n logger.debug(\"verb: %s\", prepped.method)\n logger.debug(\"url: %s\", prepped.url)\n logger.debug(\"headers: %s\", prepped.headers)\n logger.debug(\"params: %s\", req.params)\n logger.debug(\"response: %s\", response)\n\n # trigger an exception to be raised for 4XX or 5XX http responses\n if raise_on_error:\n response.raise_for_status()\n\n return response\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.ApiClient.make_request","title":"<code>make_request(self, uri_path='/', headers=None, params=None, data=None, verb=None, conductor_url=None, raise_on_error=True, tries=5, use_api_key=False)</code>","text":"<p>Make a request to the Conductor API.</p> <p>Parameters:</p> Name Type Description Default <code>uri_path</code> <code>str</code> <p>The path to the resource to request.</p> <code>'/'</code> <code>headers</code> <code>dict</code> <p>A dictionary of headers to send with the request.</p> <code>None</code> <code>params</code> <code>dict</code> <p>A dictionary of query parameters to send with the request.</p> <code>None</code> <code>data</code> <code>dict</code> <p>A dictionary of data to send with the request.</p> <code>None</code> <code>verb</code> <code>str</code> <p>The HTTP verb to use.</p> <code>None</code> <code>conductor_url</code> <code>str</code> <p>The Conductor URL.</p> <code>None</code> <code>raise_on_error</code> <code>bool</code> <p>Whether or not to raise an exception if the request fails.</p> <code>True</code> <code>tries</code> <code>int</code> <p>The number of times to retry the request.</p> <code>5</code> <code>use`_api_key</code> <code>bool</code> <p>Whether or not to use the API key for authentication.</p> required <p>Returns:</p> Type Description <code>tuple(str, int)</code> <p>The response text and status code.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def make_request(\n self,\n uri_path=\"/\",\n headers=None,\n params=None,\n data=None,\n verb=None,\n conductor_url=None,\n raise_on_error=True,\n tries=5,\n use_api_key=False,\n):\n\"\"\"\n Make a request to the Conductor API.\n\n Args:\n uri_path (str): The path to the resource to request.\n headers (dict): A dictionary of headers to send with the request.\n params (dict): A dictionary of query parameters to send with the request.\n data (dict): A dictionary of data to send with the request.\n verb (str): The HTTP verb to use.\n conductor_url (str): The Conductor URL.\n raise_on_error (bool): Whether or not to raise an exception if the request fails.\n tries (int): The number of times to retry the request.\n use`_api_key (bool): Whether or not to use the API key for authentication.\n\n Returns:\n tuple(str, int): The response text and status code.\n \"\"\"\n cfg = config.get()\n # TODO: set Content Content-Type to json if data arg\n if not headers:\n headers = {\"Content-Type\": \"application/json\", \"Accept\": \"application/json\"}\n\n logger.debug(\"read_conductor_credentials({})\".format(use_api_key))\n bearer_token = read_conductor_credentials(use_api_key)\n if not bearer_token:\n raise Exception(\"Error: Could not get conductor credentials!\")\n\n headers[\"Authorization\"] = \"Bearer %s\" % bearer_token\n\n if not ApiClient.user_agent_header:\n self.register_client(\"ciocore\")\n\n headers[\"User-Agent\"] = ApiClient.user_agent_header\n\n # Construct URL\n if not conductor_url:\n conductor_url = parse.urljoin(cfg[\"url\"], uri_path)\n\n if not verb:\n if data:\n verb = \"POST\"\n else:\n verb = \"GET\"\n\n assert verb in self.http_verbs, \"Invalid http verb: %s\" % verb\n\n # Create a retry wrapper function\n retry_wrapper = common.DecRetry(\n retry_exceptions=CONNECTION_EXCEPTIONS, tries=tries\n )\n\n # wrap the request function with the retry wrapper\n wrapped_func = retry_wrapper(self._make_request)\n\n # call the wrapped request function\n response = wrapped_func(\n verb, conductor_url, headers, params, data, raise_on_error=raise_on_error\n )\n\n return response.text, response.status_code\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.ApiClient.register_client","title":"<code>register_client(client_name, client_version=None)</code> <code>classmethod</code>","text":"<p>Generates the http User Agent header that includes helpful debug info.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>@classmethod\ndef register_client(cls, client_name, client_version=None):\n\"\"\"\n Generates the http User Agent header that includes helpful debug info.\n \"\"\"\n\n # Use the provided client_version.\n if not client_version:\n client_version = 'unknown'\n\n python_version = platform.python_version()\n system_info = platform.system()\n release_info = platform.release()\n\n\n # Get the MD5 hex digest of the path to the python executable\n python_executable_path = truncate_middle(sys.executable.encode('utf-8'), cls.USER_AGENT_MAX_PATH_LENGTH)\n md5_hash = hashlib.md5(python_executable_path).hexdigest()\n\n user_agent = (\n f\"{client_name}/{client_version} \"\n f\"(python {python_version}; {system_info} {release_info}; {md5_hash})\"\n )\n cls.user_agent_header = user_agent\n\n return user_agent\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.truncate_middle","title":"<code>truncate_middle(s, max_length)</code>","text":"<p>Truncate the string <code>s</code> to <code>max_length</code> by removing characters from the middle.</p> <p>:param s: The original string to be truncated. :type s: str :param max_length: The maximum allowed length of the string after truncation. :type max_length: int :return: The truncated string. :rtype: str</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def truncate_middle(s, max_length):\n\"\"\"\n Truncate the string `s` to `max_length` by removing characters from the middle.\n\n :param s: The original string to be truncated.\n :type s: str\n :param max_length: The maximum allowed length of the string after truncation.\n :type max_length: int\n :return: The truncated string.\n :rtype: str\n \"\"\"\n\n if len(s) <= max_length:\n # String is already at or below the maximum length, return it as is\n return s\n\n # Calculate the number of characters to keep from the start and end of the string\n num_keep_front = (max_length // 2)\n num_keep_end = max_length - num_keep_front - 1 # -1 for the ellipsis\n\n # Construct the truncated string\n return s[:num_keep_front] + '~' + s[-num_keep_end:]\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.read_conductor_credentials","title":"<code>read_conductor_credentials(use_api_key=False)</code>","text":"<p>Read the conductor credentials file.</p> <p>If the credentials file exists, it will contain a bearer token from either the user or the API key.</p> <p>If the credentials file doesn't exist, or is expired, or is from a different domain, we try to fetch a new one in the API key scenario or prompt the user to log in. </p> <p>Args: use_api_key (bool): Whether or not to try to use the API key</p> <p>Returns: A Bearer token in the event of a success or None</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def read_conductor_credentials(use_api_key=False):\n\"\"\"\n Read the conductor credentials file.\n\n If the credentials file exists, it will contain a bearer token from either\n the user or the API key.\n\n If the credentials file doesn't exist, or is\n expired, or is from a different domain, we try to fetch a new one in the API key scenario or\n prompt the user to log in. \n\n Args: \n use_api_key (bool): Whether or not to try to use the API key\n\n Returns: \n A Bearer token in the event of a success or None\n\n \"\"\"\n\n cfg = config.get()\n\n logger.debug(\"Reading conductor credentials...\")\n if use_api_key:\n if not cfg.get(\"api_key\"):\n use_api_key = False\n if use_api_key and not cfg[\"api_key\"].get(\"client_id\"):\n use_api_key = False\n logger.debug(\"use_api_key = %s\" % use_api_key)\n creds_file = get_creds_path(use_api_key)\n\n logger.debug(\"Creds file is %s\" % creds_file)\n logger.debug(\"Auth url is %s\" % cfg[\"url\"])\n if not os.path.exists(creds_file):\n if use_api_key:\n if not cfg[\"api_key\"]:\n logger.debug(\"Attempted to use API key, but no api key in in config!\")\n return None\n\n # Exchange the API key for a bearer token\n logger.debug(\"Attempting to get API key bearer token\")\n get_api_key_bearer_token(creds_file)\n\n else:\n auth.run(creds_file, cfg[\"url\"])\n if not os.path.exists(creds_file):\n return None\n\n logger.debug(\"Reading credentials file...\")\n with open(creds_file, \"r\", encoding=\"utf-8\") as fp:\n file_contents = json.loads(fp.read())\n expiration = file_contents.get(\"expiration\")\n same_domain = creds_same_domain(file_contents)\n if same_domain and expiration and expiration >= int(time.time()):\n return file_contents[\"access_token\"]\n logger.debug(\"Credentials have expired or are from a different domain\")\n if use_api_key:\n logger.debug(\"Refreshing API key bearer token!\")\n get_api_key_bearer_token(creds_file)\n else:\n logger.debug(\"Sending to auth page...\")\n auth.run(creds_file, cfg[\"url\"])\n # Re-read the creds file, since it has been re-upped\n with open(creds_file, \"r\", encoding=\"utf-8\") as fp:\n file_contents = json.loads(fp.read())\n return file_contents[\"access_token\"]\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_api_key_bearer_token","title":"<code>get_api_key_bearer_token(creds_file=None)</code>","text":"<p>Get a bearer token from the API key.</p> <p>Parameters:</p> Name Type Description Default <code>creds_file</code> <code>str</code> <p>The path to the credentials file. If not provided, the bearer token will not be written to disk.</p> <code>None</code> <p>Returns:</p> Type Description <p>A dictionary containing the bearer token and other information.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_api_key_bearer_token(creds_file=None):\n\"\"\"\n Get a bearer token from the API key.\n\n Args:\n creds_file (str): The path to the credentials file. If not provided, the bearer token will not be written to disk.\n\n Returns:\n A dictionary containing the bearer token and other information.\n \"\"\"\n cfg = config.get()\n url = \"{}/api/oauth_jwt\".format(cfg[\"url\"])\n response = requests.get(\n url,\n params={\n \"grant_type\": \"client_credentials\",\n \"scope\": \"owner admin user\",\n \"client_id\": cfg[\"api_key\"][\"client_id\"],\n \"client_secret\": cfg[\"api_key\"][\"private_key\"],\n },\n )\n if response.status_code == 200:\n response_dict = json.loads(response.text)\n credentials_dict = {\n \"access_token\": response_dict[\"access_token\"],\n \"token_type\": \"Bearer\",\n \"expiration\": int(time.time()) + int(response_dict[\"expires_in\"]),\n \"scope\": \"user admin owner\",\n }\n\n if not creds_file:\n return credentials_dict\n\n if not os.path.exists(os.path.dirname(creds_file)):\n os.makedirs(os.path.dirname(creds_file))\n\n with open(creds_file, \"w\") as fp:\n fp.write(json.dumps(credentials_dict))\n return\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_creds_path","title":"<code>get_creds_path(api_key=False)</code>","text":"<p>Get the path to the credentials file.</p> <p>Parameters:</p> Name Type Description Default <code>api_key</code> <code>bool</code> <p>Whether or not to use the API key.</p> <code>False</code> <p>Returns:</p> Type Description <code>str</code> <p>The path to the credentials file.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_creds_path(api_key=False):\n\"\"\"\n Get the path to the credentials file.\n\n Args:\n api_key (bool): Whether or not to use the API key.\n\n Returns:\n str: The path to the credentials file.\n \"\"\"\n creds_dir = os.path.join(os.path.expanduser(\"~\"), \".config\", \"conductor\")\n if api_key:\n creds_file = os.path.join(creds_dir, \"api_key_credentials\")\n else:\n creds_file = os.path.join(creds_dir, \"credentials\")\n return creds_file\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_bearer_token","title":"<code>get_bearer_token(refresh=False)</code>","text":"<p>Return the bearer token.</p> <p>Parameters:</p> Name Type Description Default <code>refresh</code> <code>bool</code> <p>Whether or not to refresh the token.</p> <code>False</code> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_bearer_token(refresh=False):\n\"\"\"\n Return the bearer token.\n\n Args:\n refresh (bool): Whether or not to refresh the token.\n\n TODO: Thread safe multiproc caching, like it used to be pre-python3.7.\n \"\"\"\n return read_conductor_credentials(True)\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.creds_same_domain","title":"<code>creds_same_domain(creds)</code>","text":"<p>Check if the creds are for the same domain as the config.</p> <p>Parameters:</p> Name Type Description Default <code>creds</code> <code>dict</code> <p>The credentials dictionary.</p> required <p>Returns:</p> Type Description <code>bool</code> <p>Whether or not the creds are for the same domain as the config.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def creds_same_domain(creds):\n\"\"\"\n Check if the creds are for the same domain as the config.\n\n Args:\n creds (dict): The credentials dictionary.\n\n Returns:\n bool: Whether or not the creds are for the same domain as the config.\n \"\"\"\n cfg = config.get()\n\"\"\"Ensure the creds file refers to the domain in config\"\"\"\n token = creds.get(\"access_token\")\n if not token:\n return False\n\n decoded = jwt.decode(creds[\"access_token\"], verify=False, algorithms=['RS256'], options={\"verify_signature\": False})\n audience_domain = decoded.get(\"aud\")\n return (\n audience_domain\n and audience_domain.rpartition(\"/\")[-1] == cfg[\"api_url\"].rpartition(\"/\")[-1]\n )\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.account_id_from_jwt","title":"<code>account_id_from_jwt(token)</code>","text":"<p>Fetch the accounts id from a jwt token value.</p> <p>Parameters:</p> Name Type Description Default <code>token</code> <code>str</code> <p>The jwt token.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The account id.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def account_id_from_jwt(token):\n\"\"\"\n Fetch the accounts id from a jwt token value.\n\n Args:\n token (str): The jwt token.\n\n Returns:\n str: The account id.\n \"\"\"\n payload = jwt.decode(token, verify=False, algorithms=['RS256'], options={\"verify_signature\": False})\n return payload.get(\"account\")\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.account_name_from_jwt","title":"<code>account_name_from_jwt(token)</code>","text":"<p>Fetch the accounts name from a jwt token value.</p> <p>Parameters:</p> Name Type Description Default <code>token</code> <code>str</code> <p>The jwt token.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The account name.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def account_name_from_jwt(token):\n\"\"\"\n Fetch the accounts name from a jwt token value.\n\n Args:\n token (str): The jwt token.\n\n Returns:\n str: The account name.\n \"\"\"\n account_id = account_id_from_jwt(token)\n cfg = config.get()\n if account_id:\n url = \"%s/api/v1/accounts/%s\" % (cfg[\"api_url\"], account_id)\n response = requests.get(url, headers={\"authorization\": \"Bearer %s\" % token})\n if response.status_code == 200:\n response_dict = json.loads(response.text)\n return response_dict[\"data\"][\"name\"]\n return None\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.request_instance_types","title":"<code>request_instance_types(as_dict=False)</code>","text":"<p>Get the list of available instances types.</p> <p>Parameters:</p> Name Type Description Default <code>as_dict</code> <code>bool</code> <p>Whether or not to return the instance types as a dictionary.</p> <code>False</code> <p>Returns:</p> Type Description <code>list</code> <p>The list of instance types.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def request_instance_types(as_dict=False):\n\"\"\"\n Get the list of available instances types.\n\n Args:\n as_dict (bool): Whether or not to return the instance types as a dictionary.\n\n Returns:\n list: The list of instance types.\n \"\"\"\n api = ApiClient()\n response, response_code = api.make_request(\n \"api/v1/instance-types\", use_api_key=True, raise_on_error=False\n )\n if response_code not in (200,):\n msg = \"Failed to get instance types\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n instance_types = json.loads(response).get(\"data\", [])\n logger.debug(\"Found available instance types: %s\", instance_types)\n\n if as_dict:\n return dict(\n [(instance[\"description\"], instance) for instance in instance_types]\n )\n return instance_types\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.request_projects","title":"<code>request_projects(statuses=('active',))</code>","text":"<p>Query Conductor for all client Projects that are in the given status(es).</p> <p>Parameters:</p> Name Type Description Default <code>statuses</code> <code>tuple</code> <p>The statuses to filter for.</p> <code>('active',)</code> <p>Returns:</p> Type Description <code>list</code> <p>The list of project names.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def request_projects(statuses=(\"active\",)):\n\"\"\"\n Query Conductor for all client Projects that are in the given status(es).\n\n Args:\n statuses (tuple): The statuses to filter for.\n\n Returns:\n list: The list of project names.\n \"\"\"\n api = ApiClient()\n\n logger.debug(\"statuses: %s\", statuses)\n\n uri = \"api/v1/projects/\"\n\n response, response_code = api.make_request(\n uri_path=uri, verb=\"GET\", raise_on_error=False, use_api_key=True\n )\n logger.debug(\"response: %s\", response)\n logger.debug(\"response: %s\", response_code)\n if response_code not in [200]:\n msg = \"Failed to get available projects from Conductor\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n projects = []\n\n # Filter for only projects of the proper status\n for project in json.loads(response).get(\"data\") or []:\n if not statuses or project.get(\"status\") in statuses:\n projects.append(project[\"name\"])\n return projects\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.request_software_packages","title":"<code>request_software_packages()</code>","text":"<p>Query Conductor for all software packages for the currently available sidecar.</p> <p>Returns:</p> Type Description <code>list</code> <p>The list of software packages.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def request_software_packages():\n\"\"\"\n Query Conductor for all software packages for the currently available sidecar.\n\n Returns:\n list: The list of software packages.\n \"\"\"\n api = ApiClient()\n\n uri = \"api/v1/ee/packages?all=true,\"\n response, response_code = api.make_request(\n uri_path=uri, verb=\"GET\", raise_on_error=False, use_api_key=True\n )\n\n if response_code not in [200]:\n msg = \"Failed to get software packages for latest sidecar\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n software = json.loads(response).get(\"data\", [])\n software = [sw for sw in software if not (\"3dsmax\" in sw[\"product\"] and sw[\"platform\"] == \"linux\")]\n return software\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.request_extra_environment","title":"<code>request_extra_environment()</code>","text":"<p>Query Conductor for extra environment.</p> Source code in <code>ciocore/api_client.py</code> <pre><code>def request_extra_environment():\n\"\"\"\n Query Conductor for extra environment.\n \"\"\"\n api = ApiClient()\n\n uri = \"api/v1/integrations/env-vars-configs\"\n response, response_code = api.make_request(\n uri_path=uri, verb=\"GET\", raise_on_error=False, use_api_key=True\n )\n\n if response_code not in [200]:\n msg = \"Failed to get extra environment\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n all_accounts = json.loads(response).get(\"data\", [])\n\n token = read_conductor_credentials(True)\n if not token:\n raise Exception(\"Error: Could not get conductor credentials!\")\n account_id = str(account_id_from_jwt(token))\n\n if not account_id:\n raise Exception(\"Error: Could not get account id from jwt!\")\n account_env = next((account for account in all_accounts if account[\"account_id\"] == account_id), None)\n if not account_env:\n raise Exception(\"Error: Could not get account environment!\")\n return account_env.get(\"env\", [])\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_jobs","title":"<code>get_jobs(first_jid, last_jid=None)</code>","text":"<p>Query Conductor for all jobs between the given job ids.</p> <p>Returns:</p> Type Description <code>list</code> <p>The list of jobs.</p> <p>Exceptions:</p> Type Description <code>Exception</code> <p>If the request fails.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> jobs = api_client.get_jobs(1959)\n>>> len(jobs)\n1\n>>> jobs[0][\"jid\"]\n'01959'\n>>> jobs = api_client.get_jobs(1959, 1961)\n>>> len(jobs)\n3\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_jobs(first_jid, last_jid=None):\n\"\"\"\n Query Conductor for all jobs between the given job ids.\n\n Returns:\n list: The list of jobs.\n\n Raises:\n Exception: If the request fails.\n\n Examples:\n >>> from ciocore import api_client\n >>> jobs = api_client.get_jobs(1959)\n >>> len(jobs)\n 1\n >>> jobs[0][\"jid\"]\n '01959'\n >>> jobs = api_client.get_jobs(1959, 1961)\n >>> len(jobs)\n 3\n \"\"\"\n if last_jid is None:\n last_jid = first_jid\n low = str(int(first_jid) - 1).zfill(5)\n high = str(int(last_jid) + 1).zfill(5)\n api = ApiClient()\n uri = \"api/v1/jobs\"\n\n response, response_code = api.make_request(\n uri_path=uri,\n verb=\"GET\",\n raise_on_error=False,\n use_api_key=True,\n params={\"filter\": f\"jid_gt_{low},jid_lt_{high}\"},\n )\n\n if response_code not in [200]:\n msg = f\"Failed to get jobs {first_jid}-{last_jid}\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n jobs = json.loads(response).get(\"data\")\n return jobs\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_log","title":"<code>get_log(job_id, task_id)</code>","text":"<p>Get the log for the given job and task.</p> <p>Parameters:</p> Name Type Description Default <code>job_id</code> <code>str</code> <p>The job id.</p> required <code>task_id</code> <code>str</code> <p>The task id.</p> required <p>Returns:</p> Type Description <code>list</code> <p>A list of logs.</p> <p>Exceptions:</p> Type Description <code>Exception</code> <p>If the request fails.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> logs = api_client.get_log(1959, 0)\n{\n \"logs\": [\n {\n \"container_id\": \"j-5669544198668288-5619559933149184-5095331660038144-stde\",\n \"instance_name\": \"renderer-5669544198668288-170062309438-62994\",\n \"log\": [\n \"Blender 2.93.0 (hash 84da05a8b806 built 2021-06-02 11:29:24)\",\n ...\n ...\n \"Saved: '/var/folders/8r/46lmjdmj50x_0swd9klwptzm0000gq/T/blender_bmw/renders/render_0001.png'\",\n \" Time: 00:29.22 (Saving: 00:00.32)\",\n \"\",\n \"\",\n \"Blender quit\"\n ],\n \"timestamp\": \"1.700623521101516E9\"\n }\n ],\n \"new_num_lines\": [\n 144\n ],\n \"status_description\": \"\",\n \"task_status\": \"success\"\n}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_log(job_id, task_id):\n\"\"\"\n Get the log for the given job and task.\n\n Args:\n job_id (str): The job id.\n task_id (str): The task id.\n\n Returns:\n list: A list of logs.\n\n Raises:\n Exception: If the request fails.\n\n Examples:\n >>> from ciocore import api_client\n >>> logs = api_client.get_log(1959, 0)\n {\n \"logs\": [\n {\n \"container_id\": \"j-5669544198668288-5619559933149184-5095331660038144-stde\",\n \"instance_name\": \"renderer-5669544198668288-170062309438-62994\",\n \"log\": [\n \"Blender 2.93.0 (hash 84da05a8b806 built 2021-06-02 11:29:24)\",\n ...\n ...\n \"Saved: '/var/folders/8r/46lmjdmj50x_0swd9klwptzm0000gq/T/blender_bmw/renders/render_0001.png'\",\n \" Time: 00:29.22 (Saving: 00:00.32)\",\n \"\",\n \"\",\n \"Blender quit\"\n ],\n \"timestamp\": \"1.700623521101516E9\"\n }\n ],\n \"new_num_lines\": [\n 144\n ],\n \"status_description\": \"\",\n \"task_status\": \"success\"\n }\n \"\"\"\n job_id = str(job_id).zfill(5)\n task_id = str(task_id).zfill(3)\n\n api = ApiClient()\n uri = f\"get_log_file?job={job_id}&task={task_id}&num_lines[]=0\"\n\n response, response_code = api.make_request(\n uri_path=uri, verb=\"GET\", raise_on_error=False, use_api_key=True\n )\n\n if response_code not in [200]:\n msg = f\"Failed to get log for job {job_id} task {task_id}\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n return response\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.kill_jobs","title":"<code>kill_jobs(job_ids)</code>","text":"<p>Kill the given jobs.</p> <p>Parameters:</p> Name Type Description Default <code>job_ids</code> <code>list</code> <p>The list of job ids.</p> <code>()</code> <p>Returns:</p> Type Description <code>dict</code> <p>The response.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> api_client.kill_jobs(\"03095\",\"03094\")\n{'body': 'success', 'message': \"Jobs [u'03095', u'03094'] have been kill.\"}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def kill_jobs(job_ids):\n\"\"\"\n Kill the given jobs.\n\n Args:\n job_ids (list): The list of job ids.\n\n Returns:\n dict: The response.\n\n Examples:\n >>> from ciocore import api_client\n >>> api_client.kill_jobs(\"03095\",\"03094\")\n {'body': 'success', 'message': \"Jobs [u'03095', u'03094'] have been kill.\"}\n\n \"\"\"\n job_ids = [str(job_id).zfill(5) for job_id in job_ids]\n api = ApiClient()\n payload = {\n \"action\": \"kill\",\n \"jobids\": job_ids,\n }\n response, response_code = api.make_request(\n uri_path=\"jobs_multi\", \n verb=\"PUT\", \n raise_on_error=False, \n use_api_key=True, \n data=json.dumps(payload)\n )\n\n if response_code not in [200]:\n msg = f\"Failed to kill jobs {job_ids}\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n return json.loads(response)\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.kill_tasks","title":"<code>kill_tasks(job_id, task_ids)</code>","text":"<p>Kill the given tasks.</p> <p>Parameters:</p> Name Type Description Default <code>job_id</code> <code>str</code> <p>The job id.</p> required <code>task_ids</code> <code>list</code> <p>The list of task ids.</p> <code>()</code> <p>Returns:</p> Type Description <code>dict</code> <p>The response.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> api_client.kill_tasks(\"03096\", range(50,56))\n{'body': 'success', 'message': ' 6 Tasks set to \"kill\"\n050\n051\n052\n053\n054\n055'}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def kill_tasks(job_id, task_ids):\n\"\"\"\n Kill the given tasks.\n\n Args:\n job_id (str): The job id.\n task_ids (list): The list of task ids.\n\n Returns:\n dict: The response.\n\n Examples:\n >>> from ciocore import api_client\n >>> api_client.kill_tasks(\"03096\", range(50,56))\n {'body': 'success', 'message': ' 6 Tasks set to \"kill\"\\n\\t050\\n\\t051\\n\\t052\\n\\t053\\n\\t054\\n\\t055'}\n \"\"\"\n\n job_id = str(job_id).zfill(5)\n task_ids = [str(task_id).zfill(3) for task_id in task_ids]\n api = ApiClient()\n payload = {\n \"action\": \"kill\",\n \"jobid\": job_id,\n \"taskids\": task_ids,\n }\n response, response_code = api.make_request(\n uri_path=\"tasks_multi\", \n verb=\"PUT\", \n raise_on_error=False, \n use_api_key=True, \n data=json.dumps(payload)\n )\n\n if response_code not in [200]:\n msg = f\"Failed to kill tasks {task_ids} of job {job_id}\"\n msg += \"\\nError %s ...\\n%s\" % (response_code, response)\n raise Exception(msg)\n\n return json.loads(response)\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_compute_usage","title":"<code>get_compute_usage(start_time, end_time)</code>","text":"<p>Query the compute usage for an account.</p> <p>Compute includes licenses, instances and Conductor cost. Everything involved with running a job. </p> <p>Parameters:</p> Name Type Description Default <code>start_time</code> <code>datetime.datetime</code> <p>The first day to include in the report. Only the date is considered and it's assumed to be in UTC.</p> required <code>end_time</code> <code>datetime.datetime</code> <p>The last day to include in the report. Only the date is considered and it's assumed to be in UTC.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>Each key is a date (UTC). The value is a dict with values for: - cost: The total accumulated compute cost for the day - corehours: The total accumulated core hours for the day - walltime: The number of minutes that instances (regardless of type) were running</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> api_client.get_compute_usage(start_time, end_time)\n{ '2024-01-09': { 'cost': 0.08,\n 'corehours': 0.9, \n 'walltime': 13.93},\n '2024-01-16': { 'cost': 0.12,\n 'corehours': 0.9613, \n 'walltime': 7.21}}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_compute_usage(start_time, end_time):\n'''\n Query the compute usage for an account.\n\n Compute includes licenses, instances and Conductor cost. Everything involved\n with running a job. \n\n Args:\n start_time (datetime.datetime): The first day to include in the report. Only the date is considered and it's assumed to be in UTC.\n end_time (datetime.datetime): The last day to include in the report. Only the date is considered and it's assumed to be in UTC.\n\n Returns:\n dict: Each key is a date (UTC). The value is a dict with values for:\n - cost: The total accumulated compute cost for the day\n - corehours: The total accumulated core hours for the day\n - walltime: The number of minutes that instances (regardless of type) were running\n\n Examples:\n >>> from ciocore import api_client\n >>> api_client.get_compute_usage(start_time, end_time)\n { '2024-01-09': { 'cost': 0.08,\n 'corehours': 0.9, \n 'walltime': 13.93},\n '2024-01-16': { 'cost': 0.12,\n 'corehours': 0.9613, \n 'walltime': 7.21}}\n '''\n date_format = \"%a, %d %b %Y %H:%M:%S %Z\"\n data = _get_compute_usage(start_time, end_time)\n\n # Create a nested default dictionary with initial float values of 0.0\n results = collections.defaultdict(lambda: collections.defaultdict(float))\n\n for entry in data:\n entry_start_date = datetime.datetime.strptime(entry['start_time'], date_format).date().isoformat()\n\n results[entry_start_date]['walltime'] += entry['minutes']\n results[entry_start_date]['corehours'] += entry['cores']\n results[entry_start_date]['cost'] += entry['license_cost'] + entry['instance_cost']\n\n # Round the data to avoid FP errors\n results[entry_start_date]['walltime'] = round(results[entry_start_date]['walltime'], 4)\n results[entry_start_date]['corehours'] = round(results[entry_start_date]['corehours'], 4)\n results[entry_start_date]['cost'] = round(results[entry_start_date]['cost'], 4)\n\n return results\n</code></pre>"},{"location":"apidoc/api_client/#ciocore.api_client.get_storage_usage","title":"<code>get_storage_usage(start_time, end_time)</code>","text":"<p>Query the storage usage for an account.</p> <p>Storage is calculated twice a day (UTC) and the average is used.</p> <p>Parameters:</p> Name Type Description Default <code>start_time</code> <code>datetime.datetime</code> <p>The first day to include in the report. Only the date is considered and it's assumed to be in UTC.</p> required <code>end_time</code> <code>datetime.datetime</code> <p>The last day to include in the report. Only the date is considered and it's assumed to be in UTC.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>Each key is a date (UTC). The value is a dict with values for: - cost: The cost of accumulated storage for that one day - GiB: The total amount of storage used on that day</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client\n>>> api_client.get_storage_usage(start_time, end_time)\n{ '2024-01-01': {'cost': 4.022, 'GiB': 679.714},\n '2024-01-02': {'cost': 4.502, 'GiB': 750.34},\n '2024-01-03': {'cost': 4.502, 'GiB': 750.34}}\n</code></pre> Source code in <code>ciocore/api_client.py</code> <pre><code>def get_storage_usage(start_time, end_time):\n'''\n Query the storage usage for an account.\n\n Storage is calculated twice a day (UTC) and the average is used.\n\n Args:\n start_time (datetime.datetime): The first day to include in the report. Only the date is considered and it's assumed to be in UTC.\n end_time (datetime.datetime): The last day to include in the report. Only the date is considered and it's assumed to be in UTC.\n\n Returns:\n dict: Each key is a date (UTC). The value is a dict with values for:\n - cost: The cost of accumulated storage for that one day\n - GiB: The total amount of storage used on that day\n\n Examples:\n >>> from ciocore import api_client\n >>> api_client.get_storage_usage(start_time, end_time)\n { '2024-01-01': {'cost': 4.022, 'GiB': 679.714},\n '2024-01-02': {'cost': 4.502, 'GiB': 750.34},\n '2024-01-03': {'cost': 4.502, 'GiB': 750.34}}\n '''\n one_day = datetime.timedelta(days=1)\n\n data = _get_storage_usage(start_time, end_time)\n\n results = {}\n\n entry_date = datetime.date.fromisoformat(data['start_date'])\n\n for cnt, entry in enumerate(data[\"cost_per_day\"]):\n\n entry_start_date = entry_date.isoformat()\n results[entry_start_date] = {'cost': float(entry), 'GiB': float(data['gibs_per_day'][cnt])}\n entry_date += one_day\n\n return results\n</code></pre>"},{"location":"apidoc/apidoc/","title":"Api Overview","text":"<p>Conductor's Python API is intended to be used by developers to build custom submission tools, and to integrate Conductor into existing workflows. In order to use the API, you will need to have a Conductor account, and we recommend that you use an API key to authenticate.</p> <p>The goal of all Conductor submitters is to create a payload to be posted in an HTTP request. The payload is a JSON object that describes the submission, and includes a list of files to be uploaded to Conductor's storage. Let's take a look at a Conductor submission payload.</p>"},{"location":"apidoc/apidoc/#anatomy-of-a-conductor-submission","title":"Anatomy of a Conductor Submission","text":"<p>A Conductor submission consists of a JSON payload that describes the submission, and a collection of files that are uploaded to Conductor's storage.</p> <pre><code>{\n\"job_title\": \"Blender example bmw_half_turn_low.blend\",\n\"project\": \"default\",\n\"instance_type\": \"n1-standard-4\",\n\"software_package_ids\": [\n\"7be1b2410b3f93b2a2889f6ce191d4e1\"\n],\n\"force\": false,\n\"local_upload\": true,\n\"preemptible\": true,\n\"autoretry_policy\": {\n\"preempted\": {\n\"max_retries\": 3\n}\n},\n\"output_path\": \"projects/blender/renders\",\n\"environment\": {\n\"PATH\": \"/opt/blenderfoundation/blender/2/blender-2.93.0-glibc217\",\n\"CONDUCTOR_PATHHELPER\": \"0\"\n},\n\"upload_paths\": [\n\"/projects/blender/bmw_half_turn_low.blend\",\n\"/projects/blender/sourceimages/img_0001.png\",\n],\n\"scout_frames\": \"1,3\",\n\"tasks_data\": [\n{\n\"command\": \"blender -b \\\"/projects/blender/bmw_half_turn_low.blend\\\" -E CYCLES --render-output \\\"/projects/blender/renders/img_\\\" --render-frame 1..1 \",\n\"frames\": \"1\"\n},\n{\n\"command\": \"blender -b \\\"/projects/blender/bmw_half_turn_low.blend\\\" -E CYCLES --render-output \\\"/projects/blender/renders/img_\\\" --render-frame 2..2 \",\n\"frames\": \"2\"\n},\n{\n\"command\": \"blender -b \\\"/projects/blender/bmw_half_turn_low.blend\\\" -E CYCLES --render-output \\\"/projects/blender/renders/img_\\\" --render-frame 3..3 \",\n\"frames\": \"3\"\n},\n]\n}\n</code></pre>"},{"location":"apidoc/apidoc/#authentication","title":"Authentication","text":"<p>WIP</p>"},{"location":"apidoc/config/","title":"ciocore.config","text":"<p>Config is a configuration object implemented as a module-level singleton.</p> <p>Configuration variables can be shared by importing the module. If there are changes in environment variables or other sources, the config can be refreshed.</p>"},{"location":"apidoc/config/#ciocore.config.Config","title":"<code> Config </code>","text":"Source code in <code>ciocore/config.py</code> <pre><code>class Config(object):\n def __init__(self):\n\"\"\"\n Initialize the config object.\n\n A config object is a dictionary containing configuration values. It is a singleton, so there is only one instance of it. It is instantiated the first time it is needed. It can be refreshed by calling get() with the `force` keyword argument set to `True`.\n\n A Config object has the following properties:\n\n `thread_count` The number of threads to use for downloading files. Defaults to the number of CPUs on the system times 2. It can be overridden by the `CONDUCTOR_THREAD_COUNT` environment variable.\n * `priority` Set the priority for submissions. Defaults to 5. It can be overridden by the `CONDUCTOR_PRIORITY` environment variable.\n * `md5_caching` Whether to cache MD5s. Defaults to `True`. It can be overridden by the `CONDUCTOR_MD5_CACHING` environment variable. Cachine MD5s significantly improves submission performance, but on rare occasions it can cause submissions to fail. If you experience this, set `md5_caching` to `False`.\n * `log_level` The logging level. Defaults to `INFO`. It can be overridden by the `CONDUCTOR_LOG_LEVEL` environment variable.\n * `url` The URL of the Conductor dashboard. Defaults to `https://dashboard.conductortech.com`. It can be overridden by the `CONDUCTOR_URL` environment variable.\n * `auth_url` The URL of the Conductor dashboard. Defaults to `https://dashboard.conductortech.com`. It can be overridden by the `CONDUCTOR_AUTH_URL` environment variable. This is deprecated. Use `url` instead.\n * `api_url` The URL of the Conductor API. Defaults to `https://api.conductortech.com`. It can be overridden by the `CONDUCTOR_API_URL` environment variable.\n * `api_key` The API key. The API key can be acquired from the Conductor dashboard, and can be stored in an environment variable or a file. In both cases the API KEY can be a JSON object or a base64 encoded JSON object. If it is base64 encoded, it can be a string or bytes. If it is a string, it will be decoded as ASCII. If it is bytes, it will be decoded as UTF-8.\n * Environment variable: The `CONDUCTOR_API_KEY` variable can hold the API KEY directly.\n * File: The `CONDUCTOR_API_KEY_PATH` variable can hold the path to a file containing the API KEY.\n * `downloader_page_size` The number of files to request from the Conductor API at a time. Defaults to 50. It can be overridden by the `CONDUCTOR_DOWNLOADER_PAGE_SIZE` environment variable.\n\n Returns:\n Config: A config object.\n\n Raises:\n ValueError -- Invalid inputs, such as badly formed URLs.\n \"\"\"\n default_downloader_page_size = 50\n\n\n\n try:\n default_thread_count = min( os.cpu_count() - 1, 15)\n except NotImplementedError:\n default_thread_count = 15\n\n url = os.environ.get(\"CONDUCTOR_URL\", \"https://dashboard.conductortech.com\")\n\n if not URL_REGEX.match(url):\n raise ValueError(\"CONDUCTOR_URL is not valid '{}'\".format(url))\n\n api_url = os.environ.get(\"CONDUCTOR_API_URL\", url.replace(\"dashboard\", \"api\"))\n if not URL_REGEX.match(api_url):\n raise ValueError(\"CONDUCTOR_API_URL is not valid '{}'\".format(api_url))\n\n falsy = [\"false\", \"no\", \"off\", \"0\"]\n\n log_level = os.environ.get(\"CONDUCTOR_LOG_LEVEL\", \"INFO\")\n if log_level not in [\"CRITICAL\", \"ERROR\", \"WARNING\", \"INFO\", \"DEBUG\"]:\n log_level = \"INFO\"\n\n self.config = {\n \"thread_count\": int(\n os.environ.get(\"CONDUCTOR_THREAD_COUNT\", default_thread_count)\n ),\n \"downloader_page_size\": int(\n os.environ.get(\n \"CONDUCTOR_DOWNLOADER_PAGE_SIZE\", default_downloader_page_size\n )\n ),\n \"priority\": int(os.environ.get(\"CONDUCTOR_PRIORITY\", 5)),\n \"md5_caching\": False\n if os.environ.get(\"CONDUCTOR_MD5_CACHING\", \"True\").lower() in falsy\n else True,\n \"log_level\": log_level,\n \"url\": url,\n # Keep \"auth_url\" for backwwards compatibillity only.\n # Clients should use \"url\" moving forward.\n # Remove \"auth_url\" on the next major version bump.\n \"auth_url\": url,\n \"api_url\": api_url,\n \"api_key\": self.get_api_key_from_variable() or self.get_api_key_from_file(),\n \"user_dir\": os.environ.get('CONDUCTOR_USER_DIR', DEFAULT_USER_DIR)\n }\n\n @staticmethod\n def get_api_key_from_variable():\n\"\"\"\n Attempt to get an API key from the `CONDUCTOR_API_KEY` environment variable.\n\n Raises:\n ValueError: An error occurred while reading or loading the key into JSON.\n\n Returns:\n str: JSON object containing the key - base 64 decoded if necessary.\n\n \"\"\"\n api_key = os.environ.get(\"CONDUCTOR_API_KEY\")\n if not api_key:\n return\n logger.info(\"Attempting to read API key from CONDUCTOR_API_KEY\")\n try:\n return json.loads(api_key.replace(\"\\n\", \"\").replace(\"\\r\", \"\"))\n except ValueError:\n try:\n result = base64.b64decode(api_key)\n return Config._to_json(result)\n except BaseException:\n result = base64.b64decode(api_key.encode()).decode(\"ascii\")\n return Config._to_json(result)\n except BaseException:\n message = \"An error occurred reading the API key from the CONDUCTOR_API_KEY variable\"\n logger.error(message)\n raise ValueError(message)\n\n @staticmethod\n def get_api_key_from_file():\n\"\"\"\n Attempt to get an API key from the file in the CONDUCTOR_API_KEY_PATH environment variable.\n\n Raises:\n ValueError: An error occurred while reading or loading the key into JSON.\n\n Returns:\n str: JSON object containing the key - base 64 decoded if necessary.\n\n \"\"\"\n api_key_path = os.environ.get(\"CONDUCTOR_API_KEY_PATH\")\n if not api_key_path:\n return\n logger.info(\"Attempting to read API key from CONDUCTOR_API_KEY_PATH\")\n try:\n with open(api_key_path, \"r\") as fp:\n return Config._to_json(fp.read())\n except BaseException:\n message = \"An error occurred reading the API key from the path described in the CONDUCTOR_API_KEY_PATH variable\"\n logger.error(message)\n raise ValueError(message)\n\n @staticmethod\n def _to_json(content):\n return json.loads(content.replace(\"\\n\", \"\").replace(\"\\r\", \"\"))\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.Config.__init__","title":"<code>__init__(self)</code> <code>special</code>","text":"<p>Initialize the config object.</p> <p>A config object is a dictionary containing configuration values. It is a singleton, so there is only one instance of it. It is instantiated the first time it is needed. It can be refreshed by calling get() with the <code>force</code> keyword argument set to <code>True</code>.</p> <p>A Config object has the following properties:</p> <ul> <li><code>thread_count</code> The number of threads to use for downloading files. Defaults to the number of CPUs on the system times 2. It can be overridden by the <code>CONDUCTOR_THREAD_COUNT</code> environment variable.</li> <li><code>priority</code> Set the priority for submissions. Defaults to 5. It can be overridden by the <code>CONDUCTOR_PRIORITY</code> environment variable.</li> <li><code>md5_caching</code> Whether to cache MD5s. Defaults to <code>True</code>. It can be overridden by the <code>CONDUCTOR_MD5_CACHING</code> environment variable. Cachine MD5s significantly improves submission performance, but on rare occasions it can cause submissions to fail. If you experience this, set <code>md5_caching</code> to <code>False</code>.</li> <li><code>log_level</code> The logging level. Defaults to <code>INFO</code>. It can be overridden by the <code>CONDUCTOR_LOG_LEVEL</code> environment variable.</li> <li><code>url</code> The URL of the Conductor dashboard. Defaults to <code>https://dashboard.conductortech.com</code>. It can be overridden by the <code>CONDUCTOR_URL</code> environment variable.</li> <li><code>auth_url</code> The URL of the Conductor dashboard. Defaults to <code>https://dashboard.conductortech.com</code>. It can be overridden by the <code>CONDUCTOR_AUTH_URL</code> environment variable. This is deprecated. Use <code>url</code> instead.</li> <li><code>api_url</code> The URL of the Conductor API. Defaults to <code>https://api.conductortech.com</code>. It can be overridden by the <code>CONDUCTOR_API_URL</code> environment variable.</li> <li><code>api_key</code> The API key. The API key can be acquired from the Conductor dashboard, and can be stored in an environment variable or a file. In both cases the API KEY can be a JSON object or a base64 encoded JSON object. If it is base64 encoded, it can be a string or bytes. If it is a string, it will be decoded as ASCII. If it is bytes, it will be decoded as UTF-8.<ul> <li>Environment variable: The <code>CONDUCTOR_API_KEY</code> variable can hold the API KEY directly.</li> <li>File: The <code>CONDUCTOR_API_KEY_PATH</code> variable can hold the path to a file containing the API KEY.</li> </ul> </li> <li><code>downloader_page_size</code> The number of files to request from the Conductor API at a time. Defaults to 50. It can be overridden by the <code>CONDUCTOR_DOWNLOADER_PAGE_SIZE</code> environment variable.</li> </ul> <p>Returns:</p> Type Description <code>Config</code> <p>A config object.</p> Source code in <code>ciocore/config.py</code> <pre><code>def __init__(self):\n\"\"\"\n Initialize the config object.\n\n A config object is a dictionary containing configuration values. It is a singleton, so there is only one instance of it. It is instantiated the first time it is needed. It can be refreshed by calling get() with the `force` keyword argument set to `True`.\n\n A Config object has the following properties:\n\n * `thread_count` The number of threads to use for downloading files. Defaults to the number of CPUs on the system times 2. It can be overridden by the `CONDUCTOR_THREAD_COUNT` environment variable.\n * `priority` Set the priority for submissions. Defaults to 5. It can be overridden by the `CONDUCTOR_PRIORITY` environment variable.\n * `md5_caching` Whether to cache MD5s. Defaults to `True`. It can be overridden by the `CONDUCTOR_MD5_CACHING` environment variable. Cachine MD5s significantly improves submission performance, but on rare occasions it can cause submissions to fail. If you experience this, set `md5_caching` to `False`.\n * `log_level` The logging level. Defaults to `INFO`. It can be overridden by the `CONDUCTOR_LOG_LEVEL` environment variable.\n * `url` The URL of the Conductor dashboard. Defaults to `https://dashboard.conductortech.com`. It can be overridden by the `CONDUCTOR_URL` environment variable.\n * `auth_url` The URL of the Conductor dashboard. Defaults to `https://dashboard.conductortech.com`. It can be overridden by the `CONDUCTOR_AUTH_URL` environment variable. This is deprecated. Use `url` instead.\n * `api_url` The URL of the Conductor API. Defaults to `https://api.conductortech.com`. It can be overridden by the `CONDUCTOR_API_URL` environment variable.\n * `api_key` The API key. The API key can be acquired from the Conductor dashboard, and can be stored in an environment variable or a file. In both cases the API KEY can be a JSON object or a base64 encoded JSON object. If it is base64 encoded, it can be a string or bytes. If it is a string, it will be decoded as ASCII. If it is bytes, it will be decoded as UTF-8.\n * Environment variable: The `CONDUCTOR_API_KEY` variable can hold the API KEY directly.\n * File: The `CONDUCTOR_API_KEY_PATH` variable can hold the path to a file containing the API KEY.\n * `downloader_page_size` The number of files to request from the Conductor API at a time. Defaults to 50. It can be overridden by the `CONDUCTOR_DOWNLOADER_PAGE_SIZE` environment variable.\n\n Returns:\n Config: A config object.\n\n Raises:\n ValueError -- Invalid inputs, such as badly formed URLs.\n \"\"\"\n default_downloader_page_size = 50\n\n\n\n try:\n default_thread_count = min( os.cpu_count() - 1, 15)\n except NotImplementedError:\n default_thread_count = 15\n\n url = os.environ.get(\"CONDUCTOR_URL\", \"https://dashboard.conductortech.com\")\n\n if not URL_REGEX.match(url):\n raise ValueError(\"CONDUCTOR_URL is not valid '{}'\".format(url))\n\n api_url = os.environ.get(\"CONDUCTOR_API_URL\", url.replace(\"dashboard\", \"api\"))\n if not URL_REGEX.match(api_url):\n raise ValueError(\"CONDUCTOR_API_URL is not valid '{}'\".format(api_url))\n\n falsy = [\"false\", \"no\", \"off\", \"0\"]\n\n log_level = os.environ.get(\"CONDUCTOR_LOG_LEVEL\", \"INFO\")\n if log_level not in [\"CRITICAL\", \"ERROR\", \"WARNING\", \"INFO\", \"DEBUG\"]:\n log_level = \"INFO\"\n\n self.config = {\n \"thread_count\": int(\n os.environ.get(\"CONDUCTOR_THREAD_COUNT\", default_thread_count)\n ),\n \"downloader_page_size\": int(\n os.environ.get(\n \"CONDUCTOR_DOWNLOADER_PAGE_SIZE\", default_downloader_page_size\n )\n ),\n \"priority\": int(os.environ.get(\"CONDUCTOR_PRIORITY\", 5)),\n \"md5_caching\": False\n if os.environ.get(\"CONDUCTOR_MD5_CACHING\", \"True\").lower() in falsy\n else True,\n \"log_level\": log_level,\n \"url\": url,\n # Keep \"auth_url\" for backwwards compatibillity only.\n # Clients should use \"url\" moving forward.\n # Remove \"auth_url\" on the next major version bump.\n \"auth_url\": url,\n \"api_url\": api_url,\n \"api_key\": self.get_api_key_from_variable() or self.get_api_key_from_file(),\n \"user_dir\": os.environ.get('CONDUCTOR_USER_DIR', DEFAULT_USER_DIR)\n }\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.Config.get_api_key_from_variable","title":"<code>get_api_key_from_variable()</code> <code>staticmethod</code>","text":"<p>Attempt to get an API key from the <code>CONDUCTOR_API_KEY</code> environment variable.</p> <p>Exceptions:</p> Type Description <code>ValueError</code> <p>An error occurred while reading or loading the key into JSON.</p> <p>Returns:</p> Type Description <code>str</code> <p>JSON object containing the key - base 64 decoded if necessary.</p> Source code in <code>ciocore/config.py</code> <pre><code>@staticmethod\ndef get_api_key_from_variable():\n\"\"\"\n Attempt to get an API key from the `CONDUCTOR_API_KEY` environment variable.\n\n Raises:\n ValueError: An error occurred while reading or loading the key into JSON.\n\n Returns:\n str: JSON object containing the key - base 64 decoded if necessary.\n\n \"\"\"\n api_key = os.environ.get(\"CONDUCTOR_API_KEY\")\n if not api_key:\n return\n logger.info(\"Attempting to read API key from CONDUCTOR_API_KEY\")\n try:\n return json.loads(api_key.replace(\"\\n\", \"\").replace(\"\\r\", \"\"))\n except ValueError:\n try:\n result = base64.b64decode(api_key)\n return Config._to_json(result)\n except BaseException:\n result = base64.b64decode(api_key.encode()).decode(\"ascii\")\n return Config._to_json(result)\n except BaseException:\n message = \"An error occurred reading the API key from the CONDUCTOR_API_KEY variable\"\n logger.error(message)\n raise ValueError(message)\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.Config.get_api_key_from_file","title":"<code>get_api_key_from_file()</code> <code>staticmethod</code>","text":"<p>Attempt to get an API key from the file in the CONDUCTOR_API_KEY_PATH environment variable.</p> <p>Exceptions:</p> Type Description <code>ValueError</code> <p>An error occurred while reading or loading the key into JSON.</p> <p>Returns:</p> Type Description <code>str</code> <p>JSON object containing the key - base 64 decoded if necessary.</p> Source code in <code>ciocore/config.py</code> <pre><code>@staticmethod\ndef get_api_key_from_file():\n\"\"\"\n Attempt to get an API key from the file in the CONDUCTOR_API_KEY_PATH environment variable.\n\n Raises:\n ValueError: An error occurred while reading or loading the key into JSON.\n\n Returns:\n str: JSON object containing the key - base 64 decoded if necessary.\n\n \"\"\"\n api_key_path = os.environ.get(\"CONDUCTOR_API_KEY_PATH\")\n if not api_key_path:\n return\n logger.info(\"Attempting to read API key from CONDUCTOR_API_KEY_PATH\")\n try:\n with open(api_key_path, \"r\") as fp:\n return Config._to_json(fp.read())\n except BaseException:\n message = \"An error occurred reading the API key from the path described in the CONDUCTOR_API_KEY_PATH variable\"\n logger.error(message)\n raise ValueError(message)\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.config","title":"<code>config(force=False)</code>","text":"<p>Instantiate a config object if necessary.</p> <p>Deprecated</p> <p>Use get() instead.</p> <p>Parameters:</p> Name Type Description Default <code>force</code> <code>bool</code> <p>Discards any existing config object and instantiate a new one -- Defaults to <code>False</code>.</p> <code>False</code> <p>Returns:</p> Type Description <code>dict</code> <p>A dictionary containing configuration values.</p> Source code in <code>ciocore/config.py</code> <pre><code>def config(force=False):\n\"\"\"\n Instantiate a config object if necessary.\n\n Deprecated:\n Use [get()](#get) instead.\n\n Args:\n force (bool): Discards any existing config object and instantiate a new one -- Defaults to `False`.\n\n Returns:\n dict: A dictionary containing configuration values.\n \"\"\"\n\n global __config__\n if force or not __config__:\n __config__ = Config()\n return __config__\n</code></pre>"},{"location":"apidoc/config/#ciocore.config.get","title":"<code>get(force=False)</code>","text":"<p>Instantiate a config object if necessary and return the dictionary.</p> <p>Parameters:</p> Name Type Description Default <code>force</code> <code>bool</code> <p>Discards any existing config object and instantiate a new one -- Defaults to <code>False</code>.</p> <code>False</code> <p>Returns:</p> Type Description <code>dict</code> <p>A dictionary containing configuration values.</p> <p>Examples:</p> <pre><code>>>> from ciocore import config\n>>> config.get()\n{\n 'thread_count': 16,\n 'priority': 5,\n 'md5_caching': True,\n 'log_level': 'INFO',\n 'url': 'https://dashboard.conductortech.com',\n 'auth_url': 'https://dashboard.conductortech.com',\n 'api_url': 'https://api.conductortech.com',\n 'api_key': None,\n 'downloader_page_size': 50,\n}\n</code></pre> Source code in <code>ciocore/config.py</code> <pre><code>def get(force=False):\n\"\"\"\n Instantiate a config object if necessary and return the dictionary.\n\n Args:\n force (bool): Discards any existing config object and instantiate a new one -- Defaults to `False`.\n\n Returns:\n dict: A dictionary containing configuration values.\n\n Example:\n >>> from ciocore import config\n >>> config.get()\n {\n 'thread_count': 16,\n 'priority': 5,\n 'md5_caching': True,\n 'log_level': 'INFO',\n 'url': 'https://dashboard.conductortech.com',\n 'auth_url': 'https://dashboard.conductortech.com',\n 'api_url': 'https://api.conductortech.com',\n 'api_key': None,\n 'downloader_page_size': 50,\n }\n \"\"\"\n global __config__\n if force or not __config__:\n __config__ = Config()\n return __config__.config\n</code></pre>"},{"location":"apidoc/data/","title":"ciocore.data","text":"<p>This module is a singleton that provides the data from Conductor endpoints. Specifically, it provides projects, instance types, and software package data.</p> <p>Since the data is stored at the module level, you can access it from anywhere in your code without the need to pass it around.</p>"},{"location":"apidoc/data/#ciocore.data.init","title":"<code>init(products, kwargs)</code>","text":"<p>Initialize the module and let it know what host products to provide.</p> <p>Parameters:</p> Name Type Description Default <code>products</code> <code>str</code> <p>Provide a list of products for which to get software packages. If no products are given, the software data contains all products from the packages endpoint. If you provide more than one product, they must all be host level products.</p> <code>()</code> <p>Keyword arguments:</p> Name Type Description <code>product</code> <code>str</code> <p><code>DEPRECATED</code> Provide one product for which to get software packages.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n# OR\n>>> coredata.init(\"maya-io\")\n# OR LEGACY\n>>> coredata.init(product=\"all\")\n# OR\n>>> coredata.init(product=\"maya-io\")\n</code></pre> Source code in <code>ciocore/data.py</code> <pre><code>def init(products, *kwargs):\n\"\"\"\n Initialize the module and let it know what host products to provide.\n\n Args:\n products (str): Provide a list of products for which to get software packages. If no products are given, the software data contains all products from the packages endpoint. If you provide more than one product, they must all be host level products.\n\n Keyword Args:\n product (str): `DEPRECATED` Provide one product for which to get software packages.\n\n Examples:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n # OR\n >>> coredata.init(\"maya-io\")\n # OR LEGACY\n >>> coredata.init(product=\"all\")\n # OR\n >>> coredata.init(product=\"maya-io\")\n \"\"\"\n global __products__\n global __platforms__\n if products:\n if kwargs.get(\"product\"):\n raise ValueError(\n \"Arguments: `products` and `product` specified. Please don't use both together. The `product` arg is deprecated.\"\n )\n __products__ = list(products)\n elif kwargs.get(\"product\"):\n if kwargs.get(\"product\") == \"all\":\n __products__ = []\n else:\n __products__ = [kwargs.get(\"product\")]\n else:\n __products__ = []\n\n __platforms__ = set(kwargs.get(\"platforms\", [\"windows\", \"linux\"]))\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.data","title":"<code>data(force=False)</code>","text":"<p>Provide projects, instance types, and software package data.</p> <p>Keyword arguments:</p> Name Type Description <code>force</code> <p>(bool) If <code>True</code>, then force the system to fetch fresh data -- Defaults to <code>False</code>.</p> <p>Exceptions:</p> Type Description <code>ValueError</code> <p>Module was not initialized with init().</p> <p>Returns:</p> Type Description <code>dict</code> <p>Keys are <code>projects</code>, <code>instance_types</code>, <code>software</code>.</p> <p>When you access the data, if it has already been fetched, it will be returned. Otherwise, requests will be made to fetch the data. You may need to authenticate in order to access the data.</p> <p>The set of instance types and software can be pruned to match the available platforms represented by each other. For example, if the instance types come from an orchestrator that provides both Windows and Linux machines, and the software product(s) are available on both platforms, no pruning occurs. However, if there are no Windows machines available, any Windows software will be removed from the package tree. Similarly, if a product is chosen that only runs on Windows, Linux instance types will not appearin the list of available hardware.</p> <p>Here is a breakdown of each key in the dictionary:</p> <ul> <li> <p>projects is a list of project names for your authenticated account.</p> </li> <li> <p>instance_types is an instance of HardwareSet, providing you with access to the list of available machines configurations.</p> </li> <li> <p>software is a PackageTree object containing either all the software available at Conductor, or a subset based on specified products.</p> </li> </ul> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init(product=\"maya-io\")\n</code></pre> <pre><code>>>> coredata.data()[\"software\"]\n<ciocore.package_tree.PackageTree object at 0x10e9a4040>\n</code></pre> <pre><code>>>> coredata.data()[\"projects\"][0]\nATestForScott\n</code></pre> <pre><code>>>> coredata.data()[\"instance_types\"]\n<ciocore.hardware_set.HardwareSet object at 0x0000028941CD9DC0>\n</code></pre> Source code in <code>ciocore/data.py</code> <pre><code>def data(force=False):\n\"\"\"\n Provide projects, instance types, and software package data.\n\n Keyword Args:\n force: (bool) If `True`, then force the system to fetch fresh data -- Defaults to `False`.\n\n Raises:\n ValueError: Module was not initialized with [init()](/data/#ciocore.data.init).\n\n Returns:\n dict: Keys are `projects`, `instance_types`, `software`.\n\n When you access the data, if it has already been fetched, it will be returned. Otherwise,\n requests will be made to fetch the data. You may need to authenticate in order to access the\n data.\n\n The set of instance types and software can be pruned to match the available platforms\n represented by each other. For example, if the instance types come from an orchestrator that\n provides both Windows and Linux machines, and the software product(s) are available on both\n platforms, no pruning occurs. However, if there are no Windows machines available, any Windows\n software will be removed from the package tree. Similarly, if a product is chosen that only runs\n on Windows, Linux instance types will not appearin the list of available hardware.\n\n Here is a breakdown of each key in the dictionary:\n\n projects is a list of project names for your authenticated account.\n\n * instance_types is an instance of HardwareSet, providing you with access to the list of\n available machines configurations.\n\n * software is a PackageTree object containing either all\n the software available at Conductor, or a subset based on specified products.\n\n\n Examples:\n >>> from ciocore import data as coredata\n >>> coredata.init(product=\"maya-io\")\n\n >>> coredata.data()[\"software\"]\n <ciocore.package_tree.PackageTree object at 0x10e9a4040>\n\n >>> coredata.data()[\"projects\"][0]\n ATestForScott\n\n >>> coredata.data()[\"instance_types\"]\n <ciocore.hardware_set.HardwareSet object at 0x0000028941CD9DC0>\n \"\"\"\n\n global __data__\n global __products__\n global __platforms__\n\n if __products__ is None:\n raise ValueError(\n 'Data must be initialized before use, e.g. data.init(\"maya-io\") or data.init().'\n )\n products_copy = copy.copy(__products__)\n\n if force:\n clear()\n init(products_copy)\n\n if __data__ == {}:\n # PROJECTS\n __data__[\"projects\"] = sorted(api_client.request_projects())\n # INST_TYPES\n instance_types = api_client.request_instance_types()\n # SOFTWARE\n software = api_client.request_software_packages()\n\n # EXTRA ENV VARS\n extra_env_vars = []\n try:\n extra_env_vars = api_client.request_extra_environment()\n except Exception:\n pass \n __data__[\"extra_environment\"] = extra_env_vars\n\n # PLATFORMS\n it_platforms = set([it[\"operating_system\"] for it in instance_types])\n valid_platforms = it_platforms.intersection(__platforms__)\n kwargs = {\"platforms\": valid_platforms}\n\n # If there's only one product, it's possible to initialize the software tree with a plugin.\n # So we set the product kwarg. Otherwise, we set the host_products kwarg\n host_products = __products__\n if len(__products__) == 1:\n host_products = []\n kwargs[\"product\"] = __products__[0]\n\n software_tree = PackageTree(software, host_products, kwargs)\n\n if software_tree:\n __data__[\"software\"] = software_tree\n # Revisit instance types to filter out any that are not needed for any software package.\n sw_platforms = software_tree.platforms()\n\n instance_types = [\n it for it in instance_types if it[\"operating_system\"] in sw_platforms\n ]\n # Then adjust __platforms__ to match the instance types that are represented.\n __platforms__ = set([it[\"operating_system\"] for it in instance_types])\n\n __data__[\"instance_types\"] = HardwareSet(instance_types)\n\n return __data__\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.valid","title":"<code>valid()</code>","text":"<p>Check validity.</p> <p>Returns:</p> Type Description <code>bool</code> <p>True if <code>projects</code>, <code>instance_types</code>, and <code>software</code> are valid.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.valid()\nTrue\n</code></pre> Source code in <code>ciocore/data.py</code> <pre><code>def valid():\n\"\"\"\n Check validity.\n\n Returns:\n bool: True if `projects`, `instance_types`, and `software` are valid.\n\n Examples:\n >>> from ciocore import data as coredata\n >>> coredata.valid()\n True\n \"\"\"\n\n if not __data__.get(\"projects\"):\n return False\n if not __data__.get(\"instance_types\"):\n return False\n if not __data__.get(\"software\"):\n return False\n return True\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.clear","title":"<code>clear()</code>","text":"<p>Clear out data.</p> <p>valid() returns False after clear().</p> Source code in <code>ciocore/data.py</code> <pre><code>def clear():\n\"\"\"\n Clear out data.\n\n [valid()](/data/#ciocore.data.valid) returns False after clear().\n \"\"\"\n global __data__\n global __products__\n global __platforms__\n __data__ = {}\n __products__ = None\n __platforms__ = None\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.products","title":"<code>products()</code>","text":"<p>Returns:</p> Type Description <code>list(str)</code> <p>The product names. An empty list signifies all products.</p> Source code in <code>ciocore/data.py</code> <pre><code>def products():\n\"\"\"\n\n Returns:\n list(str): The product names. An empty list signifies all products.\n \"\"\"\n return __products__\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.set_fixtures_dir","title":"<code>set_fixtures_dir(_)</code>","text":"<p>Deprecated. </p> Source code in <code>ciocore/data.py</code> <pre><code>def set_fixtures_dir(_):\n\"\"\"Deprecated. \"\"\"\n pass\n</code></pre>"},{"location":"apidoc/data/#ciocore.data.platforms","title":"<code>platforms()</code>","text":"<p>The set of platforms that both software and instance types are valid on.</p> <p>Returns:</p> Type Description <code>set</code> <p>A set containing platforms: windows and/or linux.</p> Source code in <code>ciocore/data.py</code> <pre><code>def platforms():\n\"\"\"\n The set of platforms that both software and instance types are valid on.\n\n Returns:\n set: A set containing platforms: windows and/or linux.\n \"\"\"\n return __platforms__\n</code></pre>"},{"location":"apidoc/hardware_set/","title":"ciocore.hardware_set","text":"<p>This module contains the HardwareSet class. </p> <p>It is designed to allow submission tools and other clients that consume instance types to be able to display them in categories. This is particularly useful for UIs that utilize combo-boxes, since they can be orgnaized into a nested structure and displayed in a tree-like fashion.</p>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet","title":"<code> HardwareSet </code>","text":"<p>A class to manage categorized instance types.</p> <p>A HardwareSet encapsulates the instance types available to an account. It accepts a flat list of instance types and builds a nested structure where those instance types exist in categories.</p> <p>It keeps a dictionary of instance types (<code>instance_types</code>) with the name field as key. This allows easy lookup by name.</p> <p>In addition, it keeps the nested structure of categories (<code>categories</code>) that contain the instance types. Each category is a dictionary with keys: <code>label</code>, <code>content</code>, and <code>order</code>.</p> <p><code>content</code> is a list of instance types in the category. The order is used to sort the categories. The order of the instance types within a category is determined by the number of cores and memory.</p> <p>If all instance_types have not been assigned any categories, then the structure is built with two default categories: CPU and GPU.</p> Source code in <code>ciocore/hardware_set.py</code> <pre><code>class HardwareSet(object):\n\"\"\"A class to manage categorized instance types.\n\n A HardwareSet encapsulates the instance types available to an account. It accepts a flat list of instance types and builds a nested structure where those instance types exist in categories.\n\n It keeps a dictionary of instance types (`instance_types`) with the name field as key. This allows easy lookup by name.\n\n In addition, it keeps the nested structure of categories (`categories`) that contain the instance types. Each category is a dictionary with keys: `label`, `content`, and `order`.\n\n `content` is a list of instance types in the category. The order is used to sort the categories. The order of the instance types within a category is determined by the number of cores and memory.\n\n If all instance_types have not been assigned any categories, then the structure is built with two default categories: CPU and GPU.\n \"\"\"\n\n def __init__(self, instance_types):\n\"\"\"Initialize the HardwareSet with a list of instance types.\n Typically, you would access the HardwareSet through the ciocore.data.data() function, which initializes it for you. However, you can also initialize it directly with a list of instance types straight from ciocore.api_client. The difference being that the latter contains all instance types, whereas the former contains only the instance types compatible with the products you have specified, as well as being cached.\n\n Args:\n instance_types (list): A list of instance types.\n\n Returns:\n HardwareSet: The initialized HardwareSet.\n\n Examples:\n ### Initialize with a list of instance types\n >>> from ciocore import api_client\n >>> from ciocore.hardware_set import HardwareSet\n >>> instance_types = api_client.request_instance_types()\n >>> hardware_set = HardwareSet(instance_types)\n <ciocore.hardware_set.HardwareSet object at 0x104c43d30>\n\n ### Initialize implicitly with a list of instance types from ciocore.data (recommended).\n >>> from ciocore import data as coredata\n >>> coredata.init(\"cinema4d\")\n >>> hardware_set = coredata.data()[\"instance_types\"]\n <ciocore.hardware_set.HardwareSet object at 0x104c43ee0>\n\n !!! note\n To avoid repetition, we use the implicit initialization for the examples below.\n \"\"\"\n\n self.instance_types = self._build_unique(instance_types)\n self.categories = self._build_categories()\n self.provider = self._get_provider()\n\n def labels(self):\n\"\"\"Get the list of category labels.\n\n Returns:\n list: A list of category labels.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.labels()\n ['CPU', 'GPU']\n\n \"\"\"\n return [c[\"label\"] for c in self.categories]\n\n def number_of_categories(self):\n\"\"\"Get the number of categories in the data.\n\n Returns:\n int: The number of categories.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.number_of_categories()\n 2\n\n \"\"\"\n return len(self.categories)\n\n def recategorize(self, partitioner):\n\"\"\"Recategorize the instance types.\n\n Args:\n partitioner (function): A function that takes an instance type and returns a list of categories to assign to it. The function should return an empty list if the instance type should not be categorized.\n\n Example:\n # Confirm current categories\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> print(hardware_set.labels()\n ['CPU', 'GPU']\n\n # Recategorize\n >>> hardware_set.recategorize(lambda x: [{'label': 'Low cores', 'order': 10}] if x[\"cores\"] < 16 else [{'label': 'High cores', 'order': 20}])\n >>> print(hardware_set.labels()\n ['Low cores', 'High cores']\n \"\"\"\n for key in self.instance_types:\n self.instance_types[key][\"categories\"] = partitioner(\n self.instance_types[key]\n )\n self.categories = self._build_categories()\n\n def find(self, name, category=None):\n\"\"\"Find an instance type by its name (sku).\n\n Args:\n name (str): The name of the instance type.\n\n Returns:\n dict: The instance type or None if not found.\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find(\"n2-highmem-80\")\n {\n 'cores': 80,\n 'description': '80 core, 640GB Mem',\n 'gpu': None,\n 'memory': '640',\n 'name': 'n2-highmem-80',\n 'operating_system': 'linux',\n 'categories': [\n {'label': 'High cores', 'order': 20}\n ]\n }\n\n \"\"\"\n if not category:\n return self.instance_types.get(name)\n\n return self.find_first(\n lambda x: x[\"name\"] == name\n and category in [c[\"label\"] for c in x[\"categories\"]]\n )\n\n def find_category(self, label):\n\"\"\"Find a category by label.\n\n Args:\n label (str): The label of the category.\n\n Returns:\n dict: The category or None if not found.\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find_category(\"High cores\")\n {\n \"label\": \"Low cores\",\n \"content\": [\n {\n \"cores\": 8,\n \"description\": \"8 core, 52GB Mem\",\n \"gpu\": None,\n \"memory\": \"52\",\n \"name\": \"n1-highmem-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 7.2GB Mem\",\n \"gpu\": None,\n \"memory\": \"7.2\",\n \"name\": \"n1-highcpu-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ],\n \"order\": 10\n }\n \"\"\"\n return next((c for c in self.categories if c[\"label\"] == label), None)\n\n def find_all(self, condition):\n\"\"\"Find all instance types that match a condition.\n\n Args:\n condition (function): A function that takes an instance type and returns True or False.\n\n Returns:\n list: A list of instance types that match the condition.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find_all(lambda x: x[\"gpu\"])\n [\n {\n \"cores\": 4,\n \"description\": \"4 core, 15GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"15\",\n \"name\": \"n1-standard-4-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 30GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"30\",\n \"name\": \"n1-standard-8-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ]\n \"\"\"\n result = []\n for key in self.instance_types:\n if condition(self.instance_types[key]):\n result.append(self.instance_types[key])\n return result\n\n def find_first(self, condition):\n\"\"\"Find the first instance type that matches a condition.\n\n Please see find_all() above for more details. This method is just a convenience wrapper around find_all() that returns the first result or None if not found.\n\n Args:\n condition (function): A function that takes an instance type and returns True or False.\n\n Returns:\n dict: The first instance type that matches the condition or None if not found.\n \"\"\"\n return next(iter(self.find_all(condition)), None)\n\n # DEPRECATED\n def get_model(self, with_misc=False):\n\"\"\"Get the categories structure with renaming ready for some specific widget, such as a Qt Combobox.\n\n Deprecated:\n The get_model() method is deprecated. The `with_misc` parameter is no longer used, which means that this function only serves to rename a few keys. What's more, the init function ensures that every instance type has a category. This function is no longer needed. Submitters that use it will work but should be updated to use the categories structure directly as it minimizes the levels of indirection necessary to work with it.\n \"\"\"\n if with_misc:\n logger.warning(\"with_misc is no longer used\")\n result = []\n for category in self.categories:\n result.append(\n {\n \"label\": category[\"label\"],\n \"content\": [\n {\"label\": k[\"description\"], \"value\": k[\"name\"]}\n for k in category[\"content\"]\n ],\n }\n )\n\n return result\n\n # PRIVATE METHODS\n @classmethod\n def _build_unique(cls, instance_types):\n\"\"\"Build a dictionary of instance types using the name field as key. This allows fast lookup by name.\n\n Args:\n instance_types (list): A list of instance types.\n\n Returns:\n dict: A dictionary of instance types with the name field as key.\n \"\"\"\n\n instance_types = cls._rewrite_descriptions(instance_types)\n categories = [\n category\n for it in instance_types\n for category in (it.get(\"categories\") or [])\n ]\n result = {}\n for it in instance_types:\n is_gpu = it.get(\"gpu\", False)\n if categories:\n if it.get(\"categories\") in [[], None]:\n continue\n else:\n # make our own categories GPU/CPU\n it[\"categories\"] = (\n [{\"label\": \"GPU\", \"order\": 2}]\n if is_gpu\n else [{\"label\": \"CPU\", \"order\": 1}]\n )\n result[it[\"name\"]] = it\n\n return result\n\n\n\n\n @classmethod\n def _rewrite_descriptions(cls, instance_types):\n\"\"\"Rewrite the descriptions of the instance types.\n\n If there are both OS types, then the descriptions are prefixed with the OS type.\n\n Args:\n instance_types (list): A list of instance types.\n\n Returns:\n list: A list of instance types with rewritten descriptions.\n \"\"\"\n if not instance_types:\n return instance_types\n\n first_os = instance_types[0][\"operating_system\"]\n dual_platforms = next((it for it in instance_types if it[\"operating_system\"] != first_os), False)\n\n if dual_platforms:\n for it in instance_types:\n flat_dict = flatten_dict(it)\n is_gpu = \"gpu_count\" in flat_dict\n if is_gpu:\n it[\"description\"] = DESCRIPTION_TEMPLATE_OS[\"gpu\"].format(flat_dict)\n else:\n it[\"description\"] = DESCRIPTION_TEMPLATE_OS[\"cpu\"].format(flat_dict)\n else:\n for it in instance_types:\n flat_dict = flatten_dict(it)\n is_gpu = \"gpu_count\" in flat_dict\n if is_gpu:\n it[\"description\"] = DESCRIPTION_TEMPLATE[\"gpu\"].format(flat_dict)\n else:\n it[\"description\"] = DESCRIPTION_TEMPLATE[\"cpu\"].format(*flat_dict)\n\n return instance_types\n\n def _build_categories(self):\n\"\"\"Build a sorted list of categories where each category contains a sorted list of machines.\n\n Returns:\n list: A list of categories where each category is a dictionary with keys: `label`, `content`, and `order`.\n \"\"\"\n\n dikt = {}\n for key in self.instance_types:\n it = self.instance_types[key]\n categories = it[\"categories\"]\n for category in categories:\n label = category[\"label\"]\n if label not in dikt:\n dikt[label] = {\n \"label\": label,\n \"content\": [],\n \"order\": category[\"order\"],\n }\n dikt[label][\"content\"].append(it)\n\n result = []\n for label in dikt:\n category = dikt[label]\n category[\"content\"].sort(key=lambda k: (k[\"cores\"], k[\"memory\"]))\n result.append(category)\n return sorted(result, key=lambda k: k[\"order\"])\n\n def _get_provider(self):\n\"\"\"Get the provider from the first instance type.\n\n Returns:\n str: The provider.\n \"\"\"\n first_name = next(iter(self.instance_types))\n if not first_name:\n return None\n if first_name.startswith(\"cw-\"):\n return \"cw\"\n if \".\" in first_name:\n return \"aws\"\n return \"gcp\"\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.__init__","title":"<code>__init__(self, instance_types)</code> <code>special</code>","text":"<p>Initialize the HardwareSet with a list of instance types. Typically, you would access the HardwareSet through the ciocore.data.data() function, which initializes it for you. However, you can also initialize it directly with a list of instance types straight from ciocore.api_client. The difference being that the latter contains all instance types, whereas the former contains only the instance types compatible with the products you have specified, as well as being cached.</p> <p>Parameters:</p> Name Type Description Default <code>instance_types</code> <code>list</code> <p>A list of instance types.</p> required <p>Returns:</p> Type Description <code>HardwareSet</code> <p>The initialized HardwareSet.</p> <p>Examples:</p>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.__init__--initialize-with-a-list-of-instance-types","title":"Initialize with a list of instance types","text":"<pre><code>>>> from ciocore import api_client\n>>> from ciocore.hardware_set import HardwareSet\n>>> instance_types = api_client.request_instance_types()\n>>> hardware_set = HardwareSet(instance_types)\n<ciocore.hardware_set.HardwareSet object at 0x104c43d30>\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.__init__--initialize-implicitly-with-a-list-of-instance-types-from-ciocoredata-recommended","title":"Initialize implicitly with a list of instance types from ciocore.data (recommended).","text":"<pre><code>>>> from ciocore import data as coredata\n>>> coredata.init(\"cinema4d\")\n>>> hardware_set = coredata.data()[\"instance_types\"]\n<ciocore.hardware_set.HardwareSet object at 0x104c43ee0>\n</code></pre> <p>Note</p> <p>To avoid repetition, we use the implicit initialization for the examples below.</p> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def __init__(self, instance_types):\n\"\"\"Initialize the HardwareSet with a list of instance types.\n Typically, you would access the HardwareSet through the ciocore.data.data() function, which initializes it for you. However, you can also initialize it directly with a list of instance types straight from ciocore.api_client. The difference being that the latter contains all instance types, whereas the former contains only the instance types compatible with the products you have specified, as well as being cached.\n\n Args:\n instance_types (list): A list of instance types.\n\n Returns:\n HardwareSet: The initialized HardwareSet.\n\n Examples:\n ### Initialize with a list of instance types\n >>> from ciocore import api_client\n >>> from ciocore.hardware_set import HardwareSet\n >>> instance_types = api_client.request_instance_types()\n >>> hardware_set = HardwareSet(instance_types)\n <ciocore.hardware_set.HardwareSet object at 0x104c43d30>\n\n ### Initialize implicitly with a list of instance types from ciocore.data (recommended).\n >>> from ciocore import data as coredata\n >>> coredata.init(\"cinema4d\")\n >>> hardware_set = coredata.data()[\"instance_types\"]\n <ciocore.hardware_set.HardwareSet object at 0x104c43ee0>\n\n !!! note\n To avoid repetition, we use the implicit initialization for the examples below.\n \"\"\"\n\n self.instance_types = self._build_unique(instance_types)\n self.categories = self._build_categories()\n self.provider = self._get_provider()\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.labels","title":"<code>labels(self)</code>","text":"<p>Get the list of category labels.</p> <p>Returns:</p> Type Description <code>list</code> <p>A list of category labels.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.labels()\n['CPU', 'GPU']\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def labels(self):\n\"\"\"Get the list of category labels.\n\n Returns:\n list: A list of category labels.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.labels()\n ['CPU', 'GPU']\n\n \"\"\"\n return [c[\"label\"] for c in self.categories]\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.number_of_categories","title":"<code>number_of_categories(self)</code>","text":"<p>Get the number of categories in the data.</p> <p>Returns:</p> Type Description <code>int</code> <p>The number of categories.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.number_of_categories()\n2\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def number_of_categories(self):\n\"\"\"Get the number of categories in the data.\n\n Returns:\n int: The number of categories.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.number_of_categories()\n 2\n\n \"\"\"\n return len(self.categories)\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.recategorize","title":"<code>recategorize(self, partitioner)</code>","text":"<p>Recategorize the instance types.</p> <p>Parameters:</p> Name Type Description Default <code>partitioner</code> <code>function</code> <p>A function that takes an instance type and returns a list of categories to assign to it. The function should return an empty list if the instance type should not be categorized.</p> required <p>Examples:</p>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.recategorize--confirm-current-categories","title":"Confirm current categories","text":"<pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> print(hardware_set.labels()\n['CPU', 'GPU']\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.recategorize--recategorize","title":"Recategorize","text":"<pre><code>>>> hardware_set.recategorize(lambda x: [{'label': 'Low cores', 'order': 10}] if x[\"cores\"] < 16 else [{'label': 'High cores', 'order': 20}])\n>>> print(hardware_set.labels()\n['Low cores', 'High cores']\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def recategorize(self, partitioner):\n\"\"\"Recategorize the instance types.\n\n Args:\n partitioner (function): A function that takes an instance type and returns a list of categories to assign to it. The function should return an empty list if the instance type should not be categorized.\n\n Example:\n # Confirm current categories\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> print(hardware_set.labels()\n ['CPU', 'GPU']\n\n # Recategorize\n >>> hardware_set.recategorize(lambda x: [{'label': 'Low cores', 'order': 10}] if x[\"cores\"] < 16 else [{'label': 'High cores', 'order': 20}])\n >>> print(hardware_set.labels()\n ['Low cores', 'High cores']\n \"\"\"\n for key in self.instance_types:\n self.instance_types[key][\"categories\"] = partitioner(\n self.instance_types[key]\n )\n self.categories = self._build_categories()\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.find","title":"<code>find(self, name, category=None)</code>","text":"<p>Find an instance type by its name (sku).</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>The name of the instance type.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>The instance type or None if not found.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.find(\"n2-highmem-80\")\n{\n 'cores': 80,\n 'description': '80 core, 640GB Mem',\n 'gpu': None,\n 'memory': '640',\n 'name': 'n2-highmem-80',\n 'operating_system': 'linux',\n 'categories': [\n {'label': 'High cores', 'order': 20}\n ]\n}\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def find(self, name, category=None):\n\"\"\"Find an instance type by its name (sku).\n\n Args:\n name (str): The name of the instance type.\n\n Returns:\n dict: The instance type or None if not found.\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find(\"n2-highmem-80\")\n {\n 'cores': 80,\n 'description': '80 core, 640GB Mem',\n 'gpu': None,\n 'memory': '640',\n 'name': 'n2-highmem-80',\n 'operating_system': 'linux',\n 'categories': [\n {'label': 'High cores', 'order': 20}\n ]\n }\n\n \"\"\"\n if not category:\n return self.instance_types.get(name)\n\n return self.find_first(\n lambda x: x[\"name\"] == name\n and category in [c[\"label\"] for c in x[\"categories\"]]\n )\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.find_category","title":"<code>find_category(self, label)</code>","text":"<p>Find a category by label.</p> <p>Parameters:</p> Name Type Description Default <code>label</code> <code>str</code> <p>The label of the category.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>The category or None if not found.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.find_category(\"High cores\")\n{\n \"label\": \"Low cores\",\n \"content\": [\n {\n \"cores\": 8,\n \"description\": \"8 core, 52GB Mem\",\n \"gpu\": None,\n \"memory\": \"52\",\n \"name\": \"n1-highmem-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 7.2GB Mem\",\n \"gpu\": None,\n \"memory\": \"7.2\",\n \"name\": \"n1-highcpu-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ],\n \"order\": 10\n}\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def find_category(self, label):\n\"\"\"Find a category by label.\n\n Args:\n label (str): The label of the category.\n\n Returns:\n dict: The category or None if not found.\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find_category(\"High cores\")\n {\n \"label\": \"Low cores\",\n \"content\": [\n {\n \"cores\": 8,\n \"description\": \"8 core, 52GB Mem\",\n \"gpu\": None,\n \"memory\": \"52\",\n \"name\": \"n1-highmem-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 7.2GB Mem\",\n \"gpu\": None,\n \"memory\": \"7.2\",\n \"name\": \"n1-highcpu-8\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ],\n \"order\": 10\n }\n \"\"\"\n return next((c for c in self.categories if c[\"label\"] == label), None)\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.find_all","title":"<code>find_all(self, condition)</code>","text":"<p>Find all instance types that match a condition.</p> <p>Parameters:</p> Name Type Description Default <code>condition</code> <code>function</code> <p>A function that takes an instance type and returns True or False.</p> required <p>Returns:</p> Type Description <code>list</code> <p>A list of instance types that match the condition.</p> <p>Examples:</p> <pre><code>>>> from ciocore import data as coredata\n>>> coredata.init()\n>>> hardware_set = coredata.data()[\"instance_types\"]\n>>> hardware_set.find_all(lambda x: x[\"gpu\"])\n[\n {\n \"cores\": 4,\n \"description\": \"4 core, 15GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"15\",\n \"name\": \"n1-standard-4-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 30GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"30\",\n \"name\": \"n1-standard-8-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n]\n</code></pre> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def find_all(self, condition):\n\"\"\"Find all instance types that match a condition.\n\n Args:\n condition (function): A function that takes an instance type and returns True or False.\n\n Returns:\n list: A list of instance types that match the condition.\n\n Example:\n >>> from ciocore import data as coredata\n >>> coredata.init()\n >>> hardware_set = coredata.data()[\"instance_types\"]\n >>> hardware_set.find_all(lambda x: x[\"gpu\"])\n [\n {\n \"cores\": 4,\n \"description\": \"4 core, 15GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"15\",\n \"name\": \"n1-standard-4-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n {\n \"cores\": 8,\n \"description\": \"8 core, 30GB Mem (1 T4 Tensor GPU with 16GB Mem)\",\n \"gpu\": {\n \"gpu_architecture\": \"NVIDIA Turing\",\n \"gpu_count\": 1,\n \"gpu_cuda_cores\": 2560,\n \"gpu_memory\": \"16\",\n \"gpu_model\": \"T4 Tensor\",\n \"gpu_rt_cores\": 0,\n \"gpu_tensor_cores\": 0,\n \"total_gpu_cuda_cores\": 2560,\n \"total_gpu_memory\": \"16\",\n \"total_gpu_rt_cores\": 0,\n \"total_gpu_tensor_cores\": 0,\n },\n \"memory\": \"30\",\n \"name\": \"n1-standard-8-t4-1\",\n \"operating_system\": \"linux\",\n \"categories\": [{\"label\": \"Low cores\", \"order\": 10}],\n },\n ...\n ]\n \"\"\"\n result = []\n for key in self.instance_types:\n if condition(self.instance_types[key]):\n result.append(self.instance_types[key])\n return result\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.find_first","title":"<code>find_first(self, condition)</code>","text":"<p>Find the first instance type that matches a condition.</p> <p>Please see find_all() above for more details. This method is just a convenience wrapper around find_all() that returns the first result or None if not found.</p> <p>Parameters:</p> Name Type Description Default <code>condition</code> <code>function</code> <p>A function that takes an instance type and returns True or False.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>The first instance type that matches the condition or None if not found.</p> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def find_first(self, condition):\n\"\"\"Find the first instance type that matches a condition.\n\n Please see find_all() above for more details. This method is just a convenience wrapper around find_all() that returns the first result or None if not found.\n\n Args:\n condition (function): A function that takes an instance type and returns True or False.\n\n Returns:\n dict: The first instance type that matches the condition or None if not found.\n \"\"\"\n return next(iter(self.find_all(condition)), None)\n</code></pre>"},{"location":"apidoc/hardware_set/#ciocore.hardware_set.HardwareSet.get_model","title":"<code>get_model(self, with_misc=False)</code>","text":"<p>Get the categories structure with renaming ready for some specific widget, such as a Qt Combobox.</p> <p>Deprecated</p> <p>The get_model() method is deprecated. The <code>with_misc</code> parameter is no longer used, which means that this function only serves to rename a few keys. What's more, the init function ensures that every instance type has a category. This function is no longer needed. Submitters that use it will work but should be updated to use the categories structure directly as it minimizes the levels of indirection necessary to work with it.</p> Source code in <code>ciocore/hardware_set.py</code> <pre><code>def get_model(self, with_misc=False):\n\"\"\"Get the categories structure with renaming ready for some specific widget, such as a Qt Combobox.\n\n Deprecated:\n The get_model() method is deprecated. The `with_misc` parameter is no longer used, which means that this function only serves to rename a few keys. What's more, the init function ensures that every instance type has a category. This function is no longer needed. Submitters that use it will work but should be updated to use the categories structure directly as it minimizes the levels of indirection necessary to work with it.\n \"\"\"\n if with_misc:\n logger.warning(\"with_misc is no longer used\")\n result = []\n for category in self.categories:\n result.append(\n {\n \"label\": category[\"label\"],\n \"content\": [\n {\"label\": k[\"description\"], \"value\": k[\"name\"]}\n for k in category[\"content\"]\n ],\n }\n )\n\n return result\n</code></pre>"},{"location":"apidoc/package_environment/","title":"ciocore.package_environment","text":"<p>Manage environment variables for both Windows and Linux render nodes.</p>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment","title":"<code> PackageEnvironment </code>","text":"Source code in <code>ciocore/package_environment.py</code> <pre><code>class PackageEnvironment(object):\n\n\n def __init__(self, env_list=None, platform=None):\n\"\"\"\n Encapsulate a list of environment variables.\n\n Typically, one would initialize a PackageEnvironment with a package, and then modify by adding more packages or lists of variables. Extra variables can be added by the customer, or programmatically such as during asset scraping.\n\n Args:\n env_list (object\|list): An object that provides a list of dictionaries with properties: `name`, `value`, and `merge_policy`.\n platform (str): If the env_list is a regular list, then this is required.\n\n Args are delegated to [extend()](/package_environment/#ciocore.package_environment.PackageEnvironment.extend).\n \"\"\"\n self.platform = None\n self._env = {}\n\n self.extend(env_list, platform)\n\n\n def extend(self, env_list, platform=None):\n\"\"\"\n Extend the Package environment with the given variable specifications.\n\n Args:\n env_list (object\|list): Either:\n A list of dictionaries with properties: `name`, `value`, and `merge_policy`.\n * An object with an `environment` key that contains a list of the dictionaries described above. The latter is the structure of a package. Therefore we can initialize or extend a PackageEnvironment with a package.\n platform (str): Defaults to `None`. If env_list is a package, then the platform is taken from the package and the `platform` keyword is ignored. If env_list is a list, then if this is the first add, a platform should be specified, otherwise it will default to linux.\n\n The first time data is added to a PackageEnvironment, the platform is set in stone. Subsequent `adds` that try to change the platform are considered an error.\n\n Each variable to be added specifies a merge_policy: `append`, `prepend`, or `exclusive` `append` and `prepend` can be thought of as lists= types. Once an individual variable has been initialized as a list, it can't be changed to `exclusive`. This means:\n\n 1. It's not possible to overwrite variables that have been added as `append` or `prepend`.\n 2. Exclusive variables are always overwritten by subsequent adds.\n\n Raises:\n ValueError: Either an attempt to change the platform once initialized, or an invalid merge policy.\n\n\n Example:\n >>> from ciocore import api_client, package_tree, package_environment\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> one_dcc_name = pt.supported_host_names()[0]\n cinema4d 21.209.RB305619 linux\n\n >>> pkg = pt.find_by_name(one_dcc_name)\n >>> pe = package_environment.PackageEnvironment(pkg)\n >>> print(dict(pe))\n {\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n }\n\n >>> extra_env = [\n {\"name\":\"PATH\", \"value\": \"/my/custom/scripts\", \"merge_policy\":\"append\"},\n {\"name\":\"DEBUG_MODE\", \"value\": \"1\", \"merge_policy\":\"exclusive\"}\n ]\n >>> pe.extend(extra_env)\n >>> print(dict(pe))\n {\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin:/my/custom/scripts\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n \"DEBUG_MODE\": \"1\",\n }\n \"\"\"\n\n if not env_list:\n return\n\n try:\n others = env_list[\"environment\"]\n requested_platform = env_list.get(\"platform\")\n except TypeError:\n others = env_list\n requested_platform = platform\n\n if not self.platform:\n self.platform = requested_platform or \"linux\"\n elif requested_platform and requested_platform != self.platform:\n raise ValueError(\"Can't change platform once initialized.\")\n\n for var in others:\n name = var[\"name\"]\n value = var[\"value\"]\n policy = var[\"merge_policy\"]\n if policy not in [\"append\", \"prepend\", \"exclusive\"]:\n raise ValueError(\"Unexpected merge policy: %s\" % policy)\n\n if policy == \"append\":\n self._append(name, value)\n elif policy == \"prepend\":\n self._prepend(name, value)\n else:\n self._set(name, value)\n\n\n def _set(self, name, value):\n\"\"\"Set the value of an exclusive variable.\n\n Can be overwritten by subsequent adds.\n\n It is an error if the variable has already been declared with policy=append.\n \"\"\"\n if self._env.get(name) and isinstance(self._env[name], list):\n raise ValueError(\n \"Can't change merge policy for '{}' from 'append' to 'exclusive'.\".format(name)\n )\n self._env[name] = value\n\n def _append(self, name, value):\n\"\"\"Set the value of an append variable.\n\n Can be appended to with subsequent adds.\n\n It is an error if the variable has already been declared with policy=exclusive.\n \"\"\"\n if self._env.get(name):\n if not isinstance(self._env[name], list):\n raise ValueError(\n \"Can't change merge policy for '{}' from 'exclusive' to 'append'.\".format(name)\n )\n else:\n self._env[name] = []\n self._env[name].append(value)\n\n def _prepend(self, name, value):\n\"\"\"Set the value of an append/prepend variable.\n\n Can be appended to with subsequent adds.\n\n It is an error if the variable has already been declared with policy=exclusive.\n \"\"\"\n if self._env.get(name):\n if not isinstance(self._env[name], list):\n raise ValueError(\n \"Can't change merge policy for '{}' from 'exclusive' to 'prepend'.\".format(name)\n )\n else:\n self._env[name] = []\n self._env[name].insert(0, value)\n\n def __iter__(self):\n\"\"\"Cast the object as a dict.\"\"\"\n sep = \";\" if self.platform == \"windows\" else \":\"\n for key in self._env:\n var = self._env[key]\n if isinstance(var, list):\n yield key, sep.join(var)\n else:\n yield key, var\n\n def __getitem__(self, key):\n\"\"\"Allow access by key.\"\"\"\n sep = \";\" if self.platform == \"windows\" else \":\"\n var = self._env.__getitem__(key)\n if isinstance(var, list):\n return sep.join(var)\n return var\n</code></pre>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment.__init__","title":"<code>__init__(self, env_list=None, platform=None)</code> <code>special</code>","text":"<p>Encapsulate a list of environment variables.</p> <p>Typically, one would initialize a PackageEnvironment with a package, and then modify by adding more packages or lists of variables. Extra variables can be added by the customer, or programmatically such as during asset scraping.</p> <p>Parameters:</p> Name Type Description Default <code>env_list</code> <code>object\|list</code> <p>An object that provides a list of dictionaries with properties: <code>name</code>, <code>value</code>, and <code>merge_policy</code>.</p> <code>None</code> <code>platform</code> <code>str</code> <p>If the env_list is a regular list, then this is required.</p> <code>None</code> <p>Args are delegated to extend().</p> Source code in <code>ciocore/package_environment.py</code> <pre><code>def __init__(self, env_list=None, platform=None):\n\"\"\"\n Encapsulate a list of environment variables.\n\n Typically, one would initialize a PackageEnvironment with a package, and then modify by adding more packages or lists of variables. Extra variables can be added by the customer, or programmatically such as during asset scraping.\n\n Args:\n env_list (object\|list): An object that provides a list of dictionaries with properties: `name`, `value`, and `merge_policy`.\n platform (str): If the env_list is a regular list, then this is required.\n\n Args are delegated to [extend()](/package_environment/#ciocore.package_environment.PackageEnvironment.extend).\n \"\"\"\n self.platform = None\n self._env = {}\n\n self.extend(env_list, platform)\n</code></pre>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment.extend","title":"<code>extend(self, env_list, platform=None)</code>","text":"<p>Extend the Package environment with the given variable specifications.</p> <p>Parameters:</p> Name Type Description Default <code>env_list</code> <code>object\|list</code> <ul> <li>A list of dictionaries with properties: <code>name</code>, <code>value</code>, and <code>merge_policy</code>.</li> <li>An object with an <code>environment</code> key that contains a list of the dictionaries described above. The latter is the structure of a package. Therefore we can initialize or extend a PackageEnvironment with a package.</li> </ul> required <code>platform</code> <code>str</code> <p>Defaults to <code>None</code>. If env_list is a package, then the platform is taken from the package and the <code>platform</code> keyword is ignored. If env_list is a list, then if this is the first add, a platform should be specified, otherwise it will default to linux.</p> <code>None</code> <p>The first time data is added to a PackageEnvironment, the platform is set in stone. Subsequent <code>adds</code> that try to change the platform are considered an error.</p> <p>Each variable to be added specifies a merge_policy: <code>append</code>, <code>prepend</code>, or <code>exclusive</code> <code>append</code> and <code>prepend</code> can be thought of as lists= types. Once an individual variable has been initialized as a list, it can't be changed to <code>exclusive</code>. This means:</p> <ol> <li>It's not possible to overwrite variables that have been added as <code>append</code> or <code>prepend</code>.</li> <li>Exclusive variables are always overwritten by subsequent adds.</li> </ol> <p>Exceptions:</p> Type Description <code>ValueError</code> <p>Either an attempt to change the platform once initialized, or an invalid merge policy.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree, package_environment\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> one_dcc_name = pt.supported_host_names()[0]\ncinema4d 21.209.RB305619 linux\n</code></pre> <pre><code>>>> pkg = pt.find_by_name(one_dcc_name)\n>>> pe = package_environment.PackageEnvironment(pkg)\n>>> print(dict(pe))\n{\n\"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin\",\n\"g_licenseServerRLM\": \"conductor-rlm:6112\",\n\"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n\"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n}\n</code></pre> <pre><code>>>> extra_env = [\n {\"name\":\"PATH\", \"value\": \"/my/custom/scripts\", \"merge_policy\":\"append\"},\n {\"name\":\"DEBUG_MODE\", \"value\": \"1\", \"merge_policy\":\"exclusive\"}\n]\n>>> pe.extend(extra_env)\n>>> print(dict(pe))\n{\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin:/my/custom/scripts\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n \"DEBUG_MODE\": \"1\",\n}\n</code></pre> Source code in <code>ciocore/package_environment.py</code> <pre><code>def extend(self, env_list, platform=None):\n\"\"\"\n Extend the Package environment with the given variable specifications.\n\n Args:\n env_list (object\|list): Either:\n * A list of dictionaries with properties: `name`, `value`, and `merge_policy`.\n * An object with an `environment` key that contains a list of the dictionaries described above. The latter is the structure of a package. Therefore we can initialize or extend a PackageEnvironment with a package.\n platform (str): Defaults to `None`. If env_list is a package, then the platform is taken from the package and the `platform` keyword is ignored. If env_list is a list, then if this is the first add, a platform should be specified, otherwise it will default to linux.\n\n The first time data is added to a PackageEnvironment, the platform is set in stone. Subsequent `adds` that try to change the platform are considered an error.\n\n Each variable to be added specifies a merge_policy: `append`, `prepend`, or `exclusive` `append` and `prepend` can be thought of as lists= types. Once an individual variable has been initialized as a list, it can't be changed to `exclusive`. This means:\n\n 1. It's not possible to overwrite variables that have been added as `append` or `prepend`.\n 2. Exclusive variables are always overwritten by subsequent adds.\n\n Raises:\n ValueError: Either an attempt to change the platform once initialized, or an invalid merge policy.\n\n\n Example:\n >>> from ciocore import api_client, package_tree, package_environment\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> one_dcc_name = pt.supported_host_names()[0]\n cinema4d 21.209.RB305619 linux\n\n >>> pkg = pt.find_by_name(one_dcc_name)\n >>> pe = package_environment.PackageEnvironment(pkg)\n >>> print(dict(pe))\n {\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n }\n\n >>> extra_env = [\n {\"name\":\"PATH\", \"value\": \"/my/custom/scripts\", \"merge_policy\":\"append\"},\n {\"name\":\"DEBUG_MODE\", \"value\": \"1\", \"merge_policy\":\"exclusive\"}\n ]\n >>> pe.extend(extra_env)\n >>> print(dict(pe))\n {\n \"PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin:/my/custom/scripts\",\n \"g_licenseServerRLM\": \"conductor-rlm:6112\",\n \"LD_LIBRARY_PATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64:/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/embree.module/libs/linux64\",\n \"PYTHONPATH\": \"/opt/maxon/cinema4d/21/cinema4d21.209vRB305619/bin/resource/modules/python/libs/linux64/python.linux64.framework/lib64/python2.7/lib-dynload\",\n \"DEBUG_MODE\": \"1\",\n }\n \"\"\"\n\n if not env_list:\n return\n\n try:\n others = env_list[\"environment\"]\n requested_platform = env_list.get(\"platform\")\n except TypeError:\n others = env_list\n requested_platform = platform\n\n if not self.platform:\n self.platform = requested_platform or \"linux\"\n elif requested_platform and requested_platform != self.platform:\n raise ValueError(\"Can't change platform once initialized.\")\n\n for var in others:\n name = var[\"name\"]\n value = var[\"value\"]\n policy = var[\"merge_policy\"]\n if policy not in [\"append\", \"prepend\", \"exclusive\"]:\n raise ValueError(\"Unexpected merge policy: %s\" % policy)\n\n if policy == \"append\":\n self._append(name, value)\n elif policy == \"prepend\":\n self._prepend(name, value)\n else:\n self._set(name, value)\n</code></pre>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment.__iter__","title":"<code>__iter__(self)</code> <code>special</code>","text":"<p>Cast the object as a dict.</p> Source code in <code>ciocore/package_environment.py</code> <pre><code>def __iter__(self):\n\"\"\"Cast the object as a dict.\"\"\"\n sep = \";\" if self.platform == \"windows\" else \":\"\n for key in self._env:\n var = self._env[key]\n if isinstance(var, list):\n yield key, sep.join(var)\n else:\n yield key, var\n</code></pre>"},{"location":"apidoc/package_environment/#ciocore.package_environment.PackageEnvironment.__getitem__","title":"<code>__getitem__(self, key)</code> <code>special</code>","text":"<p>Allow access by key.</p> Source code in <code>ciocore/package_environment.py</code> <pre><code>def __getitem__(self, key):\n\"\"\"Allow access by key.\"\"\"\n sep = \";\" if self.platform == \"windows\" else \":\"\n var = self._env.__getitem__(key)\n if isinstance(var, list):\n return sep.join(var)\n return var\n</code></pre>"},{"location":"apidoc/package_tree/","title":"ciocore.package_tree","text":"<p>A class to provide available packages as DAG structure. In reality however, the structure is just two levels deep: hosts and plugins. </p> <ul> <li>DCCs such as Maya and Cinema4D are top-level host packages. </li> <li>Renderers and other plugins are children of those hosts.</li> </ul> <p>Methods are provided to traverse the tree to find packages by name, version, platform and so on. If you are writing submission tools there's no need to create a Package tree directly. It is recommended to use the singleton module: ciocore.data.</p> <p>The only functions you should need from this module are:</p> <ul> <li>supported_host_names()</li> <li>supported_plugins()</li> </ul>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree","title":"<code> PackageTree </code>","text":"Source code in <code>ciocore/package_tree.py</code> <pre><code>class PackageTree(object):\n def __init__(self, packages, host_products, kwargs):\n\"\"\"Build the tree with a list of packages.\n\n Args:\n packages (list): List of packages direct from the [Conductor packages endpoint](https://dashboard.conductortech.com/api/v1/ee/packages).\n\n host_products: Filter the tree to contain only top-level host packages of products specified in this list and their plugins. If there are no host_products specified, and the product keyword is omitted, the tree contains all packages.\n\n Keyword Args:\n product (str): Build the tree from versions of a single product and its compatible plugins. Defaults to `None`, in which case the tree is built from all packages. It is an error to specify both host_products and product. If a nonexistent product is given, the PackageTree is empty. By specifying `product`, you can build the object based on a single plugin product.\n platforms (set): Build the tree from versions for a specific platform. Defaults to the set `{\"linux\", \"windows\"}`.\n\n Raises:\n KeyError: An invalid platform was provided.\n ValueError: Cannot choose both product and host_products.\n\n Example:\n >>> from ciocore import api_client, package_tree\n # Request packages as a flat list from Conductor.\n >>> packages = api_client.request_software_packages()\n # Build tree of dependencies from packages list\n >>> pt = package_tree.PackageTree(packages, \"cinema4d\", \"maya-io\")\n >>> for path in pt.to_path_list():\n >>> print(path)\n cinema4d 22.118.RB320081 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\n maya-io 2022.SP3 linux\n \"\"\"\n\n\n platforms = kwargs.get(\"platforms\",PLATFORMS)\n product=kwargs.get(\"product\")\n\n unknown_platforms = set(platforms) - PLATFORMS\n if unknown_platforms:\n raise KeyError(\"Unrecognized platform {}\".format(\" \".join(unknown_platforms)))\n\n if product and host_products:\n raise ValueError(\"You cannot choose both product and host_products.\")\n\n packages = [_clean_package(p) for p in packages if p[\"platform\"] in platforms]\n\n root_ids = [] \n if product:\n root_ids = [p[\"package_id\"] for p in packages if p[\"product\"] == product]\n else:\n for p in packages:\n if not p[\"plugin_host_product\"]:\n if p[\"product\"] in host_products or not host_products:\n root_ids.append(p[\"package_id\"])\n\n self._tree = _build_tree(packages, {\"children\": [], \"plugins\": root_ids})\n\n def supported_host_names(self):\n\"\"\"\n All host names from the software tree.\n\n These names can be used to populate a dropdown menu. Then a single selection from that menu\n can be used to retrieve the complete package in order to generate an environment dictionary\n and get package IDs.\n\n Returns:\n list(str): Fully qualified DCC hosts of the form: `product version platform`.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.supported_host_names()\n cinema4d 21.209.RB305619 linux\n cinema4d 22.116.RB316423 linux\n cinema4d 22.118.RB320081 linux\n cinema4d 23.110.RB330286 linux\n cinema4d 24.111 linux\n cinema4d 24.111 windows\n \"\"\"\n\n paths = []\n for pkg in self._tree[\"children\"]:\n paths.append(to_name(pkg))\n return sorted(paths)\n\n def supported_plugins(self, host):\n\"\"\"\n Find the plugins that are children of the given DCC host.\n\n The result does not contain platform information since we assume that plugins are compatible with the DCC host that was used to request them.\n\n Args:\n host (str): Name of the DCC host, typically one of the entries returned by [supported_host_names()](/package_tree/#ciocore.package_tree.PackageTree.supported_host_names).\n\n Returns:\n list(dict): Each entry contains a plugin product and a list of versions.\n\n Example:\n >>> from ciocore import api_client, package_tree \n >>> packages = api_client.request_software_packages() \n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\") \n >>> name = pt.supported_host_names()[0]\n >>> pt.supported_plugins(name)\n [\n {\n \"plugin\": \"arnold-cinema4d\",\n \"versions\": [\n \"3.3.2.100\",\n \"3.3.3.0\"\n ]\n },\n {\n \"plugin\": \"redshift-cinema4d\",\n \"versions\": [\n \"2.6.54\",\n \"2.6.56\",\n \"3.0.21\",\n \"3.0.22\",\n ],\n },\n ]\n \"\"\"\n\n try:\n subtree = self.find_by_name(host)\n plugin_versions = _to_path_list(subtree)\n except TypeError:\n return []\n\n if not plugin_versions:\n return []\n\n plugin_dict = {}\n for plugin, version, _ in [pv.split(\" \") for pv in plugin_versions]:\n if plugin not in plugin_dict:\n plugin_dict[plugin] = []\n plugin_dict[plugin].append(version)\n\n # convert to list so it can be sorted\n plugins = []\n for key in plugin_dict:\n plugins.append({\"plugin\": key, \"versions\": sorted(plugin_dict[key])})\n\n return sorted(plugins, key=lambda k: k[\"plugin\"])\n\n def find_by_name(self, name, limit=None):\n\"\"\"\n Search the tree for a product with the given name.\n\n Args:\n name (str): The name constructed from the package using to_name(). It must be an exact match with product, version, and platform. For example: `maya 2018.0 windows`\n\n Keyword Args:\n limit (int): Limit the search depth. Defaults to `None`.\n\n Returns:\n object: The package that matches.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.find_by_name(\"redshift-cinema4d 3.0.64 linux\")\n {\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '64',\n 'vendor': 'maxon',\n 'children': [],\n ...\n }\n \"\"\"\n\n return _find_by_name(self._tree, name, limit, 0)\n\n def find_by_path(self, path):\n\"\"\"\n Find the package uniquely described by the given path.\n\n The path is of the form returned by the to_path_list() method.\n\n Args:\n path (str): The path\n\n Returns:\n object: The package or None if no package exists with the given path.\n\n Example:\n >>> from ciocore import api_client, package_tree, package_environment\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.find_by_path(\"cinema4d 24.111 linux/redshift-cinema4d 3.0.62 linux\")\n {\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '62',\n 'vendor': 'maxon',\n 'children': [],\n 'plugin_host_version': \"24\",\n ...\n }\n \"\"\"\n return _find_by_path(self._tree, path)\n\n\n def to_path_list(self, name=None):\n\"\"\"\n Get paths to all nodes.\n\n This is useful for populating a chooser to choose packages fully qualified by path.\n Houdini's tree widget, for example, takes the below format unchanged and generates the\n appropriate UI.\n\n Args:\n name (str): Get paths below the tree represented by the name. Defaults to None (root node).\n\n Returns:\n list(str): Paths to all nodes in the tree.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.to_path_list()\n cinema4d 22.118.RB320081 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.22 linux\n cinema4d 22.118.RB320081 linux/arnold-cinema4d 3.3.2.100 linux\n ...\n\n >>> pt.to_path_list(name=\"cinema4d 24.111 linux\")\n redshift-cinema4d 3.0.57 linux\n redshift-cinema4d 3.0.62 linux\n redshift-cinema4d 3.0.45 linux\n redshift-cinema4d 3.0.64 linux\n \"\"\"\n if name:\n subtree = self.find_by_name(name)\n return _to_path_list(subtree)\n return _to_path_list(self._tree)\n\n def platforms(self):\n\"\"\"\n Get the platforms represented by packages in the tree.\n\n Returns:\n set: The set of platforms.\n \"\"\"\n\n # No need to recurse. Plugins are assumed to be compatible with the host.\n return set([host[\"platform\"] for host in self._tree[\"children\"]])\n\n def json(self):\n\"\"\"\n The whole tree of softwware as json.\n\n Returns:\n str: JSON.\n\n \"\"\"\n return json.dumps(self._tree)\n\n def __bool__(self):\n return True if self._tree[\"children\"] else False\n\n def __nonzero__(self):\n # Python 2.7\n return self.__bool__()\n\n def as_dict(self):\n\"\"\"\n Returns:\n dict: The underlying software dictionary.\n\n \"\"\"\n return self._tree\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.__init__","title":"<code>__init__(self, packages, host_products, kwargs)</code> <code>special</code>","text":"<p>Build the tree with a list of packages.</p> <p>Parameters:</p> Name Type Description Default <code>packages</code> <code>list</code> <p>List of packages direct from the Conductor packages endpoint.</p> required <code>host_products</code> <p>Filter the tree to contain only top-level host packages of products specified in this list and their plugins. If there are no host_products specified, and the product keyword is omitted, the tree contains all packages.</p> <code>()</code> <p>Keyword arguments:</p> Name Type Description <code>product</code> <code>str</code> <p>Build the tree from versions of a single product and its compatible plugins. Defaults to <code>None</code>, in which case the tree is built from all packages. It is an error to specify both host_products and product. If a nonexistent product is given, the PackageTree is empty. By specifying <code>product</code>, you can build the object based on a single plugin product.</p> <code>platforms</code> <code>set</code> <p>Build the tree from versions for a specific platform. Defaults to the set <code>{\"linux\", \"windows\"}</code>.</p> <p>Exceptions:</p> Type Description <code>KeyError</code> <p>An invalid platform was provided.</p> <code>ValueError</code> <p>Cannot choose both product and host_products.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n# Request packages as a flat list from Conductor.\n>>> packages = api_client.request_software_packages()\n# Build tree of dependencies from packages list\n>>> pt = package_tree.PackageTree(packages, \"cinema4d\", \"maya-io\")\n>>> for path in pt.to_path_list():\n>>> print(path)\ncinema4d 22.118.RB320081 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\nmaya-io 2022.SP3 linux\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def __init__(self, packages, host_products, kwargs):\n\"\"\"Build the tree with a list of packages.\n\n Args:\n packages (list): List of packages direct from the [Conductor packages endpoint](https://dashboard.conductortech.com/api/v1/ee/packages).\n\n host_products: Filter the tree to contain only top-level host packages of products specified in this list and their plugins. If there are no host_products specified, and the product keyword is omitted, the tree contains all packages.\n\n Keyword Args:\n product (str): Build the tree from versions of a single product and its compatible plugins. Defaults to `None`, in which case the tree is built from all packages. It is an error to specify both host_products and product. If a nonexistent product is given, the PackageTree is empty. By specifying `product`, you can build the object based on a single plugin product.\n platforms (set): Build the tree from versions for a specific platform. Defaults to the set `{\"linux\", \"windows\"}`.\n\n Raises:\n KeyError: An invalid platform was provided.\n ValueError: Cannot choose both product and host_products.\n\n Example:\n >>> from ciocore import api_client, package_tree\n # Request packages as a flat list from Conductor.\n >>> packages = api_client.request_software_packages()\n # Build tree of dependencies from packages list\n >>> pt = package_tree.PackageTree(packages, \"cinema4d\", \"maya-io\")\n >>> for path in pt.to_path_list():\n >>> print(path)\n cinema4d 22.118.RB320081 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\n maya-io 2022.SP3 linux\n \"\"\"\n\n\n platforms = kwargs.get(\"platforms\",PLATFORMS)\n product=kwargs.get(\"product\")\n\n unknown_platforms = set(platforms) - PLATFORMS\n if unknown_platforms:\n raise KeyError(\"Unrecognized platform {}\".format(\" \".join(unknown_platforms)))\n\n if product and host_products:\n raise ValueError(\"You cannot choose both product and host_products.\")\n\n packages = [_clean_package(p) for p in packages if p[\"platform\"] in platforms]\n\n root_ids = [] \n if product:\n root_ids = [p[\"package_id\"] for p in packages if p[\"product\"] == product]\n else:\n for p in packages:\n if not p[\"plugin_host_product\"]:\n if p[\"product\"] in host_products or not host_products:\n root_ids.append(p[\"package_id\"])\n\n self._tree = _build_tree(packages, {\"children\": [], \"plugins\": root_ids})\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.supported_host_names","title":"<code>supported_host_names(self)</code>","text":"<p>All host names from the software tree.</p> <p>These names can be used to populate a dropdown menu. Then a single selection from that menu can be used to retrieve the complete package in order to generate an environment dictionary and get package IDs.</p> <p>Returns:</p> Type Description <code>list(str)</code> <p>Fully qualified DCC hosts of the form: <code>product version platform</code>.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> pt.supported_host_names()\ncinema4d 21.209.RB305619 linux\ncinema4d 22.116.RB316423 linux\ncinema4d 22.118.RB320081 linux\ncinema4d 23.110.RB330286 linux\ncinema4d 24.111 linux\ncinema4d 24.111 windows\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def supported_host_names(self):\n\"\"\"\n All host names from the software tree.\n\n These names can be used to populate a dropdown menu. Then a single selection from that menu\n can be used to retrieve the complete package in order to generate an environment dictionary\n and get package IDs.\n\n Returns:\n list(str): Fully qualified DCC hosts of the form: `product version platform`.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.supported_host_names()\n cinema4d 21.209.RB305619 linux\n cinema4d 22.116.RB316423 linux\n cinema4d 22.118.RB320081 linux\n cinema4d 23.110.RB330286 linux\n cinema4d 24.111 linux\n cinema4d 24.111 windows\n \"\"\"\n\n paths = []\n for pkg in self._tree[\"children\"]:\n paths.append(to_name(pkg))\n return sorted(paths)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.supported_plugins","title":"<code>supported_plugins(self, host)</code>","text":"<p>Find the plugins that are children of the given DCC host.</p> <p>The result does not contain platform information since we assume that plugins are compatible with the DCC host that was used to request them.</p> <p>Parameters:</p> Name Type Description Default <code>host</code> <code>str</code> <p>Name of the DCC host, typically one of the entries returned by supported_host_names().</p> required <p>Returns:</p> Type Description <code>list(dict)</code> <p>Each entry contains a plugin product and a list of versions.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree \n>>> packages = api_client.request_software_packages() \n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\") \n>>> name = pt.supported_host_names()[0]\n>>> pt.supported_plugins(name)\n[\n {\n \"plugin\": \"arnold-cinema4d\",\n \"versions\": [\n \"3.3.2.100\",\n \"3.3.3.0\"\n ]\n },\n {\n \"plugin\": \"redshift-cinema4d\",\n \"versions\": [\n \"2.6.54\",\n \"2.6.56\",\n \"3.0.21\",\n \"3.0.22\",\n ],\n },\n]\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def supported_plugins(self, host):\n\"\"\"\n Find the plugins that are children of the given DCC host.\n\n The result does not contain platform information since we assume that plugins are compatible with the DCC host that was used to request them.\n\n Args:\n host (str): Name of the DCC host, typically one of the entries returned by [supported_host_names()](/package_tree/#ciocore.package_tree.PackageTree.supported_host_names).\n\n Returns:\n list(dict): Each entry contains a plugin product and a list of versions.\n\n Example:\n >>> from ciocore import api_client, package_tree \n >>> packages = api_client.request_software_packages() \n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\") \n >>> name = pt.supported_host_names()[0]\n >>> pt.supported_plugins(name)\n [\n {\n \"plugin\": \"arnold-cinema4d\",\n \"versions\": [\n \"3.3.2.100\",\n \"3.3.3.0\"\n ]\n },\n {\n \"plugin\": \"redshift-cinema4d\",\n \"versions\": [\n \"2.6.54\",\n \"2.6.56\",\n \"3.0.21\",\n \"3.0.22\",\n ],\n },\n ]\n \"\"\"\n\n try:\n subtree = self.find_by_name(host)\n plugin_versions = _to_path_list(subtree)\n except TypeError:\n return []\n\n if not plugin_versions:\n return []\n\n plugin_dict = {}\n for plugin, version, _ in [pv.split(\" \") for pv in plugin_versions]:\n if plugin not in plugin_dict:\n plugin_dict[plugin] = []\n plugin_dict[plugin].append(version)\n\n # convert to list so it can be sorted\n plugins = []\n for key in plugin_dict:\n plugins.append({\"plugin\": key, \"versions\": sorted(plugin_dict[key])})\n\n return sorted(plugins, key=lambda k: k[\"plugin\"])\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.find_by_name","title":"<code>find_by_name(self, name, limit=None)</code>","text":"<p>Search the tree for a product with the given name.</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>The name constructed from the package using to_name(). It must be an exact match with product, version, and platform. For example: <code>maya 2018.0 windows</code></p> required <p>Keyword arguments:</p> Name Type Description <code>limit</code> <code>int</code> <p>Limit the search depth. Defaults to <code>None</code>.</p> <p>Returns:</p> Type Description <code>object</code> <p>The package that matches.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> pt.find_by_name(\"redshift-cinema4d 3.0.64 linux\")\n{\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '64',\n 'vendor': 'maxon',\n 'children': [],\n ...\n}\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def find_by_name(self, name, limit=None):\n\"\"\"\n Search the tree for a product with the given name.\n\n Args:\n name (str): The name constructed from the package using to_name(). It must be an exact match with product, version, and platform. For example: `maya 2018.0 windows`\n\n Keyword Args:\n limit (int): Limit the search depth. Defaults to `None`.\n\n Returns:\n object: The package that matches.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.find_by_name(\"redshift-cinema4d 3.0.64 linux\")\n {\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '64',\n 'vendor': 'maxon',\n 'children': [],\n ...\n }\n \"\"\"\n\n return _find_by_name(self._tree, name, limit, 0)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.find_by_path","title":"<code>find_by_path(self, path)</code>","text":"<p>Find the package uniquely described by the given path.</p> <p>The path is of the form returned by the to_path_list() method.</p> <p>Parameters:</p> Name Type Description Default <code>path</code> <code>str</code> <p>The path</p> required <p>Returns:</p> Type Description <code>object</code> <p>The package or None if no package exists with the given path.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree, package_environment\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> pt.find_by_path(\"cinema4d 24.111 linux/redshift-cinema4d 3.0.62 linux\")\n{\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '62',\n 'vendor': 'maxon',\n 'children': [],\n 'plugin_host_version': \"24\",\n ...\n}\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def find_by_path(self, path):\n\"\"\"\n Find the package uniquely described by the given path.\n\n The path is of the form returned by the to_path_list() method.\n\n Args:\n path (str): The path\n\n Returns:\n object: The package or None if no package exists with the given path.\n\n Example:\n >>> from ciocore import api_client, package_tree, package_environment\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.find_by_path(\"cinema4d 24.111 linux/redshift-cinema4d 3.0.62 linux\")\n {\n 'platform': 'linux',\n 'plugin_host_product': 'cinema4d',\n 'product': 'redshift-cinema4d',\n 'major_version': '3',\n 'release_version': '62',\n 'vendor': 'maxon',\n 'children': [],\n 'plugin_host_version': \"24\",\n ...\n }\n \"\"\"\n return _find_by_path(self._tree, path)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.to_path_list","title":"<code>to_path_list(self, name=None)</code>","text":"<p>Get paths to all nodes.</p> <p>This is useful for populating a chooser to choose packages fully qualified by path. Houdini's tree widget, for example, takes the below format unchanged and generates the appropriate UI.</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>Get paths below the tree represented by the name. Defaults to None (root node).</p> <code>None</code> <p>Returns:</p> Type Description <code>list(str)</code> <p>Paths to all nodes in the tree.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n>>> packages = api_client.request_software_packages()\n>>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n>>> pt.to_path_list()\ncinema4d 22.118.RB320081 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\ncinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.22 linux\ncinema4d 22.118.RB320081 linux/arnold-cinema4d 3.3.2.100 linux\n...\n</code></pre> <pre><code>>>> pt.to_path_list(name=\"cinema4d 24.111 linux\")\nredshift-cinema4d 3.0.57 linux\nredshift-cinema4d 3.0.62 linux\nredshift-cinema4d 3.0.45 linux\nredshift-cinema4d 3.0.64 linux\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def to_path_list(self, name=None):\n\"\"\"\n Get paths to all nodes.\n\n This is useful for populating a chooser to choose packages fully qualified by path.\n Houdini's tree widget, for example, takes the below format unchanged and generates the\n appropriate UI.\n\n Args:\n name (str): Get paths below the tree represented by the name. Defaults to None (root node).\n\n Returns:\n list(str): Paths to all nodes in the tree.\n\n Example:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> pt = package_tree.PackageTree(packages, product=\"cinema4d\")\n >>> pt.to_path_list()\n cinema4d 22.118.RB320081 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.43 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.45 linux\n cinema4d 22.118.RB320081 linux/redshift-cinema4d 3.0.22 linux\n cinema4d 22.118.RB320081 linux/arnold-cinema4d 3.3.2.100 linux\n ...\n\n >>> pt.to_path_list(name=\"cinema4d 24.111 linux\")\n redshift-cinema4d 3.0.57 linux\n redshift-cinema4d 3.0.62 linux\n redshift-cinema4d 3.0.45 linux\n redshift-cinema4d 3.0.64 linux\n \"\"\"\n if name:\n subtree = self.find_by_name(name)\n return _to_path_list(subtree)\n return _to_path_list(self._tree)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.platforms","title":"<code>platforms(self)</code>","text":"<p>Get the platforms represented by packages in the tree.</p> <p>Returns:</p> Type Description <code>set</code> <p>The set of platforms.</p> Source code in <code>ciocore/package_tree.py</code> <pre><code>def platforms(self):\n\"\"\"\n Get the platforms represented by packages in the tree.\n\n Returns:\n set: The set of platforms.\n \"\"\"\n\n # No need to recurse. Plugins are assumed to be compatible with the host.\n return set([host[\"platform\"] for host in self._tree[\"children\"]])\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.json","title":"<code>json(self)</code>","text":"<p>The whole tree of softwware as json.</p> <p>Returns:</p> Type Description <code>str</code> <p>JSON.</p> Source code in <code>ciocore/package_tree.py</code> <pre><code>def json(self):\n\"\"\"\n The whole tree of softwware as json.\n\n Returns:\n str: JSON.\n\n \"\"\"\n return json.dumps(self._tree)\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.PackageTree.as_dict","title":"<code>as_dict(self)</code>","text":"<p>Returns:</p> Type Description <code>dict</code> <p>The underlying software dictionary.</p> Source code in <code>ciocore/package_tree.py</code> <pre><code>def as_dict(self):\n\"\"\"\n Returns:\n dict: The underlying software dictionary.\n\n \"\"\"\n return self._tree\n</code></pre>"},{"location":"apidoc/package_tree/#ciocore.package_tree.to_name","title":"<code>to_name(pkg)</code>","text":"<p>Generate a name like <code>houdini 16.5.323 linux</code> or <code>maya 2016.SP3 linux</code>.</p> <p>This name is derived from the product and version fields in a package. Note: It is not necessarily possible to go the other way and extract version fields from the name.</p> <p>Parameters:</p> Name Type Description Default <code>pkg</code> <code>object</code> <p>An object with product, platform, and all version fields.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The package name.</p> <p>Examples:</p> <pre><code>>>> from ciocore import api_client, package_tree\n>>> packages = api_client.request_software_packages()\n>>> package_tree.to_name(packages[0])\nredshift-maya 3.0.64 linux\n</code></pre> Source code in <code>ciocore/package_tree.py</code> <pre><code>def to_name(pkg):\n\"\"\"\n Generate a name like `houdini 16.5.323 linux` or `maya 2016.SP3 linux`.\n\n This name is derived from the product and version fields in a package. Note: It is not\n necessarily possible to go the other way and extract version fields from the name.\n\n Args:\n pkg (object): An object with product, platform, and all version fields.\n\n Returns:\n str: The package name.\n\n Examples:\n >>> from ciocore import api_client, package_tree\n >>> packages = api_client.request_software_packages()\n >>> package_tree.to_name(packages[0])\n redshift-maya 3.0.64 linux\n\n \"\"\" \n\n version_parts = [\n pkg[\"major_version\"],\n pkg[\"minor_version\"],\n pkg[\"release_version\"],\n pkg[\"build_version\"],\n ]\n version_string = (\".\").join([p for p in version_parts if p])\n if pkg[\"platform\"] not in PLATFORMS:\n raise KeyError(\"Invalid platform: {}\".format(pkg[\"platform\"]))\n return \" \".join(filter(None, [pkg[\"product\"], version_string, pkg[\"platform\"]]))\n</code></pre>"},{"location":"cmdline/docs/","title":"Docs","text":"<p>Open the Conductor Core documentation in a web browser.</p> <p>To use this tool, you need to run the following command:</p> <pre><code>conductor docs [OPTIONS]\n</code></pre>"},{"location":"cmdline/docs/#options","title":"Options","text":"<p>The <code>docs</code> command provides an option to choose the port in the event of a conflict.</p> Option Type Default -p, --port INTEGER 8025 <p>The port on which to serve the documentation.</p> --help <p>Show this message and exit.</p>"},{"location":"cmdline/downloader/","title":"Downloader","text":"<p>The Conductor Download tool downloads renders and other output files from Conductor. You can give a list of job ids to download, or you can omit jobids and the downloader will run in daemon mode.</p> <p>If you provide jobids, the default behavior is to download all the files from completed tasks for those jobs. You can however specify an explicit set of tasks to download by providing a task range spec after each job id. To do so, append a colon to the job id and then a compact task specification. See the examples.</p> <p>In daemon mode, the downloader polls for new jobs to download. Examples:</p> <ul> <li><code>conductor download</code></li> <li><code>conductor download 1234 1235</code></li> <li><code>conductor download 1234:1-10</code></li> <li><code>conductor download 1234:1-5x2,10,12-14</code></li> <li><code>conductor download 1234:1-5 1235:5-10</code></li> </ul>"},{"location":"cmdline/downloader/#options","title":"Options:","text":"Option Type Default -d, --destination TEXT <p>Override the output directory</p> -lv, --log_level ENUM INFO <p>The logging level to display. Options are: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET.</p> -ld, --log_dir TEXT <p>Write a log file to the given directory. The log rotates, creating a new log file every day, while storing logs for the last 7 days.</p> <p>The log file is named conductor_ul.log</p> -tc, --thread_count INTEGER <p>The number of threads that should download simultaneously.</p> <p>Increasing the thread count may improve upload performance, but it can also consume more system resources.</p> -lc, --location TEXT <p>Specify a location tag to associate with uploads, downloads, and submissions.</p> <p>A location tag allows you to limit the scope of your uploads and downloads to jobs sharing the same location tag. This is useful while using the uploader or downloader in daemon mode.</p> --help <p>Show this message and exit.</p>"},{"location":"cmdline/packages/","title":"Packages","text":"<p>List the software packages available on the render nodes in the cloud.</p> <p>To use this tool, you need to run the following command:</p> <pre><code>conductor packages [OPTIONS]\n</code></pre>"},{"location":"cmdline/packages/#options","title":"Options","text":"<p>The <code>packages</code> command provides one option to format its output.</p> Option Type Default -f, --fmt, --format TEXT <ul> <li>text: The output is a simple list of software names and versions, with nesting to indicate plugin compatibility. Output is sent to stdout.</li> <li>markdown: Designed for use in other markdown documentation systems where it benefits from consistent styling. Output is sent to stdout and can be piped to a file.</li> <li>html: Opens a browser window and displays the output in an HTML page.</li> </ul> <p>Valid values are: [text\|markdown\|html]</p> --help <p>Show this message and exit.</p> <p>Feel free to explore all the software packages available on the render nodes with the <code>conductor packages</code> command. You have the flexibility to choose the format of the output as per your preference. Whether you want a simple text list, styled markdown for documentation, or an interactive HTML page, we've got you covered!</p>"},{"location":"cmdline/uploader/","title":"Uploader","text":"<p>The Conductor Upload tool allows you to upload files to Conductor. It provides multiple options for customization and flexibility.</p>"},{"location":"cmdline/uploader/#usage","title":"Usage","text":"<p>To use the Conductor Upload tool, you need to run the following command:</p> <pre><code>conductor upload [OPTIONS] [PATHS]...\n</code></pre> <p>By default, without any arguments, the uploader runs in daemon mode. In this mode, it continuously watches for files to upload for submitted jobs. If you want to manually specify a list of paths to upload, you can provide them as arguments after the command.</p> <p>For example, to upload multiple files named file1, file2, and file3, you can run:</p> <pre><code>conductor upload file1 file2 file3\n</code></pre>"},{"location":"cmdline/uploader/#options","title":"Options","text":"<p>The Conductor Upload tool provides several options to customize its behavior. Here are the available options:</p> Option Type Default -db, --database_filepath TEXT <p>This option allows you to specify a filepath to the local md5 caching database. The md5 caching feature improves uploading times by skipping md5 generation for previously uploaded files that haven't been modified since the last upload.</p> -md5, --md5_caching BOOLEAN <p>When set to true (default), the tool uses cached md5s to skip md5 checking for subsequent uploads of the same files. This dramatically improves uploading times. However, if there is concern that files may not be getting re-uploaded properly, you can set this flag to false.</p> -lv, --log_level ENUM <p>This option allows you to set the logging level to display. You can choose from the following levels: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET. By default, the log level is set to INFO.</p> <p>Options are: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET</p> -ld, --log_dir TEXT <p>If you want to write a log file, you can specify the directory using this option. The log file rotates, creating a new log file every day and storing logs for the last 7 days. The log file is named conductor_ul.log.</p> -tc, --thread_count INTEGER <p>This option sets the number of threads that should download simultaneously. Increasing the thread count may improve upload performance, but it can also consume more system resources.</p> -lc, --location TEXT <p>If you want to associate uploads, downloads, and submissions with a specific location tag, you can use this option. A location tag allows you to limit the scope of your uploads and downloads to jobs sharing the same location tag. This is particularly useful when using the uploader or downloader in daemon mode. </p> --help <p>Print help. </p>"},{"location":"cmdline/uploader/#conclusion","title":"Conclusion","text":"<p>The Conductor Upload tool provides a convenient way to upload files to Conductor. With its options and flexibility you can customize its behavior to suit your needs.</p>"}]}