cid-cmd 4.3.3__tar.gz → 4.3.6__tar.gz

{cid_cmd-4.3.3/cid_cmd.egg-info → cid_cmd-4.3.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cid-cmd
-Version: 4.3.3
+Version: 4.3.6
 Summary: Cloud Intelligence Dashboards deployment helper tool
 Home-page: https://github.com/aws-samples/aws-cudos-framework-deployment
 Author: AWS CUDOS Team
@@ -47,7 +47,7 @@ Dynamic: license-file
 1. [Notices](#Notices)
 
 ## Overview
-The Cloud Intelligence Dashboards is an open-source framework, lovingly cultivated and maintained by a group of customer-obsessed AWSers, that gives customers the power to get high-level and granular insight into their cost and usage data. Supported by the Well-Architected framework, the dashboards can be deployed by any customer using a CloudFormation template or a command-line tool in their environment in under 30 minutes. These dashboards help you to drive financial accountability, optimize cost, track usage goals, implement best-practices for governance, and achieve operational excellence across all your organization.
+The Cloud Intelligence Dashboards is an open-source framework, lovingly cultivated and maintained by a group of customer-obsessed AWSers, that gives customers the power to get high-level and granular insight into their cost and usage data. Supported by the Well-Architected framework, the dashboards can be deployed by any customer using a CloudFormation template or a command-line tool in their environment in under 30 minutes. These dashboards help you to drive financial accountability, optimize cost, track usage goals, implement best practices for governance, and achieve operational excellence across all your organization.
 
 Cloud Intelligence Dashboards Framework provides AWS customers with [more then 20 Dashboards](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/dashboards.html).
 * Foundational Dashboards - A set of main Dashboards that only require Cost and Usage Report(CUR)
@@ -81,7 +81,7 @@ This foundational architecture is recommended for starting and allows deployment
 5. User Teams (Executives, FinOps, Engineers) can access Cloud Intelligence Dashboards in [Amazon QuickSight](https://aws.amazon.com/quicksight/). Access is secured through [AWS IAM](https://aws.amazon.com/iam/), IIC ([AWS IAM Identity Center](https://aws.amazon.com/iam/identity-center/), formerly SSO), and optional [Row Level Security](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/row-level-security.html).
 6. Optionally, the Advanced data collection can be deployed to enable advanced dashboards based on [AWS Trusted Advisor](https://aws.amazon.com/premiumsupport/trustedadvisor/), [AWS Health](https://aws.amazon.com/premiumsupport/technology/aws-health-dashboard/) Events and other sources. Additional data is retrieved from [AWS Organizations](https://aws.amazon.com/organizations/) or Linked Accounts. In this case [Amazon EventBridge](https://aws.amazon.com/eventbridge/) rule triggers an [AWS Step Functions](https://aws.amazon.com/step-functions/) workflow for data collection modules on a configurable schedule.
 7. The "Account Collector" [AWS Lambda](https://aws.amazon.com/lambda/) in AWS Step Functions retrieves linked account details using AWS Organizations API.
-8. The "Data Collection" Lambda function in AWS Step Functions assumes role in each linked account to retrieve account-specific data via [AWS SDK](https://aws.amazon.com/developer/tools/).
+8. The "Data Collection" Lambda function in AWS Step Functions assumes a role in each linked account to retrieve account-specific data via [AWS SDK](https://aws.amazon.com/developer/tools/).
 9. Retrieved data is stored in a centralized Amazon S3 Bucket.
 10. [Advanced Cloud Intelligence Dashboards](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/dashboards.html#advanced-dashboards) leverage Amazon Athena and Amazon QuickSight for comprehensive data analysis.
 
@@ -102,16 +102,16 @@ The following table provides a sample cost breakdown for deploying of Foundation
 | **Total Estimated Monthly Cost** |                              | **$100-$200**    |
 
 \* Costs are relative to the size of your Cost and Usage Report (CUR) data  
-\** Costs are relative to number of Users
+\** Costs are relative to the number of users
 
 **Additional Notes:**
 - Free trial available for 30 days for 4 QuickSight users
 - Actual costs may vary based on specific usage and data volume
 
-Pleas use AWS Pricing Calculator for precise estimation.
+Please use the AWS Pricing Calculator for precise estimation.
 
 ## Prerequisites
-You need access to AWS Accounts. We recommend deployment of the Dashboards in a dedicated Data Collection Account, other than your Management (Payer) Account. We provide provides a CloudFormation templates to copy CUR 2.0 data from your Management Account to a dedicated one. You can use it to aggregate data from multiple Management (Payer) Accounts or multiple Linked Accounts.
+You need access to AWS Accounts. We recommend deployment of the Dashboards in a dedicated Data Collection Account, other than your Management (Payer) Account. We provide CloudFormation templates to copy CUR 2.0 data from your Management Account to a dedicated one. You can use it to aggregate data from multiple Management (Payer) Accounts or multiple Linked Accounts.
 
 If you do not have access to the Management/Payer Account, you can still collect the data across multiple Linked accounts using the same approach.
 
@@ -119,7 +119,7 @@ The ownership of CID is usually with the FinOps team, who do not have administra
 
 
 ## Regions
-Make sure you are installing data collection in the same region where you are going to use the data to avoid cross region charges.
+Make sure you are installing data collection in the same region where you are going to use the data to avoid cross-region charges.
 CFN deployment is only available in a limited number of regions, while CLI deployment is region agnostic.
 
 | Region Name | Region Code | Support CLI  | Support CFN |
@@ -151,7 +151,7 @@ CFN deployment is only available in a limited number of regions, while CLI deplo
 
 ## Deployment Steps
 There are several ways we can deploy dashboards:
-1. Using cid-cmd tool from command line
+1. Using cid-cmd tool from the command line
 1. [CloudFormation Template](./cfn-templates/cid-cfn.yml) using cid-cmd tool in Amazon Lambda. (Recommended)
 
 Please refer to the deployment documentation [here](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/deployment-in-global-regions.html).
@@ -166,8 +166,8 @@ Please refer to the documentation [here](https://docs.aws.amazon.com/guidance/la
 Please refer to the documentation [here](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/faq.html).
 
 ## Changelogs
-For dashboards please check Change Logs [here](changes/)
-For CID deployment tool, including Cli and CFN please check [Releases](/releases)
+For dashboards please check change log [here](changes/)
+For CID deployment tool, including CLI and CFN please check [Releases](/releases)
 
 ## Feedback
 Please reference to [this page](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/feedback-support.html)
@@ -184,4 +184,4 @@ See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more inform
 This library is licensed under the MIT-0 License. See the [LICENSE](LICENSE) file.
 
 ## Notices
-Dashboards and their content: (a) are for informational purposes only, (b) represents current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.
+Dashboards and their content: (a) are for informational purposes only, (b) represent current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

{cid_cmd-4.3.3 → cid_cmd-4.3.6}/README.md

@@ -19,7 +19,7 @@
 1. [Notices](#Notices)
 
 ## Overview
-The Cloud Intelligence Dashboards is an open-source framework, lovingly cultivated and maintained by a group of customer-obsessed AWSers, that gives customers the power to get high-level and granular insight into their cost and usage data. Supported by the Well-Architected framework, the dashboards can be deployed by any customer using a CloudFormation template or a command-line tool in their environment in under 30 minutes. These dashboards help you to drive financial accountability, optimize cost, track usage goals, implement best-practices for governance, and achieve operational excellence across all your organization.
+The Cloud Intelligence Dashboards is an open-source framework, lovingly cultivated and maintained by a group of customer-obsessed AWSers, that gives customers the power to get high-level and granular insight into their cost and usage data. Supported by the Well-Architected framework, the dashboards can be deployed by any customer using a CloudFormation template or a command-line tool in their environment in under 30 minutes. These dashboards help you to drive financial accountability, optimize cost, track usage goals, implement best practices for governance, and achieve operational excellence across all your organization.
 
 Cloud Intelligence Dashboards Framework provides AWS customers with [more then 20 Dashboards](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/dashboards.html).
 * Foundational Dashboards - A set of main Dashboards that only require Cost and Usage Report(CUR)
@@ -53,7 +53,7 @@ This foundational architecture is recommended for starting and allows deployment
 5. User Teams (Executives, FinOps, Engineers) can access Cloud Intelligence Dashboards in [Amazon QuickSight](https://aws.amazon.com/quicksight/). Access is secured through [AWS IAM](https://aws.amazon.com/iam/), IIC ([AWS IAM Identity Center](https://aws.amazon.com/iam/identity-center/), formerly SSO), and optional [Row Level Security](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/row-level-security.html).
 6. Optionally, the Advanced data collection can be deployed to enable advanced dashboards based on [AWS Trusted Advisor](https://aws.amazon.com/premiumsupport/trustedadvisor/), [AWS Health](https://aws.amazon.com/premiumsupport/technology/aws-health-dashboard/) Events and other sources. Additional data is retrieved from [AWS Organizations](https://aws.amazon.com/organizations/) or Linked Accounts. In this case [Amazon EventBridge](https://aws.amazon.com/eventbridge/) rule triggers an [AWS Step Functions](https://aws.amazon.com/step-functions/) workflow for data collection modules on a configurable schedule.
 7. The "Account Collector" [AWS Lambda](https://aws.amazon.com/lambda/) in AWS Step Functions retrieves linked account details using AWS Organizations API.
-8. The "Data Collection" Lambda function in AWS Step Functions assumes role in each linked account to retrieve account-specific data via [AWS SDK](https://aws.amazon.com/developer/tools/).
+8. The "Data Collection" Lambda function in AWS Step Functions assumes a role in each linked account to retrieve account-specific data via [AWS SDK](https://aws.amazon.com/developer/tools/).
 9. Retrieved data is stored in a centralized Amazon S3 Bucket.
 10. [Advanced Cloud Intelligence Dashboards](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/dashboards.html#advanced-dashboards) leverage Amazon Athena and Amazon QuickSight for comprehensive data analysis.
 
@@ -74,16 +74,16 @@ The following table provides a sample cost breakdown for deploying of Foundation
 | **Total Estimated Monthly Cost** |                              | **$100-$200**    |
 
 \* Costs are relative to the size of your Cost and Usage Report (CUR) data  
-\** Costs are relative to number of Users
+\** Costs are relative to the number of users
 
 **Additional Notes:**
 - Free trial available for 30 days for 4 QuickSight users
 - Actual costs may vary based on specific usage and data volume
 
-Pleas use AWS Pricing Calculator for precise estimation.
+Please use the AWS Pricing Calculator for precise estimation.
 
 ## Prerequisites
-You need access to AWS Accounts. We recommend deployment of the Dashboards in a dedicated Data Collection Account, other than your Management (Payer) Account. We provide provides a CloudFormation templates to copy CUR 2.0 data from your Management Account to a dedicated one. You can use it to aggregate data from multiple Management (Payer) Accounts or multiple Linked Accounts.
+You need access to AWS Accounts. We recommend deployment of the Dashboards in a dedicated Data Collection Account, other than your Management (Payer) Account. We provide CloudFormation templates to copy CUR 2.0 data from your Management Account to a dedicated one. You can use it to aggregate data from multiple Management (Payer) Accounts or multiple Linked Accounts.
 
 If you do not have access to the Management/Payer Account, you can still collect the data across multiple Linked accounts using the same approach.
 
@@ -91,7 +91,7 @@ The ownership of CID is usually with the FinOps team, who do not have administra
 
 
 ## Regions
-Make sure you are installing data collection in the same region where you are going to use the data to avoid cross region charges.
+Make sure you are installing data collection in the same region where you are going to use the data to avoid cross-region charges.
 CFN deployment is only available in a limited number of regions, while CLI deployment is region agnostic.
 
 | Region Name | Region Code | Support CLI  | Support CFN |
@@ -123,7 +123,7 @@ CFN deployment is only available in a limited number of regions, while CLI deplo
 
 ## Deployment Steps
 There are several ways we can deploy dashboards:
-1. Using cid-cmd tool from command line
+1. Using cid-cmd tool from the command line
 1. [CloudFormation Template](./cfn-templates/cid-cfn.yml) using cid-cmd tool in Amazon Lambda. (Recommended)
 
 Please refer to the deployment documentation [here](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/deployment-in-global-regions.html).
@@ -138,8 +138,8 @@ Please refer to the documentation [here](https://docs.aws.amazon.com/guidance/la
 Please refer to the documentation [here](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/faq.html).
 
 ## Changelogs
-For dashboards please check Change Logs [here](changes/)
-For CID deployment tool, including Cli and CFN please check [Releases](/releases)
+For dashboards please check change log [here](changes/)
+For CID deployment tool, including CLI and CFN please check [Releases](/releases)
 
 ## Feedback
 Please reference to [this page](https://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/feedback-support.html)
@@ -156,4 +156,4 @@ See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more inform
 This library is licensed under the MIT-0 License. See the [LICENSE](LICENSE) file.
 
 ## Notices
-Dashboards and their content: (a) are for informational purposes only, (b) represents current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.
+Dashboards and their content: (a) are for informational purposes only, (b) represent current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

cid_cmd-4.3.6/cid/__init__.py

@@ -0,0 +1,6 @@
+# Declare namespace
+
+try:
+    __import__('pkgutil').extend_path(__path__, __name__)
+except NameError:
+    pass

cid_cmd-4.3.6/cid/_version.py

@@ -0,0 +1,2 @@
+__version__ = '4.3.6'
+

{cid_cmd-4.3.3 → cid_cmd-4.3.6}/cid/builtin/core/data/resources.yaml

@@ -75,18 +75,6 @@ dashboards:
         'item_description', 'pricing_unit', 'region', 'pricing_term', 'linked_account_id', 'savings_plan_a_r_n', 'tags_json'
     ]
 
-  TAO:
-    category: 'Advanced'
-    name: Trusted Advisor Organizational View
-    templateId: ta-organizational-view
-    dashboardId: ta-organizational-view
-    dependsOn:
-      datasets:
-        - ta-organizational-view
-    versions:
-      minTemplateVersion: 13
-      minTemplateDescription: "v1.4.0"
-
   Trends:
     category: 'Additional'
     name: Trends Dashboard

{cid_cmd-4.3.3 → cid_cmd-4.3.6}/cid/cli.py

@@ -167,6 +167,7 @@ def export(ctx, **kwargs):
                (definition|template) A method (definition=pull json definition of Analysis OR template=create QuickSight Template)
         --export-known-datasets
             (no|yes)                 If 'yes' the export will include DataSets that are already in resources file. Default = no
+        --export-tables (no|yes).    If 'yes' the export will include Tables. Default: no = only Views
         --category TEXT              The dashboards category. Default = Custom
         --output                     A filename (.yaml) If provided an existing file it will be analyzed for default values and overridden 
     """

{cid_cmd-4.3.3 → cid_cmd-4.3.6}/cid/common.py

@@ -6,7 +6,7 @@ import functools
 import webbrowser
 from string import Template
 from typing import Dict
-from pkg_resources import resource_string
+from importlib import resources
 from importlib.metadata import entry_points
 from functools import cached_property
 
@@ -433,12 +433,21 @@ class Cid():
                     except (self.athena.client.exceptions.ClientError, CidError, CidCritical) as exc:
                         raise CidCritical(f'Failed fetching parameter {prefix}{key}: {exc}.') from exc
                     if not res_list:
-                        raise CidCritical(f'Failed fetching parameter {prefix}{key}, {value}. Athena returns empty results. {value.get("error")}')
-                    elif len(res_list) == 1:
-                        params[key] = '-'.join(res_list[0])
+                        raise CidCritical(f'Failed fetching parameter {prefix}{key}, {value}. Athena returns empty results. {value.get("error", "")}')
+                    options = ['-'.join(res) for res in res_list]
+                    default = value.get('default', '{default not provided}')
+                    if len(options) == 1:
+                        # silently taking the 1st available option
+                        params[key] = options[0]
+                    elif not utils.isatty():
+                        if default in options:
+                            # silently taking the default option if we cannot ask user
+                            params[key] = default
+                        else:
+                            # silently taking the first option
+                            params[key] = sorted(options)[0]
                     else:
-                        options = ['-'.join(res) for res in res_list]
-                        default = value.get('default')
+                        # asking user
                         params[key] = get_parameter(
                             param_name=prefix + key,
                             message=f"Required parameter: {key} ({value.get('description')})",
@@ -624,7 +633,16 @@ class Cid():
                 dataset = self.qs.describe_dataset(id=dashboard_datasets.get(dataset_name), no_cache=True)
 
             if not isinstance(dataset, Dataset):
-                # Second chance:  try to find the dataset with the id that is the name
+                # Second chance:  try to find the dataset with the id from resources
+                try:
+                    _ds_id = self.resources['datasets'].get(dataset_name, {}).get('data', {}).get('DataSetId')
+                    if _ds_id:
+                        dataset = self.qs.describe_dataset(id=_ds_id, no_cache=True)
+                except Exception as exc:
+                    logger.debug(f'Failed to describe_dataset {dataset_name} {exc}')
+
+            if not isinstance(dataset, Dataset):
+                # Third chance:  try to find the dataset with the id that is the name
                 try:
                     dataset = self.qs.describe_dataset(id=dataset_name, no_cache=True)
                 except Exception as exc:
@@ -662,14 +680,18 @@ class Cid():
 
                 if not matching_datasets:
                     reco = ''
-                    logger.warning(f'Dataset {dataset_name} is not found.')
-                    if utils.exec_env()['shell'] == 'lambda':
-                        # We are in lambda
-                        reco = 'You can try deleting existing dataset and re-run.'
-                    else:
-                        # We are in command line mode
-                        reco = 'Please retry with --update "yes" --force --recursive flags.'
-                    raise CidCritical(f'Failed to find a Dataset "{dataset_name}" with required fields. ' + reco)
+                    new_datasets = self.create_datasets([dataset_name], known_datasets=dashboard_datasets, recursive=recursive, update=update)
+                    if not new_datasets:
+                        logger.warning(f'Dataset {dataset_name} is not found.')
+                        if utils.exec_env()['shell'] == 'lambda':
+                            # We are in lambda
+                            reco = 'You can try deleting existing dataset and re-run.'
+                        else:
+                            # We are in command line mode
+                            reco = 'Please retry with --update "yes" --force --recursive flags.'
+                        raise CidCritical(f'Failed to find a Dataset "{dataset_name}" with required fields. ' + reco)
+                    _ds_id = self.resources['datasets'].get(dataset_name, {}).get('data', {}).get('DataSetId')
+                    dashboard_definition['datasets'][dataset_name] = f'arn:{self.base.partition}:quicksight:{self.base.region}:{self.base.account_id}:dataset/{_ds_id}'
                 elif len(matching_datasets) >= 1:
                     if len(matching_datasets) > 1:
                         # FIXME: propose a choice?
@@ -1018,10 +1040,9 @@ class Cid():
                             param_name='folder-name',
                             message='Please enter the folder name to create'
                         )
-                        folder_permissions_tpl = Template(resource_string(
-                            package_or_requirement='cid.builtin.core',
-                            resource_name=f'data/permissions/folder_permissions.json',
-                        ).decode('utf-8'))
+                        folder_permissions_tpl = Template(
+                            (resources.files('cid.builtin.core') / 'data/permissions/folder_permissions.json').read_text()
+                        )
                         columns_tpl = {
                             'PrincipalArn': self.qs.get_principal_arn()
                         }
@@ -1057,10 +1078,9 @@ class Cid():
             columns_tpl = {
                 'PrincipalArn': principal_arn
             }
-            dashboard_permissions_tpl = Template(resource_string(
-                package_or_requirement='cid.builtin.core',
-                resource_name=template_filename,
-            ).decode('utf-8'))
+            dashboard_permissions_tpl = Template(
+                (resources.files('cid.builtin.core') / template_filename).read_text()
+            )
             dashboard_permissions = json.loads(dashboard_permissions_tpl.safe_substitute(columns_tpl))
             dashboard_params = {
                 "GrantPermissions": [
@@ -1085,10 +1105,9 @@ class Cid():
             if share_method == 'account':
                 logger.info(f'Sharing datasets/datasources with an account is not supported, skipping')
             else:
-                data_set_permissions_tpl = Template(resource_string(
-                    package_or_requirement='cid.builtin.core',
-                    resource_name=f'data/permissions/data_set_permissions.json',
-                ).decode('utf-8'))
+                data_set_permissions_tpl = Template(
+                    (resources.files('cid.builtin.core') / 'data/permissions/data_set_permissions.json').read_text()
+                )
                 data_set_permissions = json.loads(data_set_permissions_tpl.safe_substitute(columns_tpl))
 
                 _datasources: Dict[str, Datasource] = {}
@@ -1103,10 +1122,9 @@ class Cid():
                         if not _datasources.get(_datasource.id):
                             _datasources.update({_datasource.id: _datasource})
 
-                data_source_permissions_tpl = Template(resource_string(
-                    package_or_requirement='cid.builtin.core',
-                    resource_name=f'data/permissions/data_source_permissions.json',
-                ).decode('utf-8'))
+                data_source_permissions_tpl = Template(
+                    (resources.files('cid.builtin.core') / f'data/permissions/data_source_permissions.json').read_text()
+                )
                 data_source_permissions = json.loads(data_source_permissions_tpl.safe_substitute(columns_tpl))
                 for k, v in _datasources.items():
                     logger.info(f'Sharing data source "{v.name}" ({k})')

{cid_cmd-4.3.3 → cid_cmd-4.3.6}/cid/export.py

@@ -144,10 +144,10 @@ def export_analysis(qs, athena, glue):
 
         cid_print(f'    Found DataSet <BOLD>{dataset_name}<END>.')
         if dataset_name in athena._resources.get('datasets'):
-            resources_datasets.append(dataset_name)
-            if not get_parameters().get('export-known-datasets'):
+            resources_datasets.append(dataset_name) # used later in dependencies
+            if not get_parameters().get('export-known-datasets') == 'yes':
                 cid_print(f'    DataSet <BOLD>{dataset_name}<END> is in resources. Skipping.')
-                continue
+                continue # prevent export dataset
 
         dataset_data = {
             "DataSetId": dataset.raw['DataSetId'],
@@ -284,7 +284,13 @@ def export_analysis(qs, athena, glue):
         if key in cur_tables or cur_helper.table_is_cur(name=key):
             logger.debug(f'Skipping {key} views - it is a CUR')
             continue
+        if key in ['account_map']:
+            cid_print(f'{key} is a special data. Processed separately. Skipping.')
+            continue
         if isinstance(view_data.get('data'), str):
+            if 'CREATE EXTERNAL TABLE' in view_data.get('data') and not get_parameters().get('export-tables') == 'yes':
+                cid_print(f'{key} is a Glue table. Skipping. To export table definitions use `--export-tables yes`')
+                continue
             #check if there is dependency on crawler
             crawler_names = re.findall(r"UPDATED_BY_CRAWLER\W+?'(.+?)''", view_data.get('data'))
             if crawler_names:

{cid_cmd-4.3.3 → cid_cmd-4.3.6}/cid/helpers/account_map.py

@@ -38,7 +38,7 @@ from functools import lru_cache
 
 from string import Template
 from boto3.session import Session
-from pkg_resources import resource_string
+from importlib import resources
 
 from cid.base import CidBase
 from cid.helpers import Athena, CUR
@@ -119,10 +119,7 @@ class AccountMap(CidBase):
         if view_definition.get('File'):
             view_file = view_definition.get('File')
             template = Template(
-                resource_string(
-                    view_definition.get('providedBy'),
-                    f'data/{view_file}'
-                ).decode('utf-8')
+                (resources.files(view_definition.get('providedBy')) / f'data/{view_file}').read_text()
             )
         elif view_definition.get('data'):
             template = Template(str(view_definition.get('data')))
@@ -156,10 +153,9 @@ class AccountMap(CidBase):
             sql_file = 'data/queries/shared/account_map_cur2.sql'
         else:
             sql_file = 'data/queries/shared/account_map_dummy.sql'
-        template = Template(resource_string(
-            package_or_requirement='cid.builtin.core',
-            resource_name=sql_file,
-        ).decode('utf-8'))
+        template = Template(
+            (resources.files('cid.builtin.core') / sql_file).read_text()
+        )
         columns_tpl = {
             'athena_view_name': name,
             'cur_table_name': cur.table_name,

{cid_cmd-4.3.3 → cid_cmd-4.3.6}/cid/helpers/cur.py

@@ -248,19 +248,24 @@ class CUR(AbstractCUR):
     def find_cur(self, database: str=None, table: str=None):
         """Choose CUR"""
         metadata = None
-        cur_database = database or get_parameters().get('cur-database')
-        if table or get_parameters().get('cur-table-name'):
-            table_name = table or get_parameters().get('cur-table-name')
+        cur_database = database or get_parameters().get('cur-database') or self.athena.DatabaseName
+        if not table and get_parameters().get('cur-table-name') and ',' not in str(get_parameters().get('cur-table-name')):
+            table = get_parameters().get('cur-table-name')
+        if table: # Table sepcified explicity, so no need to scan. Just need to make sure the table exists
             try:
-                metadata = self.athena.get_table_metadata(table_name, cur_database)
+                metadata = self.athena.get_table_metadata(table, cur_database)
             except self.athena.client.exceptions.MetadataException as exc:
-                raise CidCritical(f'Provided cur-table-name "{table_name}" in database "{cur_database or self.athena.DatabaseName}" is not found. Please make sure the table exists. This could also indicate a LakeFormation permission issue, see our FAQ for help.') from exc
+                logger.debug(f'Provided cur-table-name "{table}" in database "{cur_database}" is not found. ({exc})')
+                raise CidCritical(f'Provided cur-table-name "{table}" in database "{cur_database}" is not found. Please make sure the table exists. This could also indicate a LakeFormation permission issue, see our FAQ for help.')
             res, message = self.table_is_cur(table=metadata, return_reason=True)
             if not res:
-                raise CidCritical(f'Table {table_name} does not look like CUR. {message}')
-            return cur_database or self.athena.DatabaseName, metadata
+                raise CidCritical(f'Provided cur-table-name "{table}" in database "{cur_database}" is not cur. {message}')
+            return cur_database, metadata
 
         all_cur_tables = []
+        filter_names = None
+        if ',' in str(get_parameters().get('cur-table-name')):
+            filter_names = get_parameters().get('cur-table-name').split(',')
         if cur_database:
             cur_databases = [cur_database]
         else:
@@ -271,7 +276,12 @@ class CUR(AbstractCUR):
                     columns=self.cur_minimal_required_columns,
                     database_name=database,
                 )
-                all_cur_tables += [(database, table['Name']) for table in tables]
+                all_cur_tables += [
+                    (database, table['Name'])
+                    for table in tables
+                    if (not filter_names or table['Name'] in filter_names)
+                    and table['Name'] not in ('cur2_proxy', 'cur2_view')
+                ]
             except self.athena.client.exceptions.ClientError as exc:
                 if 'AccessDenied' in str(exc):
                     logger.info(f'Cannot read from athena database {database}')
@@ -279,10 +289,10 @@ class CUR(AbstractCUR):
                     raise
 
         if not all_cur_tables:
-            # FIXME : distinguish a case where we have NONE tables in any database. This might be because
+            # FIXME : distinguish a case where we have NONE tables in any database. This might be because lake formation
             raise CidCritical(
-                f'CUR table not found in {self.athena.region}. We need a least one table with these columns: {self.cur_minimal_required_columns}.'
-                 ' Please make sure you created cur and created Athena table, preferably with CID Cloud Formation stack.'
+                f'CUR table not found in {self.athena.region}. We need a least one Glue table with these columns: {self.cur_minimal_required_columns}.'
+                 ' Please make sure you have CUR, preferably with CID CloudFormation stack.'
                  ' Also if you have AWS Lake Formation, the user running the tool might need additional permissions'
             )
         databases = set([database for database, _ in all_cur_tables])
@@ -290,7 +300,7 @@ class CUR(AbstractCUR):
             choices = dict(sorted([(f'{database}.{tab}', (database, tab)) for database, tab in all_cur_tables], reverse=True))
         else:
             choices = dict(sorted([(f'{tab}', (database, tab)) for database, tab in all_cur_tables], reverse=True))
-        if not choices:
+        if not choices and not get_parameters().get('no-cur-creation'):
             choices['<CREATE CUR TABLE AND CRAWLER>'] = '<CREATE CUR TABLE AND CRAWLER>'
         answer =  get_parameter(
             param_name='cur-table-name-and-db',
@@ -303,7 +313,7 @@ class CUR(AbstractCUR):
             database, table = answer.split('.')
         else:
             database, table = answer
-        set_parameters({'cur-table-name': table,'cur-database': database, })
+        set_parameters({'cur-table-name': table,'cur-database': database})
         return database, self.athena.get_table_metadata(table, database_name=database)
 
     def ensure_column(self, column: str, column_type: str=None):

{cid_cmd-4.3.3 → cid_cmd-4.3.6}/cid/helpers/quicksight/__init__.py

@@ -6,7 +6,7 @@ import logging
 from uuid import uuid4
 from string import Template
 from typing import Dict, List, Union
-from pkg_resources import resource_string
+from importlib import resources
 
 from tqdm import tqdm
 
@@ -352,10 +352,9 @@ class QuickSight(CidBase):
         columns_tpl = {
             'PrincipalArn': self.get_principal_arn()
         }
-        data_source_permissions_tpl = Template(resource_string(
-            package_or_requirement='cid.builtin.core',
-            resource_name='data/permissions/data_source_permissions.json',
-        ).decode('utf-8'))
+        data_source_permissions_tpl = Template(
+            (resources.files('cid.builtin.core') / 'data/permissions/data_source_permissions.json').read_text()
+        )
         data_source_permissions = json.loads(data_source_permissions_tpl.safe_substitute(columns_tpl))
         datasource_name = datasource_id or "CID Athena"
         datasource_id = datasource_id or str(uuid4())
@@ -1050,10 +1049,9 @@ class QuickSight(CidBase):
         columns_tpl = {
             'PrincipalArn': self.get_principal_arn()
         }
-        data_set_permissions_tpl = Template(resource_string(
-            package_or_requirement='cid.builtin.core',
-            resource_name='data/permissions/data_set_permissions.json',
-        ).decode('utf-8'))
+        data_set_permissions_tpl = Template(
+            (resources.files('cid.builtin.core') / 'data/permissions/data_set_permissions.json').read_text()
+        )
         data_set_permissions = json.loads(data_set_permissions_tpl.safe_substitute(columns_tpl))
         definition.update({
             'AwsAccountId': self.account_id,
@@ -1265,10 +1263,9 @@ class QuickSight(CidBase):
 
         create_parameters = self._build_params_for_create_update_dash(definition)
 
-        dashboard_permissions_tpl = Template(resource_string(
-            package_or_requirement='cid.builtin.core',
-            resource_name='data/permissions/dashboard_permissions.json',
-        ).decode('utf-8'))
+        dashboard_permissions_tpl = Template(
+            (resources.files('cid.builtin.core') / 'data/permissions/dashboard_permissions.json').read_text()
+        )
         columns_tpl = {
             'PrincipalArn': self.get_principal_arn()
         }

cid_cmd-4.3.6/cid/plugin.py

@@ -0,0 +1,62 @@
+# Implements generic Plugin class to load plugins
+
+import json, yaml
+from importlib import resources
+import logging
+
+logger = logging.getLogger(__name__)
+
+class Plugin():
+
+    def __init__(self, name):
+        logger.debug(f'Initializing plugin {name}')
+        self.resources = {}
+        self.name = name
+        data_directory = 'data'
+        data_path = resources.files(self.name) / data_directory
+        if data_path.is_dir():
+            for resource_file in data_path.iterdir():
+                if resource_file.is_file():
+                    logger.debug(f'Located data file: {resource_file.name}')
+                    ext = resource_file.name.rsplit('.', -1)[-1].lower()
+                    content = None
+                    if ext == 'json':
+                        content = json.loads(resource_file.read_text())
+                        logger.debug(f'Loaded {resource_file.name} as JSON')
+                    elif ext in ['yaml', 'yml']:
+                        content = yaml.load(resource_file.read_text(), Loader=yaml.SafeLoader)
+                        logger.debug(f'Loaded {resource_file.name} as YAML')
+                    if content is None:
+                        logger.info(f'Unsupported file type: {resource_file.name}')
+                        continue
+                    # If plugin has resources defined in different files,
+                    # they will be merged into one dict
+                    resource_kind = resource_file.name.rsplit('.', -1)[0]
+                    supported_resource_kinds = ['dashboards', 'views', 'datasets']
+                    if resource_kind in supported_resource_kinds:
+                        self.resources[resource_kind] = content
+                        logger.info(f'Loaded {resource_kind} from {resource_file.name}')
+                    # If plugin has resources defined in one file,
+                    # simply add it to resources dict
+                    else:
+                        self.resources.update(content)
+                    # Add plugin name to every resource
+                    for v in self.resources.values():
+                        for item in v.values():
+                            if item is not None:
+                                item['providedBy'] = self.name
+                                item.update({'source': str(resource_file)})
+        logger.debug(f'Plugin {self.name} initialized')
+    
+    def provides(self) -> dict:
+        logger.debug(f'Provides: {self.resources}')
+        return self.resources
+
+    def get_resource(self, resource_name) -> str:
+        _resource_path = resources.files(self.name) / f'data/{resource_name}'
+        if _resource_path.is_file():
+            logger.info(f'Resource {resource_name} found')
+            _content = _resource_path.read_text()
+            logger.debug(f'Resource {resource_name} content: {_content}')
+            return _content
+        return None