@aws-mdaa/datawarehouse 1.4.0 → 1.6.0

package/README.md

@@ -1,53 +1,94 @@
 # Data Warehouse
 
-The Data Warehouse CDK application is used to configure and deploy resources required for a secure Redshift-based Data Warehouse on AWS.
+> **Note:** This documentation is also available in a rendered format [here](https://aws.github.io/modern-data-architecture-accelerator/packages/apps/analytics/datawarehouse-app/index.html).
 
-***
+Deploys a secure Redshift-based data warehouse with KMS encryption, VPC networking, SAML federation, scheduled pause/resume actions, event notifications, and automated credential rotation for database users. Common scenarios include centralizing structured data for BI reporting, running complex analytical queries across large datasets, or providing federated SQL access to business analysts.
 
-## Deployed Resources and Compliance Details
+---
 
-![datawarehouse](../../../constructs/L3/analytics/datawarehouse-l3-construct/docs/datawarehouse.png)
+## Deployed Resources
+
+This module deploys and integrates the following resources:
+
+**Warehouse KMS Key** - Customer-managed KMS key for warehouse resources.
+
+**Warehouse Bucket** - S3 bucket for warehouse utility and maintenance operations.
 
-**Warehouse KMS Key** - Will be used to encrypt all warehouse data at rest (Warehouse bucket, Redshift Cluster).
+**Warehouse Logging Bucket** (Optional) - S3 bucket for Redshift user activity audit logging. Uses SSE-S3 encryption (Redshift logging requirement).
 
-**Warehouse Bucket** - An S3 Bucket which can be used for warehouse utility/maintenance operations.
+**Execution Roles** - Externally managed IAM execution roles associated to the Redshift cluster for cross-service operations.
 
-**Warehouse Logging Bucket** - If 'enableAuditLoggingToS3' specified in config, an S3 Bucket specific to Redshift user activity logging will be created.
+**Warehouse Security Group** - Controls network connectivity to the Redshift cluster.
 
-* **Note** - This bucket will be configured with SSE-S3 encryption due to Redshift logging requirements (KMS not supported).
+**Warehouse Subnet Group** - Controls which subnets the cluster is deployed on.
 
-**Execution Roles** - List of externally managed execution roles required to be associated to the Redshift Cluster. Access to these roles may be granted to Redshift users in order to interact with other AWS services through Redshift queries and commands.
+**Warehouse Parameter Group** - Cluster configuration parameters controlling cluster behavior.
 
-**Warehouse Security Group** - Will control who can connect to the cluster according to the app config (CIDR ranges, other Security Groups).
+**Warehouse Cluster** - Redshift cluster conforming to the specified configuration.
 
-* All egress permitted by default
-* No ingress (to cluster) permitted by default
+**Warehouse Cluster Scheduled Actions** (Optional) - Scheduled actions to automatically pause and resume the Redshift cluster.
 
-**Warehouse Subnet Group** - Controls which subnets the cluster will be deployed on.
+**Warehouse Federation Roles** (Optional) - IAM roles for SAML-based federated access to the Redshift cluster via IAM Identity Providers.
 
-**Warehouse Parameter Group** - Contains cluster config parameters required to control cluster behaviour and ensure secure operation.
+**SNS Event Subscriptions** (Optional) - EventBridge subscriptions for cluster and scheduled action event notifications.
+
+**Redshift DB Service Users** (Optional) - Database users with credentials stored in Secrets Manager for programmatic access.
+
+**Warehouse Users** (Optional) - Redshift user credentials with configurable automated secret rotation.
+
+![datawarehouse](../../../constructs/L3/analytics/datawarehouse-l3-construct/docs/datawarehouse.png)
 
-* Enforces use of SSL on client connections
+---
 
-**Warehouse Cluster** - A Redshift cluster conforming to the specified configuration and security controls.
+## Related Modules
 
-* All data encrypted at rest using warehouse KMS key
-* SSL enforced on all client connections
-* Network access controlled by security group.
+- [Roles](../../governance/roles-app/README.md) — Create IAM roles for Redshift execution roles or SAML federation access
+- [Data Lake](../../datalake/datalake-app/README.md) — Redshift can query data lake S3 buckets via Redshift Spectrum with execution roles
+- [QuickSight Account](../quicksight-account-app/README.md) — Connect QuickSight to Redshift as a data source via VPC connection
+- [QuickSight Project](../quicksight-project-app/README.md) — Create QuickSight data sources pointing to the Redshift cluster
+- [OpenSearch](../opensearch-app/README.md) — Deploy OpenSearch as a complementary analytics engine for full-text search and log analytics
 
-**Warehouse Cluster Scheduled Actions** - Scheduled actions to automatically pause and resume Redshift cluster.
+---
 
-**Warehouse Federation Roles** - Roles which are used via IAM SAML Identity Providers to federate access to the cluster
+## Security/Compliance Details
 
-* Establishes assume role trust (with SAML) with IAM Identity Provider
-* Grants ability to dynamically generate cluster user and credentials, and join groups provided in the SAML claim by the identity provider
-* Groups must pre-exist in cluster, otherwise federation will fail
+This module is designed in alignment with MDAA security/compliance principles and CDK nag rulesets. Additional review is recommended prior to production deployment, ensuring organization-specific compliance requirements are met.
 
-**Warehouse Users** - Generates Redshift user credentials inside of cluster and stores them in a Secret
+- **Encryption at Rest**:
+  - All cluster data encrypted with customer-managed KMS key
+  - Warehouse and utility S3 buckets encrypted with KMS
+  - Audit logging bucket uses SSE-S3 (Redshift requirement)
+- **Encryption in Transit**:
+  - SSL enforced on all client connections via parameter group
+- **Least Privilege**:
+  - Database user credentials stored in Secrets Manager with configurable automatic rotation
+  - Execution roles scoped to specific Redshift operations
+- **Separation of Duties**:
+  - SAML federation roles support SSO access with dynamic credential generation and group membership via SAML claims
+  - Federation groups must pre-exist in the cluster
+  - Event notifications via SNS for cluster management and security events with configurable severity filtering
+- **Network Isolation**:
+  - Cluster deployed in VPC with configurable subnet group
+  - Security group denies all ingress by default; access must be explicitly granted via CIDR or security group rules
 
-* Automated secret rotation can be triggered on an configurable cycle (by days)
+---
 
-***
+## AWS Service Endpoints
+
+The following VPC endpoints may be required if public AWS service endpoint connectivity is unavailable (e.g., private subnets without NAT gateway, firewalled environments, or PrivateLink-only architectures):
+
+| AWS Service       | Endpoint Service Name                   | Type      |
+| ----------------- | --------------------------------------- | --------- |
+| Redshift          | `com.amazonaws.{region}.redshift`       | Interface |
+| Redshift Data API | `com.amazonaws.{region}.redshift-data`  | Interface |
+| KMS               | `com.amazonaws.{region}.kms`            | Interface |
+| S3                | `com.amazonaws.{region}.s3`             | Gateway   |
+| Secrets Manager   | `com.amazonaws.{region}.secretsmanager` | Interface |
+| SNS               | `com.amazonaws.{region}.sns`            | Interface |
+| CloudWatch Logs   | `com.amazonaws.{region}.logs`           | Interface |
+| STS               | `com.amazonaws.{region}.sts`            | Interface |
+
+---
 
 ## Configuration
 
@@ -56,136 +97,49 @@ The Data Warehouse CDK application is used to configure and deploy resources req
 Add the following snippet to your mdaa.yaml under the `modules:` section of a domain/env in order to use this module:
 
 ```yaml
-          datawarehouse: # Module Name can be customized
-            module_path: "@aws-caef/datawarehouse" # Must match module NPM package name
-            module_configs:
-              - ./datawarehouse.yaml # Filename/path can be customized
+datawarehouse: # Module Name can be customized
+  module_path: '@aws-mdaa/datawarehouse' # Must match module NPM package name
+  module_configs:
+    - ./datawarehouse.yaml # Filename/path can be customized
 ```
 
-### Module Config (./datawarehouse.yaml)
+### Module Config Samples and Variants
 
-[Config Schema Docs](SCHEMA.md)
+Copy the contents of the relevant sample config below into the `./datawarehouse.yaml` file referenced in the MDAA config snippet above.
+
+#### Minimal Configuration
+
+Required properties only — a basic Redshift cluster with VPC networking, security group ingress, and audit logging. Start here for a quick data warehouse deployment before adding federation, scheduled actions, or database users.
+
+[sample-config-minimal.yaml](sample_configs/sample-config-minimal.yaml)
 
 ```yaml
-# Specify the admin username to be created on the cluster. A secret will
-# be automatically generated containing the admin password.
-adminUsername: admin
-
-# The admin password will be automatically rotatated after this many days
-adminPasswordRotationDays: 30
-
-# The number of days that automated snapshots are retained (1-35 days)
-# Set 0 to disable the snapshot.
-# Default - 1
-automatedSnapshotRetentionDays: 3
-
-# An optional list of arns for keys which may be used to write data to the cluster bucket.
-# This may be useful to allow a Glue job to write data to the cluster bucket in order to load into the cluster.
-additionalBucketKmsKeyArns:
-  - arn:{{partition}}:kms:{{region}}:{{account}}:key/abcd-123123-abcd-12312421
-
-#Used to configure SAML federations
-federations:
-  - federationName: "test" # Should be descriptive and unique
-    # This is the arn of the IAM Identity Provider
-    providerArn: arn:{{partition}}:iam::{{account}}:saml-provider/sample-saml-identity-provider
-
-# This is a set of Role/Principal Arns which will be granted access to the Warehouse S3 bucket
-dataAdminRoles:
-  - arn: arn:{{partition}}:iam::{{account}}:role/Admin
-
-# A list of roles which will be provided read/write access to the warehouse bucket
-warehouseBucketUserRoles:
-  - name: User
-  - name: team2-ex-role
-
-# Set of execution roles required to be associated to the cluster
-# If execution role requires read/write access to  warehouse bucket, explicitly add that role to 'warehouseBucketUserRoles' property
-executionRoles:
-  - arn: arn:{{partition}}:iam::{{account}}:role/team1-ex-role
-  - name: team2-ex-role
-
-# The VPC and subnets on which the cluster will be deployed. If automatic cluster relocation is required,
-# at least one subnet per AZ should be specified.
-vpcId: vpc-12321421412
-subnetIds:
-  - subnet-12312312421
-  - subnet-12312321412
-
-#A preferred maintenance window day/time range. Should be specified as a range ddd:hh24:mi-ddd:hh24:mi (24H Clock UTC).
-#Example: 'Sun:23:45-Mon:00:15'
-preferredMaintenanceWindow: Sun:23:45-Mon:00:15
-
-# Port the cluster will listen on (defaults to 5440)
-clusterPort: 5440
-
-# Ingress rules to be added to the cluster security group.
-# All other traffic will be blocked
-# Can reference other security groups (prefix sg:) or ipv4 CIDR sources (prefix ipv4:)
-securityGroupIngress:
-  ipv4:
-    - 172.31.0.0/16
-  sg:
-    - ssm:/path/to/ssm
-# The node type and initial number of nodes
-nodeType: RA3_4XLARGE
-numberOfNodes: 2
-
-# Controls whether or not the cluster logs user audit activity to S3
-# Note that enabling this will result in a new S3 bucket being created
-# specifically for user audit logs. Due to Redshift limitations, this
-# S3 bucket will use S3/AES-256 encryption instead of KMS CMK.
-enableAuditLoggingToS3: true
-
-databaseUsers:
-  - userName: "serviceuserGlue"
-    dbName: "default_db"
-    secretRotationDays: 90
-    secretAccessRoles:
-      - name: "test-arn"
-  - userName: "serviceuserQuicksight"
-    dbName: "default_db"
-    secretRotationDays: 90
-
-# The list of scheduled actions to pause and/or resume cluster
-scheduledActions:
-  # Pause cluster every Friday at 6pm ET starting April 13, 2022 until Dec 31, 2099
-  - name: pause-cluster
-    enable: True
-    # Target Action must be either of: "pauseCluster" or "resumeCluster". resizeCluster is not supported yet.
-    targetAction: pauseCluster
-    # Specify the action schedule in cron format cron(Minutes Hours Day-of-month Month Day-of-week Year).
-    schedule: cron(0 22 ? * FRI *)
-    # Start Date and Time in UTC format when the schedule becomes active. This must be a future date-time.
-    startTime: "2023-12-31T00:00:00Z"
-    # End Date and Time in UTC format after which the schedule is no longer active. This must be a future date-time later than start date.
-    endTime: "2099-12-31T00:00:00Z"
-
-  - name: resume-cluster
-    # Resume cluster every Monday at 7am ET starting April 13, 2022 until Dec 31, 2099
-    enable: True
-    # Target Action must be either of: "pauseCluster" or "resumeCluster". resizeCluster is not supported yet.
-    targetAction: resumeCluster
-    # Specify the action schedule in cron format cron(Minutes Hours Day-of-month Month Day-of-week Year).
-    schedule: cron(0 12 ? * MON *)
-    # Start Date and Time in UTC format when the schedule becomes active. This must be a future date-time.
-    startTime: "2023-12-31T00:00:00Z"
-    # End Date and Time in UTC format after which the schedule is no longer active. This must be a future date-time later than start date.
-    endTime: "2099-12-31T00:00:00Z"
-
-# Cluster and Scheduled Action event notification configs
-eventNotifications:
-  # List of emails to which email notifications will be sent
-  # If not specified, an SNS topic is still created and
-  # other types of subscriptions can be directly added.
-  email:
-    - example@example.com
-  # Event severity level
-  # "ERROR" | "INFO"
-  severity: INFO
-  # Event categories to be included
-  # "configuration" | "management" | "monitoring" | "security" | "pending"
-  eventCategories:
-    - management
-    - security
+# Contents available via above link
+--8<-- "target/docs/packages/apps/analytics/datawarehouse-app/sample_configs/sample-config-minimal.yaml"
 ```
+
+#### Comprehensive Configuration
+
+Deploys a multi-node Redshift cluster with SAML federation, scheduled pause/resume, audit logging, database users with secret rotation, event notifications, workload management, parameter group tuning, and VPC networking. Use this as a reference when you need full control over cluster sizing, access patterns, cost management, and operational automation.
+
+[sample-config-comprehensive.yaml](sample_configs/sample-config-comprehensive.yaml)
+
+```yaml
+# Contents available via above link
+--8<-- "target/docs/packages/apps/analytics/datawarehouse-app/sample_configs/sample-config-comprehensive.yaml"
+```
+
+#### External Public Access Block Management
+
+Deploys a Redshift cluster with S3 bucket public access block management delegated externally — for example, via account-level S3 settings or SCPs. When `publicAccessBlockManagedExternally` is set to `true`, CDK omits the `PutBucketPublicAccessBlock` API call on provisioned S3 buckets, avoiding conflicts in environments where SCPs restrict that action. Use this variant when your organization enforces public access restrictions at the account or organizational level rather than per-bucket.
+
+[sample-config-public-access-block-external.yaml](sample_configs/sample-config-public-access-block-external.yaml)
+
+```yaml
+# Contents available via above link
+--8<-- "target/docs/packages/apps/analytics/datawarehouse-app/sample_configs/sample-config-public-access-block-external.yaml"
+```
+
+---
+
+[Config Schema Docs](SCHEMA.md)