@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.100

package/bin/skills/lakehouse-doc-en/references/DataQuality.md

@@ -0,0 +1,99 @@
+# Data Quality Management
+
+In the era of big data, data quality management is a critical step to ensure data correctness, validity, and consistency. Through data quality management, we can clean, process, and optimize massive datasets, thereby enhancing data value density to better serve business needs. The data quality module provides you with comprehensive monitoring and evaluation of data quality across six dimensions: completeness, uniqueness, consistency, accuracy, validity, and timeliness. With the data quality module, you can achieve continuous improvement and optimization of data quality.
+
+## Overview
+
+The data quality overview page provides an intuitive data quality monitoring dashboard, making it easy to view the overall status of quality rules and validation runs. Below are detailed explanations of some indicators:
+
+![](.topwrite/assets/image_1760004305684.png)
+
+The definitions of some indicators are as follows:
+
+* Quality rule count: The total number of quality rules configured across all workspaces of the service instance, including disabled rules.
+* Total covered tables: The number of tables with quality rules configured across all workspaces of the service instance.
+* Detection run count: The number of quality rule runs in the past month.
+* Validation pass rate: Number of passes / total number of validations in the past month.
+* Quality rule distribution: The distribution of quality rules across all workspaces of the service instance, grouped by owner or workspace.
+* Validation result distribution: The distribution of validation result statuses for quality rules across all workspaces.
+* High-quality tables: Tables with quality rules configured that have passed validation for the past 7 consecutive days.
+* Results needing attention today: Quality rules that failed validation.
+
+## Quality Rules
+
+The quality rules page displays all your configured quality rules in a list format. Use the filter area at the top to quickly find the rules you need.
+
+![](.topwrite/assets/image_1760004360956.png)
+
+Use the top filter area to perform detailed searches.
+
+### Create a New Quality Rule
+
+1. In the quality rules list, or in the rule list of a validation object, click the "Create Rule" button to enter the new quality rule page.
+2. Fill in the required configuration items, such as data source, workspace, validation object, owner, description, parameter configuration, value filtering, validation method, expected result, trigger method, execution cluster, and timeout duration.
+ ![](.topwrite/assets/image_1760004384816.png)
+
+| **Configuration Item**       | **Configuration Description**                                                                                                                                                                                                                                |
+| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Data Source           | The type of data source. Currently only Lakehouse data sources are supported.                                                                                                                                                                                                                |
+| Workspace          | The workspace to which the validation object belongs.                                                                                                                                                                                                                             |
+| Validation Object          | When the validation object is a Lakehouse table, select its Schema and name (table name, view name, etc.).                                                                                                                                                                                              |
+| Owner           | The person responsible for the quality rule, affecting who receives alerts.                                                                                                                                                                                                                        |
+| Description            | A description defined for the quality rule.                                                                                                                                                                                                                              |
+| Parameter Configuration          | In the quality rule, when using value filtering and custom SQL to compute metric values, you can reference predefined dynamic parameter values. For example, define a parameter: partition = $[yyyyMMdd].                                                                                                                                                           |
+| Value Filtering          | Used to filter the range of objects to validate, such as filtering by partition. Supports parameter references: dt = ${partition}.                                                                                                                                                                                       |
+| Validation Method - Built-in Metrics   | System built-in validation metrics. Select as needed.                                                                                                                                                                                                                        |
+| Validation Method - Custom SQL | If the built-in metrics do not meet your needs, you can compute metric values using custom SQL. Important: The result of custom SQL must be a single numeric value to enable comparison.                                                                                                                                                               |
+| Expected Result          | Define the expected result for the metric value.                                                                                                                                                                                                                              |
+| Trigger Method          | Configures how the quality rule is triggered to run. 1. Scheduled trigger: The system triggers a single validation run at the specified time. 2. Periodic task trigger: Triggered by an associated periodic task instance after the instance runs successfully. For periodic scheduling triggers, there are two scheduling blocking options: A. Strong blocking: If validation fails, the associated task instance is marked as failed, blocking downstream instances. B. Non-blocking: The quality rule runs as a bypass and does not affect the task instance's run status. 3. Manual trigger: Manually triggered as needed. |
+| Execution Cluster          | Specifies the compute cluster within the workspace for running the quality rule.                                                                                                                                                                                                                     |
+| Timeout Duration          | If the quality rule validation does not complete within the set timeout, it will be automatically canceled by the system.                                                                                                                                                                                                       |
+
+After filling in the required configuration items, click the "Confirm" button to create the rule.
+
+3. Example: If you want to check whether the record count of a table meets expectations, you can select "Record Count" as the built-in metric, set value filtering to a specific partition, set the expected result to a specific numeric value, trigger method to scheduled trigger, and execution cluster to your workspace's compute cluster.
+
+### Test Run a Quality Rule
+
+After creating a quality rule, it is recommended to use the "Test Run" feature to verify the correctness of the configuration. After a successful test run, you can view the results to adjust the rule.
+
+![](.topwrite/assets/image_1760004576132.png)
+
+### View Test Run Results
+
+After clicking "Test Run", follow the prompts to click "View Results" to see the test run validation results.
+
+![](.topwrite/assets/image_1699875850066.png)
+
+### Configure Monitoring Alerts
+
+To ensure data quality issues are addressed promptly, you can configure monitoring alerts for quality rules. There are two ways:
+
+1. **Enable global quality monitoring alerts**: In the monitoring and alerting module, search for "Data Quality Check Failed" and enable the system's built-in global quality validation monitoring rule.
+   ![](.topwrite/assets/image_1699875920709.png)
+2. **Configure custom quality monitoring alerts**: Create custom monitoring rules, select "Quality Rule Validation Failed" as the monitoring message, and set filter conditions such as workspace or specific validation object.
+   ![](.topwrite/assets/image_1699875932007.png)
+
+## Validation Objects
+
+On the validation objects page, you can manage all quality rules by the validation object (table) dimension. Use the search filter area to quickly locate specific rules.
+
+![](.topwrite/assets/image_1699875683208.png)
+
+## Validation Results
+
+### Validation Results List
+
+On the validation results list page, you can view the run status of all quality rules. Use the search and filter area to precisely find the required validation results.
+
+![](.topwrite/assets/image_1699875971234.png)
+
+### Operations on Validation Results
+
+For each validation result, you can perform the following operations:
+
+* **Terminate**: Cancel the current validation run.
+* **Set Success/Set Failure**: Manually set the validation result to success or failure.
+* **Revalidate**: Trigger the quality validation to run again. Note: If a rule that previously failed validation passes upon revalidation, no new monitoring alert will be sent.
+
+With the above features, you can effectively manage and monitor data quality, ensuring that the value of data in business applications is fully realized.

package/bin/skills/lakehouse-doc-en/references/DataSourceConfigGuide.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/DataSource_ADBMySQL.md

@@ -0,0 +1,35 @@
+# ADB for MySQL Data Source Configuration Guide
+
+## Overview
+
+ADB for MySQL is an analytical database service provided by Alibaba Cloud. It is based on the MySQL protocol and supports high concurrency, high availability online transaction processing, and online analytical processing. Configuring the ADB for MySQL data source allows you to achieve efficient data synchronization and analysis, meeting the needs of various business scenarios.
+
+## Parameter Configuration
+
+When configuring the ADB for MySQL data source, you need to provide the following information to ensure a successful connection to the ADB service:
+
+* **Data Source Name**: Specify a unique and easily recognizable name for your ADB for MySQL data source.
+* **JDBC Connection Address**: Provide the JDBC connection address of the ADB for MySQL instance, usually in the format `jdbc:mysql://host:port/database`.
+* **Username**: Enter the username used to connect to the ADB for MySQL instance.
+* **Password**: Provide the database password corresponding to the username.
+* **Data Source Description**: (Optional) Add descriptive information for the data source to help you or other administrators understand the purpose or characteristics of the data source.
+
+## Connection Configuration
+
+In terms of connection configuration, you can choose one of the following connection methods:
+
+* **Direct Connection**: Ensure that the connection information you entered is accessible over the public network. If the source has enabled an IP access whitelist, make sure the outbound IP address of the data integration service has been added to the whitelist. For specific IP addresses, please contact technical support.
+* **Via SSH Tunnel**: To enhance security, you can choose to connect to ADB for MySQL via an SSH tunnel. Enable this option and provide the following SSH tunnel-related configuration information:
+  * **SSH Server Address**: Provide the IP address or domain name of the SSH server.
+  * **SSH Port**: Specify the port number on which the SSH server is listening, usually `22`.
+  * **Username**: Provide the login username for the SSH server.
+  * **Password**: Provide the password corresponding to the SSH server login username.
+
+## Notes
+
+* Ensure that all provided connection information is accurate and that the ADB for MySQL service is accessible.
+* Protect your database credentials and SSH credentials to prevent them from being disclosed to unauthorized personnel.
+* Regularly check and update your data source configuration to adapt to changes in the database structure or new security requirements.
+* Monitor the running status of data synchronization tasks to promptly identify and resolve any issues that may arise.
+
+Once the configuration is complete, you can select this ADB for MySQL data source in data synchronization tasks to perform data import or export operations. Connecting via an SSH tunnel can enhance the security of data transmission, especially when handling sensitive data.

package/bin/skills/lakehouse-doc-en/references/DataSource_ADB_PostgreSQL.md

@@ -0,0 +1,36 @@
+# ADB for PostgreSQL Data Source Configuration Guide
+
+## Overview
+
+ADB for PostgreSQL is a relational database service provided by Alibaba Cloud. It is based on the open-source PostgreSQL database, supports SQL and NoSQL data models, and offers high-performance, high-availability, and elastic scaling data storage solutions. By configuring the ADB for PostgreSQL data source, you can achieve efficient data synchronization and analysis to meet complex business needs.
+
+## Parameter Configuration
+
+When configuring the ADB for PostgreSQL data source, you need to provide the following information to ensure a successful connection to the ADB service:
+
+* **Data Source Name**: Specify a unique and easily recognizable name for your ADB for PostgreSQL data source.
+* **JDBC Connection Address**: Provide the JDBC connection address of the ADB for PostgreSQL instance, usually in the format `jdbc:postgresql://host:port/database`.
+* **Username**: Enter the username used to connect to the ADB for PostgreSQL instance.
+* **Password**: Provide the database password corresponding to the username.
+* **Schema**: Specify the default schema to connect to, such as `public` or any other schema you need to use.
+* **Data Source Description**: (Optional) Add descriptive information for the data source to help you or other administrators understand the purpose or characteristics of the data source.
+
+## Connection Configuration
+
+In terms of connection configuration, you can choose one of the following connection methods:
+
+* **Direct Connection**: Ensure that the connection information you entered is accessible over the public network. If the source has enabled an IP access whitelist, make sure the outbound IP address of the data integration service has been added to the whitelist. For specific IP addresses, please contact technical support.
+* **Via SSH Tunnel**: To enhance security, you can choose to connect to ADB for PostgreSQL via an SSH tunnel. Enable this option and provide the following SSH tunnel-related configuration information:
+  * **SSH Server Address**: Provide the IP address or domain name of the SSH server.
+  * **SSH Port**: Specify the port number that the SSH server listens to, usually `22`.
+  * **Username**: Provide the login username for the SSH server.
+  * **Password**: Provide the password corresponding to the SSH server login username.
+
+## Notes
+
+* Ensure that all provided connection information is accurate and that the ADB for PostgreSQL service is accessible.
+* Protect your database credential information and SSH credential information to avoid leakage to unauthorized personnel.
+* Regularly check and update your data source configuration to adapt to changes in the database structure or new security requirements.
+* Monitor the running status of data synchronization tasks to promptly identify and resolve any potential issues.
+
+Once the configuration is complete, you can select this ADB for PostgreSQL data source in data synchronization tasks to perform data import or export operations. Connecting via SSH tunnel can enhance the security of data transmission, especially when handling sensitive data.

package/bin/skills/lakehouse-doc-en/references/DataSource_AMQP.md

@@ -0,0 +1,37 @@
+# AMQP Data Source Configuration Guide
+
+## Overview
+
+AMQP (Advanced Message Queuing Protocol) is an asynchronous messaging protocol widely used in IoT, microservice architecture, and enterprise system integration due to its high reliability and flexibility. By configuring an AMQP data source, you can achieve efficient data exchange with systems such as Alibaba Cloud IoT Platform.
+
+## Parameter Configuration
+
+When configuring an AMQP data source, you need to provide the following information to ensure a successful connection to the AMQP service:
+
+* Data Source Name: Specify a unique and easily recognizable name for your AMQP data source.
+* Host: The access domain name of the AMQP service, usually in the format `${uid}.iot-amqp.${YourRegionId}.aliyuncs.com`. For details, please refer to [Alibaba Cloud Manage Instance Endpoint](https://help.aliyun.com/zh/iot/user-guide/manage-the-endpoint-of-an-instance#task-1545804)
+* Authentication Method: Currently supports authentication via RAM account and role-based authentication based on RAM account. Please choose the appropriate authentication method.
+* IOT INSTANCE ID: The ID of the IoT instance. You can view the current instance ID on the instance overview page of the [Alibaba Cloud IoT Platform Console](https://iot.console.aliyun.com/).
+* Data Source Description: (Optional) Add descriptive information to the data source to help you or other administrators understand the purpose or characteristics of the data source.
+
+## Authentication Method Configuration
+
+Choose the authentication method that suits you and configure the following parameters according to the actual situation:
+
+* Connect via RAM Account: This mode directly uses the RAM account for connection and requires the following parameters:
+  * AccessKey ID: The AccessKey ID and AccessKey Secret of the Alibaba Cloud main account or the corresponding RAM user. Log in to the Alibaba Cloud IoT Platform Console, move the mouse over the account avatar, and then click AccessKey Management to obtain the AccessKey ID and AccessKey Secret.
+  * AccessKey Secret: The AccessKey ID and AccessKey Secret of the Alibaba Cloud main account or the corresponding RAM user. Log in to the Alibaba Cloud IoT Platform Console, move the mouse over the account avatar, and then click AccessKey Management to obtain the AccessKey ID and AccessKey Secret.
+* RAM User Authorized by RAM Role: This mode is used to authorize another RAM account to access the data source through a RAM role and requires the following parameters:
+  * STS Endpoint: The access point for the STS Token acquisition service. For specific information, please refer to [Alibaba Cloud Service Endpoint](https://www.alibabacloud.com/help/zh/ram/developer-reference/api-sts-2015-04-01-endpoint)
+  * STS AccessKey ID: The AccessKey ID and AccessKey Secret of the RAM user who assumes the RAM role under the Alibaba Cloud account of the data-holding enterprise.
+  * STS Role ARN: The ARN of the RAM role to be assumed under the Alibaba Cloud account of the data-holding enterprise, in the format `acs:ram::<account-id>:role/<role-name>`
+
+## Notes
+
+* Ensure the security and stability of the AMQP server, and configure authentication and authorization mechanisms reasonably.
+* Protect your credential information to avoid leakage to unauthorized personnel.
+* When configuring, please refer to the relevant documentation and support resources of the AMQP server to ensure correctness.
+* After configuration, you can use the "Test Connectivity" function to verify the accessibility of the data source and the correctness of the configuration information.
+* After verification, you can select this AMQP data source in the data synchronization task to perform data reading and exporting.
+
+Please ensure that you have read and followed the above guidelines to successfully complete the configuration of the AMQP data source. If you need further assistance, please refer to the relevant documentation or contact technical support.

package/bin/skills/lakehouse-doc-en/references/DataSource_Amazon_DocumentDB.md

@@ -0,0 +1,113 @@
+# AWS DocumentDB Data Source Configuration Guide
+
+## Overview
+
+AWS DocumentDB (with MongoDB compatibility) is a fully managed document database service provided by Amazon Web Services that offers a MongoDB-compatible database engine in the cloud. By configuring an AWS DocumentDB data source, you can easily achieve data synchronization and integration with other systems while enjoying the high availability, scalability, and security of a cloud-native database.
+
+## Parameter Configuration
+
+When configuring an AWS DocumentDB data source, you need to provide the following information to ensure a successful connection to the database:
+
+### Basic Configuration
+
+* **Data Source Name** (Required): Specify a unique and easily identifiable name for your AWS DocumentDB data source. It is recommended to use meaningful naming conventions, such as "Behavioral Data Source", to facilitate subsequent management and maintenance.
+* **URL Address** (Required): Provide the connection endpoint address of your AWS DocumentDB cluster in the format `hostname:port`. For example: `docdb-example.cluster-abc123.us-east-1.docdb.amazonaws.com:27017`. This address can be obtained from the DocumentDB cluster details page in the AWS Console.
+
+### Authentication Information
+
+* **Username** (Required): Provide the master username or a database user with appropriate permissions for connecting to AWS DocumentDB. The default master username is typically `admin`, though you can also use custom-created users.
+* **Password** (Required): Provide the database password corresponding to the username. Please ensure password security and avoid using weak passwords. The system will encrypt and store the password.
+
+### Security Configuration
+
+* **CA Certificate** (Optional): AWS DocumentDB enables TLS encrypted connections by default to ensure secure data transmission. If TLS is enabled on your cluster, you need to upload the CA certificate file (rds-combined-ca-bundle.pem) provided by AWS. Click the "Upload File" button to select the local CA certificate file.
+
+### Connection Method
+
+* **Connect via SSH Tunnel** (Optional): To enhance security, especially when the DocumentDB cluster is located in a private VPC, you can choose to connect through an SSH tunnel. When this option is enabled, data will be transmitted to DocumentDB through a secure SSH channel. You need to provide information about a bastion host that has public network access and can reach the DocumentDB cluster:
+
+  * SSH server IP address
+  * SSH port (default 22)
+  * SSH username and key or password
+
+### Additional Configuration
+
+* **Data Source Description** (Optional): Add descriptive information for the data source, such as "Production Environment User Behavior Database", to help you or other administrators understand the purpose, environment, or business scenario of this data source.
+
+### Authorize Workspace Access
+
+* **Specific Workspace**: Authorize only specific workspaces to use this data source, suitable for scenarios requiring strict permission control.
+* **All Workspaces** (Recommended): Allow all workspaces to use this data source, facilitating cross-team collaboration and data sharing.
+
+## Connection Configuration Instructions
+
+### Network Access Configuration
+
+Depending on your AWS DocumentDB deployment architecture, you can choose from the following connection methods:
+
+#### 1. Public Network Direct Connection (Requires VPC and Security Group Configuration)
+
+If your DocumentDB cluster is configured for public network access (via VPN or Direct Connect), ensure that:
+
+* The security group inbound rules of the DocumentDB cluster allow the data integration platform's egress IP addresses to access port 27017
+* VPC network ACL rules allow the corresponding traffic to pass through
+* For the specific list of platform egress IP addresses, please contact technical support personnel
+
+#### 2. Connect via SSH Tunnel (Recommended)
+
+For DocumentDB clusters located in a private VPC, using SSH tunnel connection is strongly recommended:
+
+* Deploy an EC2 instance as a bastion host in the same VPC as the DocumentDB cluster
+* This EC2 instance must have a public IP or Elastic IP
+* Ensure the security group of the EC2 instance allows SSH connections (port 22) from the data integration platform
+* The security group of the DocumentDB cluster must allow access from the EC2 instance (port 27017)
+* This approach prevents direct exposure of DocumentDB to the public network, enhancing security
+
+## Connection Testing
+
+After completing all configurations, click the "Test Connection" button at the bottom of the page to verify the connection configuration:
+
+* The system will attempt to connect to the AWS DocumentDB cluster using the provided parameters
+* Verify network reachability, authentication information, and permission settings
+* After a successful test, you can save the data source configuration and use it in data synchronization tasks
+
+## Important Considerations
+
+### Security Recommendations
+
+* **TLS Encryption**: AWS DocumentDB enables TLS by default. It is strongly recommended to keep it enabled and upload the correct CA certificate to ensure encrypted data transmission.
+* **Principle of Least Privilege**: Create dedicated database users for data synchronization with only the necessary read/write permissions; avoid using the master account.
+* **Password Management**: Regularly rotate database passwords and avoid reusing the same password across multiple systems.
+* **Network Isolation**: Prioritize SSH tunnels or VPC private connections to avoid exposing the database directly to the public network.
+
+### Performance Optimization
+
+* **Connection Pool Configuration**: The system automatically manages connection pools to ensure efficient connection reuse.
+* **Network Latency**: Choose data integration services in the same region as the DocumentDB cluster to reduce network latency.
+* **Read Performance**: For large-scale data synchronization, consider reading data from replica instances to reduce pressure on the primary instance.
+
+### Operations Recommendations
+
+* **Monitor Synchronization Status**: Regularly check the running status and logs of data synchronization tasks to promptly identify and resolve potential issues.
+* **Backup Strategy**: Ensure that automatic backups are configured for the DocumentDB cluster to avoid data loss risks.
+* **Version Compatibility**: Note the compatibility between AWS DocumentDB versions and the MongoDB protocol; some advanced features may have differences.
+* **Quota Limits**: Understand AWS DocumentDB's quotas for connections, storage capacity, etc., to avoid service interruptions due to exceeding limits.
+
+### Troubleshooting
+
+If the connection test fails, please check the following common issues:
+
+1. **Network Connectivity**: Use the `telnet` or `nc` command to test network connectivity from the data integration platform to DocumentDB
+2. **Security Group Configuration**: Verify that security group rules for DocumentDB and EC2 (if using SSH tunnel) are correctly configured
+3. **Authentication Information**: Verify that the username and password are correct and that the user has permissions to access the target database
+4. **CA Certificate**: If TLS is enabled, ensure the uploaded CA certificate file is valid and not expired
+5. **SSH Configuration**: If using SSH tunnel, verify that the bastion host's SSH service is functioning properly and that the key or password is correct
+
+## Complete Configuration
+
+After completing all parameter configurations and passing the connection test, you can:
+
+* Select this AWS DocumentDB data source in data synchronization tasks
+* Configure specific collections and field mappings
+* Set scheduling strategies and incremental update rules for data synchronization
+* Execute data import or export operations

package/bin/skills/lakehouse-doc-en/references/DataSource_Amazon_OpenSearch.md

@@ -0,0 +1,41 @@
+# AWS OpenSearch Data Source Configuration Guide
+
+## Overview
+
+AWS OpenSearch is a managed search and analytics engine service provided by Amazon Web Services, built on the open-source OpenSearch project. It enables real-time full-text search, log analysis, and complex data analytics. Configuring an AWS OpenSearch data source helps you efficiently operate and manage data stored in AWS OpenSearch for data synchronization and analysis tasks.
+
+## Parameter Configuration
+
+When configuring an AWS OpenSearch data source, you need to provide the following information to ensure successful connection to the AWS OpenSearch domain:
+
+* **Data Source Name**: Specify a unique and easily identifiable name for your AWS OpenSearch data source, such as `Behavioral Data Source`.
+* **Domain EndPoint**: Provide the access endpoint address of the AWS OpenSearch domain in the format `<domain-name>-<identifier>.<region>.es.amazonaws.com:443`. For example, `search-example-domain-abc123.us-east-1.es.amazonaws.com:443`.
+* **Authentication Method**: Select `Access Key ID & Access Key Secret` as the authentication method.
+* **Access Key ID**: Provide the Access Key ID of an AWS IAM user with OpenSearch access permissions.
+* **Access Key Secret**: Provide the corresponding Access Key Secret credential.
+* **Connect via SSH Tunnel**: (Optional) When the OpenSearch domain is located within a VPC and cannot be directly accessed, you can enable SSH Tunnel to establish a secure connection.
+* **Data Source Description**: (Optional) Add descriptive information for the data source to help you or other administrators understand the purpose or characteristics of this data source.
+* **Authorize for Workspace Usage**: Select the visibility scope of the data source, you can choose `Specified Workspace` or `All Workspaces`.
+
+## Connection Configuration
+
+Regarding connection configuration, you need to pay attention to the following matters:
+
+* **Direct Connection**: Ensure that your AWS OpenSearch domain is configured with appropriate access policies. If the domain has IP address access control enabled, please ensure that the egress IP addresses of the data integration service have been added to the allow list. For specific IP addresses, please contact technical support personnel.
+* **SSH Tunnel**: If the OpenSearch domain is deployed within a VPC, you can enable SSH Tunnel mode to establish a secure connection through a bastion host.
+
+## Notes
+
+* Ensure that all provided connection information is accurate and that the AWS OpenSearch domain is accessible.
+* Protect your Access Key credentials to prevent disclosure to unauthorized personnel. The Access Key Secret will be stored encrypted.
+* Ensure that the IAM user has the necessary permissions to access the OpenSearch domain. It is recommended to attach the `AmazonOpenSearchServiceFullAccess` policy or a custom policy.
+* Regularly review and update your data source configuration to adapt to changes in domain structure or new security requirements.
+* Monitor the operational status of data synchronization tasks to identify and resolve potential issues in a timely manner.
+* Be aware of costs associated with AWS data transfer, especially in cross-region access scenarios.
+
+After configuration is complete, you can select this AWS OpenSearch data source in data synchronization tasks to perform data import or export operations. Through appropriate connection methods, you can achieve rapid data transfer and improve data processing efficiency.
+
+## Related Documentation
+
+* [AWS OpenSearch Service Official Documentation](https://docs.aws.amazon.com/opensearch-service/)
+* [AWS IAM User Management](https://docs.aws.amazon.com/IAM/latest/UserGuide/)

package/bin/skills/lakehouse-doc-en/references/DataSource_Aurora_MySQL.md

@@ -0,0 +1,35 @@
+# Aurora MySQL Data Source Configuration Guide
+
+## Overview
+
+Amazon Aurora MySQL is a MySQL-compatible cloud-native relational database that offers high performance, scalability, and a MySQL-compatible programming environment. By configuring the Aurora MySQL data source, you can achieve data synchronization and integration with other systems, supporting complex data analysis and transaction processing.
+
+## Parameter Configuration
+
+When configuring the Aurora MySQL data source, you need to provide the following information to ensure a successful connection to the Aurora MySQL database:
+
+* **Data Source Name**: Specify a unique and easily recognizable name for your Aurora MySQL data source.
+* **JDBC Connection URL**: Provide the JDBC connection URL for the Aurora MySQL database, typically in the format `jdbc:mysql://host:port/database`. For example, `jdbc:mysql://aurora-mysql-host:3306/mydatabase`.
+* **Username**: Enter the username used to connect to the Aurora MySQL database.
+* **Password**: Provide the database password corresponding to the username.
+* **Data Source Description**: (Optional) Add descriptive information for the data source to help you or other administrators understand the purpose or characteristics of the data source.
+
+## Connection Configuration
+
+In terms of connection configuration, you can choose one of the following connection methods:
+
+* **Direct Connection**: Ensure that the connection information you enter is accessible over the public network. If the source has an IP access whitelist enabled, make sure the egress IP address of the data integration service has been added to the whitelist. For specific IP addresses, please contact technical support.
+* **Via SSH Tunnel**: To enhance security, you can choose to connect to Aurora MySQL via an SSH tunnel. Enable this option and provide the following SSH tunnel-related configuration information:
+  * **SSH Server Address**: Provide the IP address or domain name of the SSH server.
+  * **SSH Port**: Specify the port number on which the SSH server is listening, typically `22`.
+  * **Username**: Provide the login username for the SSH server.
+  * **Password**: Provide the password corresponding to the SSH server login username.
+
+## Notes
+
+* Ensure that all provided connection information is accurate and that the Aurora MySQL service is accessible.
+* Protect your database credentials and SSH credentials to prevent them from being disclosed to unauthorized personnel.
+* Regularly check and update your data source configuration to adapt to changes in the database structure or new security requirements.
+* Monitor the running status of data synchronization tasks to promptly identify and resolve any potential issues.
+
+Once the configuration is complete, you can select this Aurora MySQL data source in data synchronization tasks to perform data import or export operations. Connecting via an SSH tunnel can enhance the security of data transmission, especially when handling sensitive data. The direct connection method provides a more straightforward data transmission path.

package/bin/skills/lakehouse-doc-en/references/DataSource_Aurora_PostgreSQL.md

@@ -0,0 +1,36 @@
+# Aurora PostgreSQL Data Source Configuration Guide
+
+## Overview
+
+Amazon Aurora PostgreSQL is a cloud-native relational database compatible with PostgreSQL, offering high performance, scalability, and advanced security features. By configuring the Aurora PostgreSQL data source, you can achieve efficient data synchronization and analysis, meeting enterprise-level data management needs.
+
+## Parameter Configuration
+
+When configuring the Aurora PostgreSQL data source, you need to provide the following information to ensure a successful connection to the Aurora PostgreSQL database:
+
+* **Data Source Name**: Specify a unique and easily recognizable name for your Aurora PostgreSQL data source.
+* **JDBC Connection URL**: Provide the JDBC connection URL of the Aurora PostgreSQL database, usually in the format `jdbc:postgresql://host:port/database`. For example, `jdbc:postgresql://aurora-postgresql-host:5432/mydatabase`.
+* **Username**: Enter the username used to connect to the Aurora PostgreSQL database.
+* **Password**: Provide the database password corresponding to the username.
+* **Schema**: Specify the default schema to connect to, such as `public` or any other schema you need to use.
+* **Data Source Description**: (Optional) Add descriptive information for the data source to help you or other administrators understand the purpose or characteristics of the data source.
+
+## Connection Configuration
+
+In terms of connection configuration, you can choose one of the following connection methods:
+
+* **Direct Connection**: Ensure that the connection information you entered is accessible over the public network. If the source has an IP access whitelist enabled, make sure the outbound IP address of the data integration service has been added to the whitelist. For specific IP addresses, please contact technical support.
+* **Via SSH Tunnel**: To enhance security, you can choose to connect to Aurora PostgreSQL via an SSH tunnel. Enable this option and provide the following SSH tunnel-related configuration information:
+  * **SSH Server Address**: Provide the IP address or domain name of the SSH server.
+  * **SSH Port**: Specify the port number the SSH server listens to, usually `22`.
+  * **Username**: Provide the login username for the SSH server.
+  * **Password**: Provide the password corresponding to the SSH server login username.
+
+## Notes
+
+* Ensure that all provided connection information is accurate and that the Aurora PostgreSQL service is accessible.
+* Protect your database credentials and SSH credentials to avoid leakage to unauthorized personnel.
+* Regularly check and update your data source configuration to adapt to changes in the database structure or new security requirements.
+* Monitor the running status of data synchronization tasks to promptly identify and resolve any potential issues.
+
+After the configuration is complete, you can select this Aurora PostgreSQL data source in data synchronization tasks to perform data import or export operations. Connecting via SSH tunnel can enhance the security of data transmission, especially when handling sensitive data. The direct connection method provides a more straightforward data transmission path.

package/bin/skills/lakehouse-doc-en/references/DataSource_AutoMQ.md

@@ -0,0 +1,124 @@
+# AutoMQ Data Source Configuration Guide
+
+## Overview
+
+AutoMQ is a next-generation cloud-native streaming data platform that is 100% compatible with the Apache Kafka protocol. Through innovative storage architecture design, AutoMQ separates data persistence to cloud storage (such as Amazon S3), achieving a 10x cost reduction and 100x elasticity improvement compared to traditional Kafka, while maintaining single-digit millisecond latency. AutoMQ is ideal for building real-time data pipelines, stream processing, and event-driven architectures.
+
+## Parameter Configuration
+
+When configuring an AutoMQ data source, you need to provide the following information to ensure successful connection to the AutoMQ cluster:
+
+### Basic Configuration
+
+* **Data Source Name**: Specify a unique and easily identifiable name for your AutoMQ data source. For example: `UserBehaviorAutoMQ`, `OrderStreamAutoMQ`. It is recommended to use naming conventions that clearly express the data purpose.
+
+* **AutoMQ Connection Configuration**: Enter the Bootstrap Server address of the AutoMQ cluster in the format `host1:port,host2:port,host3:port`.
+
+  * Example: `automq-broker-01.example.com:9092,automq-broker-02.example.com:9092`
+
+### Security Authentication Configuration
+
+* **AutoMQ Security Authentication Protocol**: Select the appropriate authentication method based on your cluster security policy. AutoMQ supports the following authentication protocols:
+
+  * **No Authentication**: Suitable for development and testing environments or protected internal network environments
+  * **SASL\_PLAINTEXT**: Uses SASL mechanism for username and password authentication, but data transmission is not encrypted
+  * **SASL\_SSL**: Uses SASL authentication with SSL/TLS encrypted data transmission, recommended for production environments
+
+### SASL Authentication Configuration (Required when SASL authentication is selected)
+
+* **JAAS Configuration**: Provide the Java Authentication and Authorization Service (JAAS) configuration string. The configuration format for different authentication mechanisms is as follows:
+
+  **PLAIN Mechanism Example**:
+
+  ```
+  org.apache.kafka.common.security.plain.PlainLoginModule required username="automq_user" password="your_password";
+  ```
+
+  **SCRAM-SHA-256 Mechanism Example**:
+
+  ```
+  org.apache.kafka.common.security.scram.ScramLoginModule required username="automq_user" password="your_password";
+  ```
+
+  **SCRAM-SHA-512 Mechanism Example**:
+
+  ```
+  org.apache.kafka.common.security.scram.ScramLoginModule required username="automq_user" password="your_password";
+  ```
+
+### SSL/TLS Certificate Configuration (Required when SSL encryption is selected)
+
+* **Truststore (CA Certificate) File**: Upload the Truststore file to verify the identity of the AutoMQ server.
+
+  * Supports JKS and PKCS12 formats
+
+* **Truststore Password**: Provide the password to access the Truststore file to ensure certificate security.
+
+* **Keystore (Private Key) File** (Optional): If the AutoMQ cluster has mutual TLS authentication (mTLS) enabled, you need to specify the client Keystore file path.
+
+  * Example: `/security/automq.client.keystore.jks`
+
+* **Keystore Password** (Optional): Provide the password to access the Keystore file.
+
+### Advanced Configuration
+
+* **Data Source Description** (Optional): Add detailed descriptions of the data source purpose and business scenarios to facilitate team member understanding and management. For example: "Production environment user behavior data stream for real-time recommendation systems".
+
+## Connection Configuration Instructions
+
+### Network Connectivity
+
+When configuring AutoMQ connections, please note the following:
+
+* **Public Network Access**: If the AutoMQ cluster is deployed in the cloud, ensure that the Bootstrap Server address you entered is accessible via the public network. If the cluster has an IP whitelist configured, add the egress IP address of the data integration service to the whitelist. Please contact technical support personnel for the specific IP address list.
+* **VPC Internal Network Access**: If the data integration service and AutoMQ cluster are deployed in the same VPC or have VPC peering configured, it is recommended to use internal network addresses to reduce network latency and transmission costs.
+
+### Workspace Authorization
+
+* **Specify Workspace**: Authorize the data source for use by specific workspaces, suitable for scenarios requiring strict permission control.
+* **All Workspaces**: Allow all workspaces to access this data source, facilitating cross-team collaboration and data sharing.
+
+It is recommended to choose the appropriate authorization scope based on your data security policy and team collaboration needs.
+
+## Connection Testing
+
+After completing the configuration, click the "Test Connection" button to verify the correctness of the configuration:
+
+* **Test Successful**: Indicates that the configuration parameters are correct and a successful connection to the AutoMQ cluster can be established. You can proceed to create the data source.
+
+* **Test Failed**: Please check the following common issues:
+
+  * Is the Bootstrap Server address format correct
+  * Is network connectivity normal (firewall, security group rules)
+  * Is the authentication information (username, password) accurate
+  * Is the SSL certificate configuration complete and valid
+  * Does the IP whitelist configuration include the egress IP of the data integration service
+
+## Best Practices
+
+### Security Recommendations
+
+* **Production Environment Encryption**: It is strongly recommended to use the SASL\_SSL authentication protocol in production environments to ensure data transmission confidentiality and integrity.
+* **Password Management**: Properly safeguard authentication passwords and certificate keys, and avoid recording them in plaintext in code or logs.
+* **Principle of Least Privilege**: Create dedicated AutoMQ users for the data integration service and grant only the necessary Topic read and write permissions.
+* **Regular Rotation**: It is recommended to regularly update authentication credentials and SSL certificates to reduce security risks.
+
+### Operations and Monitoring Recommendations
+
+* **Metrics Monitoring**: AutoMQ natively supports Prometheus and OpenTelemetry metrics export. It is recommended to integrate with your monitoring system for real-time monitoring.
+* **Key Metrics**: Focus on core metrics such as producer-consumer latency, message backlog, and cluster throughput.
+* **Alert Configuration**: Configure timely alert notifications for abnormal situations (such as consumption latency exceeding thresholds or high disk usage).
+
+## Complete Configuration
+
+After completing all parameter configurations and passing the connection test, click "Confirm" to save the data source. You can then select this AutoMQ data source in data synchronization tasks for real-time data collection, transmission, or distribution operations, fully leveraging AutoMQ's cloud-native advantages and cost-effectiveness.
+
+## Related Resources
+
+* **AutoMQ Official Website**: <https://www.automq.com/>
+* **AutoMQ Open Source Code**: <https://github.com/AutoMQ/automq>
+* **Technical Documentation**: Visit the AutoMQ official website for detailed technical documentation and best practice guides
+
+***
+
+^

package/bin/skills/lakehouse-doc-en/references/DataSource_COS.md

@@ -0,0 +1,31 @@
+# COS Data Source Configuration Guide
+
+## Overview
+
+COS (Cloud Object Storage) is an object storage service provided by Tencent Cloud. It offers enterprises and developers a massive, secure, and cost-effective data storage solution. By configuring a COS data source, users can achieve efficient data upload, download, and synchronization, supporting various data usage scenarios.
+
+## Parameter Configuration
+
+When configuring a COS data source, you need to provide the following information to ensure a successful connection to the COS service:
+
+1. **Data Source Name**: Specify a unique and easily recognizable name for your COS data source.
+2. **regionName**: Specify the region name where the bucket is located, such as `ap-guangzhou`. This parameter is very important for the COS service as it determines the geographical location of the data storage and the access path to the service.
+3. **Bucket Name**: Specify the name of the COS bucket to connect to.
+4. **SecretId**: The SecretId of your Tencent Cloud account, used for authentication.
+5. **SecretKey**: The SecretKey corresponding to the SecretId, used for authentication.
+6. **Data Source Description** (optional): Add descriptive information for the data source to help understand the purpose or characteristics of this data source.
+
+## Connection Configuration
+
+In terms of connection configuration, you need to pay attention to the following:
+
+* **Direct Connection**: Ensure that the connection information you entered is accessible on the public network. If the source end has enabled an IP access whitelist, make sure that the egress IP address of the data integration service has been added to the whitelist. For specific IP addresses, please contact technical support.
+
+## Notes
+
+* Ensure that all provided connection information is accurate and that the COS service is accessible.
+* Protect your credential information to avoid leakage to unauthorized personnel.
+* Regularly check and update your data source configuration to adapt to changes in the bucket structure or new security requirements.
+* Monitor the running status of data synchronization tasks to promptly identify and resolve any potential issues.
+
+When configuring the COS data source, refer to Tencent Cloud's official documentation and support resources for the most accurate guidance. Once the configuration is complete, you can select this COS data source in data synchronization tasks to perform data import or export operations.