@clickzetta/cz-cli-darwin-arm64 0.3.94 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/realtime_sync.md

@@ -0,0 +1,30 @@
+# Real-time Sync Tasks
+
+The "Development" module of the Singdata Lakehouse workspace integrates the capabilities of defining and scheduling data sync tasks. With the data sync task editor in the data development module, you can quickly create data sync tasks. The following explains the creation and use of real-time sync tasks.
+
+## Create a New Data Sync Task
+
+You can create a new sync task from both the workspace and the development entry.
+
+* Create from Workspace
+  Through the console "Workspace" entry, select to create a new offline sync or real-time sync task under the "New" button on the right.
+
+* Create from Development Entry
+  You can enter the "Development" page and select to create a new sync task in the specified directory in the task area.
+
+## Real-time Sync Task Development
+
+Here, we take the real-time sync of Kafka data source to Lakehouse as an example.
+
+* Create a new real-time data sync task and configure source and target information
+  Select the Kafka data source, choose the Topic through the data object, and specify the offset and message format. Select the Lakehouse data source and the corresponding data object for the target end.
+
+  By default, the built-in fields of the Kafka Topic are used for data field mapping. If the message format in the Topic is JSON, you can also use the new calculated column method to parse the content in the value field through JSONPath rules, such as $.id, $.data.code。
+
+* Deploy to Production
+  Real-time sync tasks do not support test runs in the development state and do not require scheduling strategy configuration. You can publish the task through the "Submit" button.
+
+  After submission, you can perform start and stop operations and monitoring of real-time sync tasks in the Operations Center.
+
+
+^

package/bin/skills/lakehouse-doc-en/references/realtime_sync_and_analysis_practice.md

@@ -0,0 +1,183 @@
+# Low Latency Real-Time Data Integration and High-Performance Real-Time Analysis
+
+This document introduces the construction of a highly stable, low-latency real-time lakehouse data integration solution based on Singdata Lakehouse and high-performance real-time analysis based on the same lakehouse data.
+
+## Overview
+
+As enterprise business expands, real-time data analysis becomes crucial. It not only supports business decision-making but is also a necessary condition for driving enterprise development. To achieve real-time data analysis, enterprises typically face two technical solution choices:
+
+* Directly using the business database for BI analysis: This method is simple to operate and can be quickly implemented. However, it encounters performance bottlenecks when dealing with large-scale data and complex queries, especially when complex operations such as multi-table joins are required. The query response time will be significantly extended, failing to meet the needs of real-time analysis. Additionally, for stability and security considerations, business databases are usually not allowed to be directly used for data analysis, which limits the operational space of the data analysis team.
+* BI analysis solution based on data warehouse: This solution synchronizes data from the business database to the data warehouse, utilizing the high-performance processing capabilities of the data warehouse to support BI analysis. This method effectively solves the performance problem, but the biggest challenge lies in the real-time nature of the data. Traditional data synchronization is usually in T+1 or H+1 batch processing mode, which cannot meet the business's real-time data requirements and often requires the deployment of additional OLAP engines, further increasing costs.
+
+Singdata Lakehouse provides a highly stable, low-latency real-time lakehouse data integration solution. It can achieve real-time data synchronization from various source data sources (such as MySQL, PostgreSQL, SQL Server, etc.) and directly support real-time queries of BI reports. This solution maintains the same data in the data warehouse, supporting both real-time data writing and real-time querying of BI reports, eliminating the need for additional OLAP engine deployment in traditional solutions. This design simplifies the data architecture, reduces costs, and ensures data real-time and query efficiency, providing enterprises with an economical and efficient real-time data analysis solution.
+
+^
+
+## Real-Time Lakehouse Data Integration
+
+This section will elaborate on the productized real-time lakehouse integration solution and how to conveniently synchronize massive database table data into the Lakehouse.
+
+### Scenario and Data Description
+
+In this demonstration practice, four tables with 1 million rows of data each are prepared in the source MySQL database. The plan is to synchronize these 4 million rows of data into the target Singdata Lakehouse tables and then directly use BI reports for query analysis.
+
+:-: ![](.topwrite/assets/image_1734675535069.png =570)
+
+^
+
+This demonstration takes the common SaaS service model as an example. In a multi-tenant mode, the database adopts a sharding structure, and data exists in different sharded tables. Some tenant information will exist in the same sharded table, and different sharded tables contain custom extension fields for specific tenants. That is, the source end has four basically identical sharded tables, but some tables have slight differences in fields. As shown in the figure below, the yellow\_taxi\_02 table has an additional extension field ext\_columm\_2 compared to the yellow\_taxi\_00 table.
+
+:-: ![](.topwrite/assets/image_1734675556308.png =500)
+
+###
+
+### Prerequisites: Data Source Configuration
+
+Navigate to Management -> Data Source, click "New Data Source" and select MySQL to create the data source required for this case. Fill in the necessary configuration information, especially paying attention to selecting the correct time zone of the database. Real-time synchronization is sensitive to time zones, and configuration errors will cause synchronization to fail.
+
+![](.topwrite/assets/image_1735208242938.png)
+
+For advanced configurations such as source database permissions, refer to [this document](realtime_sync.md).
+
+### Real-Time Synchronization Task Configuration
+
+First, click "New Multi-Table Real-Time Synchronization Task" in Task Development and select the correct source data source type, such as MySQL.
+![](.topwrite/assets/image_1734939965441.png =630)
+
+Real-time synchronization tasks provide two types of data synchronization modes:
+
+* First type: Multi-table mirroring synchronization, which mirrors the source data tables to the target end as they are.
+* Second type: Multi-table merge synchronization, which supports merging multiple source tables into the same target table. This is the scenario used in this practice. In the case of large source data volumes and sharding, merging multiple tables into the same table during synchronization eliminates the need for multi-table JOIN in subsequent queries, speeding up the query.
+
+After selecting the synchronization mode, choose the data source that stores the source connection string information.
+
+![](.topwrite/assets/image_1734940034365.png)
+
+^
+
+Next, configure the data objects to be synchronized. The product provides rich filtering rules to screen source tables, such as exact name matching, regular expression matching, etc., which can be used as needed. When there are inconsistencies in the source table fields, the system will automatically detect and prompt to enable the heterogeneous table merge synchronization function. The heterogeneous table merge synchronization function creates a table at the target end by taking the union of the source table fields, ensuring that all source data can be synchronized to the target end.
+
+![](.topwrite/assets/image_1734949182748.png)
+
+After configuration, you can preview and check the fields of specific tables to ensure that the extension fields are also added, ensuring data consistency.
+
+![](.topwrite/assets/image_1734952733442.png)
+
+If you use the mirror synchronization mode instead of the multi-table merge mode, the structure of the source table will be completely mapped to the target table, but ETL processing is still required to produce a new merged table for BI queries. Adding a table also adds a processing link, which increases complexity and extends the end-to-end link, significantly reducing data freshness and timeliness, and increasing costs.
+
+As mentioned earlier, the source data is stored using a sharding method, but a more complex situation is often encountered: records with the same primary key field values in the source end may appear duplicated in different sharded tables. The data in this example is also designed to simulate a similar situation. In the source end, the same primary key ID value in two tables will have a record in each table, which poses a challenge for synchronization: when merging and writing to the target end, if only the ID is used as the primary key, these two records will be attempted to be written into one record, causing data conflicts. The two source records will be processed in sequence, with the later source table data overwriting the earlier source table data. To address this situation, the Lakehouse real-time synchronization solution provides the ability to extend fields to ensure data accuracy. By identifying the source of the data, such as setting fields like server, databasename, and tablename as composite primary keys, these two records will be treated as two separate records in the target end. This ensures that even in complex situations where the primary key data is duplicated in the sharded tables at the source end, the source data can still be accurately synchronized to the target end.
+
+![](.topwrite/assets/image_1735208417539.png)
+
+After completing the above configuration, the next step is to set the target data source type, data source name, and other configuration items in the "Target Configuration".
+
+![](.topwrite/assets/image_1735205909270.png)
+
+Next, you can preview the synchronized field configuration in the mapping relationship, where you can see the extended fields included, and the extended fields/composite primary keys are also reflected here.
+
+![](.topwrite/assets/image_1735205869904.png)
+
+Singdata Lakehouse real-time synchronization also provides rich synchronization rule strategies to dynamically adapt to changes in the source database (Schema Evolution), such as strategies for handling situations where fields are added or deleted in the source table, and the types of source change messages that need to be processed.
+
+![](.topwrite/assets/image_1735205828392.png)
+
+At this point, the entire real-time synchronization task is configured, requiring only four simple steps: **select the data source, define the source synchronization objects, select the target end, and set the synchronization strategy**.
+
+### Submit and Start Real-Time Data Synchronization
+
+Singdata Lakehouse provides two modes: development environment and production environment. To run this task in the production environment, it needs to be published and submitted first, and then started in the Operations Center.
+
+* Submit Multi-Table Real-Time Synchronization Task
+  ![](.topwrite/assets/image_1735206359814.png)
+
+* Operate Multi-Table Real-Time Synchronization Task
+  ![](.topwrite/assets/image_1735206291470.png)
+
+* Start Multi-Table Real-Time Synchronization Task: When starting, you can choose whether to perform a full data synchronization based on actual needs. For the first synchronization, it is usually recommended to perform a full data synchronization once, and then proceed with incremental data synchronization.
+  ![](.topwrite/assets/image_1735206439758.png)
+  ![](.topwrite/assets/image_1735206478684.png)
+
+After the task starts, you can see in the operations interface that the task will enter the full synchronization stage, with 2 million rows of data already written at a rate of 25,000+ rows/second.
+
+![](.topwrite/assets/image_1734677235371.png)
+
+Specifically, the multi-table real-time synchronization task only needs to perform a full synchronization of the source end once during the first start. After the full synchronization is completed, the task will automatically switch to the incremental synchronization phase without manual changes, ensuring seamless data continuity.
+
+### Verify the Effect of Data Change Synchronization
+
+In practical applications, it is common for the source database tables and structures to change due to business needs, such as adding fields, deleting fields, or changing field types in the source table. Lakehouse real-time synchronization also provides corresponding solutions for these situations.
+
+Singdata Lakehouse product supports direct manipulation of the source database, allowing modifications, queries, and other operations on the source database through SQL. First, we make changes to the source data by adding some new fields: ext\_column\_0 & ext\_column\_1, deleting a field in the source table, and changing a field type from int to bigint.
+
+![](.topwrite/assets/image_1734677264978.png)
+
+![](.topwrite/assets/image_1734677273784.png)
+
+![](.topwrite/assets/image_1734677283775.png)
+
+Next, you can see on the monitoring page that the data changes from the source have been synchronized for consumption. Based on the Schema Evolution rules configured in the synchronization task, it automatically updates without manual operation for additional configuration or task restart, ensuring the smooth operation of the entire synchronization link.
+
+![](.topwrite/assets/image_1734677295827.png)
+
+After the synchronization is complete, you can also check in the task operation and maintenance interface whether the changes from the source have been successfully synchronized to the target. You can see that the newly added fields ext\_column\_0 & ext\_column\_1 have been extended to the target, and the changed bigint field has also been updated.
+
+![](.topwrite/assets/image_1734677309588.png)
+
+![](.topwrite/assets/image_1734677315955.png)
+
+###
+
+### Stability and Metrics Monitoring of Real-time Synchronization Tasks
+
+After the real-time synchronization and analysis link is established, one of the most concerned issues is the stability of the entire link. In our product, we provide comprehensive monitoring information display for each multi-table synchronization task, such as synchronization status, synchronization delay, etc. For possible abnormal situations in the synchronization link, such as a single data write failure causing the task to fail, we also provide automatic Failover capability for the task. After the task fails, it will be automatically restarted, reducing the need for manual operation and maintenance.
+
+![](.topwrite/assets/image_1734677408156.png)
+
+Singdata Lakehouse provides a monitoring and alerting product module with rich built-in status and metrics monitoring capabilities. It supports custom configuration of monitoring rules to comprehensively monitor the status of the entire task operation, including task instance operation failure, single table stock data synchronization exception, end-to-end delay of real-time synchronization tasks, job failover, source data reading point delay, etc. A series of monitoring items can be monitored by configuring corresponding rules.
+
+![](.topwrite/assets/image_1734677421992.png)
+
+In this practice demonstration, some monitoring rules were configured. As shown in the alert notification below, it monitors whether the end-to-end synchronization delay exceeds 10 seconds. When the quantity is particularly large, the delay will be captured by the monitoring and alert notification will remind the responsible person to perceive and handle it in time.
+
+![](.topwrite/assets/image_1734677436493.png)
+
+### Operation and Maintenance of Real-time Synchronization Tasks
+
+In actual production, various complex problems are often encountered. Therefore, Singdata's real-time synchronization solution also provides a variety of supporting operation and maintenance functions for support. For example, during real-time synchronization, if the source data changes and there is a problem with the data in a certain table, we provide a data replenishment synchronization function that supports re-synchronizing the entire table. During business peak times, when the source change traffic is very large, the real-time synchronization of multiple tables' change data will affect each other. In both cases, the full data replenishment synchronization function can be used to accelerate the data synchronization process. For example, re-synchronizing the source table yellow\_taxi\_00 for full data replenishment, its incremental real-time synchronization will be paused, and the background will re-synchronize the source data in a full manner. After the full data replenishment synchronization is completed, the incremental synchronization will automatically start without manual operation.
+
+![](.topwrite/assets/image_1734677453391.png)
+
+In addition, Singdata Lakehouse also provides a priority execution function for daily operation and maintenance. When multiple tables need to be synchronized and resources are relatively limited, you can choose to prioritize the allocation of resources to more important business tables, and prioritize the data synchronization of that table in the task queue. This ensures the freshness of critical business data even when there is a backlog of consumption and increased end-to-end delay.
+
+![](.topwrite/assets/image_1734677476077.png)
+
+## High-performance Real-time Analysis
+
+In the traditional offline data warehouse architecture, data is directly connected to the BI engine for querying after ETL processing, often resulting in slow response times. Therefore, enterprises usually introduce real-time analysis engines such as Clickhouse to improve query performance. Singdata's solution simplifies this process by enabling real-time data writing from the source and immediate query analysis of the target table without the need for an additional query acceleration engine.
+
+In this demonstration, we used the Metabase BI tool to build a report showing the comparison of the number of rows of MySQL data at the source and the number of rows in Singdata Lakehouse, and conducted complex query analysis, such as calculating the average fare based on the number of passengers. During this process, real-time data synchronization and query analysis were both completed on the same table, highlighting the advantages of Singdata's solution in simplifying architecture and improving efficiency.
+
+![](.topwrite/assets/image_1734677365384.png)
+
+Through the "Job History" module provided by the product, you can view all SQL details submitted to the Lakehouse engine. As shown in the figure below, the real-time synchronization writes to the \`yellow\_taxi\_demo\` table, which is also queried in real-time on the Metabase BI dashboard. Under the relatively complex query conditions demonstrated, a full table scan of 4 million rows returns query results in just 11 milliseconds. The Singdata Lakehouse solution does not require an additional acceleration engine for query acceleration, eliminating the need to synchronize and replicate an extra copy of the data, thereby significantly reducing overall costs.
+
+![](.topwrite/assets/image_1734677380258.png)
+
+Next, let's examine the query response under high concurrency. Singdata Lakehouse employs a storage-compute separation architecture. On the query computation side, different cluster types support various query loads:
+
+* **General-purpose Clusters**: Optimized for batch processing.
+* **Analytical Clusters**: Highly efficient for online real-time subqueries.
+
+Singdata also offers excellent resource elasticity. You can set different cluster specifications and scaling methods, configure the number of concurrent queries, and the number of instance replicas to achieve dynamic scaling. For example, if the source has 8 concurrent queries, only 1 query instance is needed. By setting the elasticity to 2 replicas, when the concurrency exceeds 8, the system will automatically scale out to the second replica to handle the additional traffic. The entire process requires no manual intervention.
+
+Additionally, Singdata Lakehouse provides an auto-start and auto-stop feature. In terms of resource pricing, the product operates on a pay-as-you-go model in the SaaS mode. For example, for BI reports, when there is no usage or traffic during the night, the cluster will be automatically stopped, incurring no costs. This helps avoid idle resource waste and saves on expenses.
+
+![](.topwrite/assets/image_1734677394361.png)
+
+## Summary
+
+Singdata Lakehouse, based on one-stop product capabilities, provides solutions for real-time synchronization and real-time analysis scenarios, addressing the complex situations of source databases. The product supports various data sources such as MySQL, PostgresSQL, SQLServer, etc., meeting the synchronization requirements in heterogeneous source databases, providing relatively complete high-performance synchronization capabilities, and comprehensive supporting operation and maintenance monitoring capabilities to ensure stable data production. In terms of cost, compared to traditional synchronization solutions from leading cloud vendors, Singdata can achieve overall cost savings of over 50%. In terms of performance, it has lower end-to-end synchronization latency, usually within 20 seconds; and faster analytical query response speeds, such as achieving a query response speed of 10 milliseconds for relatively complex SQL queries. Based on Singdata Lakehouse's product capabilities, it can provide enterprises with better data freshness, enabling business decisions to be made faster and maintaining a leading position in the competition.
+
+## Appendix
+
+For more detailed usage guidelines on the multi-table real-time synchronization feature, please refer to this [help document](realtime_sync.md).

package/bin/skills/lakehouse-doc-en/references/realtimesync_m.md

@@ -0,0 +1,189 @@
+# Full Incremental Integrated Synchronization Function
+
+The full incremental integrated synchronization function is a powerful data synchronization tool that can completely synchronize data from the database to the Lakehouse. This feature includes full synchronization of historical data and incremental synchronization of real-time change data. Please note that this feature is currently in Beta version, and some known limitations will be improved and optimized in subsequent versions.
+
+## Task Creation Process
+
+### Operation Entry
+
+In the task development interface, click the new operation button and select the "Full Incremental Integrated Synchronization" task type from the drop-down menu.
+
+![Create a new full incremental integrated synchronization task](.topwrite/assets/image_1710210458365.png)
+
+### Configure Source Data Type
+
+When creating a task, you first need to select a source data type to be synchronized. The currently supported source data types and their corresponding incremental reading modes and database versions are as follows:
+
+| **Type**     | **Incremental Reading Mode** | **Database Version** |
+| ---------- | ---------- | --------- |
+| MySQL      | Binlog     | 5.x, 8.x   |
+| PostgreSQL | WALs log     | 14 and above     |
+
+### Select Synchronization Type
+
+The full incremental integrated synchronization task supports two types of synchronization. Please choose according to actual needs:
+
+1. Real-time synchronization - multi-table mirroring: Suitable for scenarios where source data is completely mirrored to the target.
+2. Real-time synchronization - multi-table merging: Suitable for scenarios where data from multiple databases and tables are merged and written into the same target table.
+
+#### Multi-table Mirroring
+
+##### Source Configuration
+* Select Data Source: Choose an existing data source. If no data source is available, please refer to the [Data Source Configuration](config-datasource.md) document to create a new data source.
+* Configure Reading Mode: Select the appropriate reading mode based on the selected data source type. For example, MySQL data sources currently only support BINLOG mode, while PostgreSQL data sources only support WALs mode.
+* Select Synchronization Objects: Select the databases and tables to be synchronized. You can quickly select multiple synchronization objects by uploading a configuration file through the batch configuration function. Please note that comments in the configuration file need to be deleted.
+* Configure Slot
+  * Only PostgreSQL needs to configure the slot, which refers to PostgreSQL's replication slot. For details, see [Documentation](https://www.postgresql.org/docs/9.4/catalog-pg-replication-slots.html).
+  * Each database needs to configure a slot. Currently, decoderbufs and pgoutput plugin types are supported.
+  * You can choose to use an existing slot; or create a new one on the page. In the creation window's lower area, you can modify the creation statement as needed. Click OK to create. The username configured in the data source must have the permission to create a slot, otherwise, the creation will fail. See the permission description below for details.
+  * Special attention, do not reuse the same slot for different tasks. When the task starts, if the slot is occupied by other running tasks, it will fail to start. If other tasks are in a stopped state, because the data in the slot will be shared and consumed, the point is shared. The incremental data consumed by the new task may not be fully synchronized to the target table.
+
+##### **Target Configuration**
+* Target Data Source: By default, select the Lakehouse data source corresponding to the current workspace.
+* Namespace Rules: Currently, only specific namespaces can be selected. Future versions will support mirroring naming or custom naming rules based on the source.
+* Target Table Naming Rules: Currently, only the source table name can be mirrored. Future versions will support adding prefixes or custom naming rules.
+* Compute Cluster: Select the available compute cluster in the workspace. It is recommended to use an AP type cluster.
+
+##### Preview Configuration
+After completing the source and target configuration, you can preview the mapping relationship between the synchronization tables and fields. If modifications are needed, you can go back to the previous step to adjust.
+
+##### Synchronization Rules
+In the synchronization rules, you can configure rules such as Schema Evolution to define the handling behavior after changes in source tables and fields.
+
+#### Multi-table Merging
+
+##### **Source Configuration**
+
+* Select Data Source
+  * Same as above, similar to multi-table mirroring synchronization
+* Configure Reading Mode
+  * Same as above, similar to multi-table mirroring synchronization
+* Select Synchronization Objects
+  Multi-table merging synchronization will use a "virtual table" as an intermediate transition to receive data from specified objects. There are two ways to configure:
+  * Method 1: Based on rule configuration, filter through regular expressions, such as selecting all tables starting with abc, fill in ^abc in the regular expression.
+  * Method 2: Based on file batch configuration, fill in the configuration file according to the template format requirements, upload it, and the page will select the objects to be synchronized according to the content given in the file. After selection, you can fine-tune as needed. Please note to delete the comments in the template file.
+
+##### **Target Configuration**
+* Same as above, similar to multi-table mirroring synchronization
+
+##### **Preview Configuration**
+* Same as above, similar to multi-table mirroring synchronization
+
+##### **Synchronization Rules**
+* Same as above, similar to multi-table mirroring synchronization
+
+### Configure Advanced Parameters
+After completing the regular task configuration, click the "Configure" button to complete the advanced parameter settings.
+* Cluster: Specify the cluster used to run the synchronization task. It is recommended to use an AP type cluster.
+* Update Frequency: Set the frequency interval for data writing.
+
+### Submit Task
+Click the submit button to submit the task to the production environment for execution. Note that the task does not start automatically after submission and needs to be started manually.
+
+## Task Operations
+
+### Start Task
+
+On the task details page, you can start the task. Here are three ways to start:
+1. Stateless start (only applicable for the first start of the task): Fully synchronize all data, first perform full synchronization, and then start incremental synchronization.
+2. Resume from the last saved state (applicable for restarting tasks after stopping): Resume from the breakpoint where it was stopped.
+3. Custom starting position: Synchronize from the given position, which can be used for data backfill. It applies uniformly to all tables configured in the task.
+
+* MySQL: Choose to start from a specified file or a specified time. You can get the last file position in the page monitoring area.
+* PostgreSQL: Specify the LSN value, and you can view the latest LSN value in the instance monitoring below.
+
+### Instance Monitoring
+
+After the task starts, it will go through three stages: initialization, full synchronization, and incremental synchronization. In the instance monitoring area, you can view the running status of these three stages.
+![Instance Monitoring](.topwrite/assets/image_1705386710920.png)
+
+### Metrics Monitoring
+
+In the metrics monitoring area, you can view the key monitoring metrics for full synchronization and incremental synchronization.
+![Metrics Monitoring](.topwrite/assets/image_1705386727702.png)
+
+| **Metric Name**   | **Description**                                               |
+| ---------- | ------------------------------------------------------ |
+| Data Read       | The number of records read from the data source by the data synchronization task during the statistical period.                                |
+| Data Written       | The number of records written to the target data source by the data synchronization task during the statistical period.                              |
+| Average Read Rate     | The average read rate of the data synchronization task during the statistical period. (Total records read during the period / period time)                     |
+| Average Write Rate     | The average write rate of the data synchronization task during the statistical period. (Total records written during the period / period time)                     |
+| Failover Count | The number of failovers that occurred during the statistical period of the data synchronization task. The number of failovers represents the stability of the data synchronization service itself. |
+
+### Synchronization Objects
+
+In the synchronization objects area, you can view the final state of single table synchronization and perform corresponding operations.
+![](.topwrite/assets/image_1710214491047.png)
+
+| **Metric** | **Description**                                    |
+| ------ | ------------------------------------------- |
+| Latest Read Position | The synchronization task reads data from the source object in real-time and writes it to the target table, using the write time of the latest record in the target table as the read position. |
+| Latest Update Time | The last time data was written to the target table.                               |
+| Data Delay   | The time interval from when the data is committed at the data source end to when it is visible at the target end.                  |
+
+| **Operation** | **Behavior Definition**                                     |
+| ------ | -------------------------------------------- |
+| Full Synchronization Details |                                              |
+| Pause     | For tables that are being synchronized, you can pause the synchronization of the table and save the synchronization position.                  |
+| Resume     | Continue synchronization from the last paused synchronization position.                            |
+| Resynchronize   | Perform full and incremental synchronization for the table again.                              |
+| View Exceptions   | Click to view the exception information of the incremental synchronization task of the table, such as Schema Evolution exceptions. |
+
+### Stop Task
+
+Stopping a task will stop the ongoing full synchronization and incremental synchronization in the task. The incremental synchronization position will be automatically saved when stopping.
+
+* If the table is in the full synchronization stage when stopping, it will perform a new full synchronization for the table that has not been fully synchronized after restarting.
+* If the table is in the incremental synchronization stage when stopping, it will continue synchronization from the position where it stopped by default after restarting.
+
+### Offline Task
+
+Taking a task offline is a high-risk operation. After the task is taken offline, the current synchronization position will not be saved. If the task is brought online and started again, it will start synchronizing data from scratch.
+
+* Taking offline will not clean the data that has been synchronized to the target end, but it will clean the cached data and position information of the intermediate process.
+* Resynchronization will not recreate the table. Full data synchronization will overwrite the old table, and incremental synchronization will update the target table with a merge into.
+
+## Permission Description
+
+When synchronizing change events from different types of data sources, appropriate permissions need to be configured on the corresponding data source server to ensure normal data synchronization. Although directly assigning an administrator or superuser permission is sufficient to ensure the task runs normally, it is usually desirable to minimize the permissions that need to be assigned to the user synchronizing the data. The specific permission configurations required for each operation step are as follows.
+
+### PostgreSQL
+
+| Requirement                             | Permission                                                             | Example                                                                                                                                                                               |
+| -------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Configure task (retrieve metadata: schema list, table list, table field list) | SELECT (on information\_schema and the tables that need details)       | Grant a role permission to read information\_schema: GRANT SELECT ON TABLE information\_schema.tables TO role\_name;

Grant a role permission to read a specific table: GRANT SELECT ON TABLE your\_schema.your\_table TO role\_name; |
+| Synchronize wal logs                    | REPLICATION&#xA;LOGIN                                                 | CREATE ROLE \<name> REPLICATION LOGIN;                                                                                                                                               |
+| Synchronize historical full data (optional) | SELECT (on the tables that need to be synchronized)                    | Grant a role permission to read a specific table: GRANT SELECT ON TABLE table\_name TO role\_name;&#xA;&#xA;Grant a role permission to read all tables in a schema: GRANT SELECT ON ALL TABLES IN SCHEMA schema\_name TO role\_name;                      |
+| Create publication (optional)           | CREATE (on the database where the publication needs to be created)&#xA;SELECT (on the tables that need to be added to the publication) | Grant CREATE permission: GRANT CREATE ON DATABASE your\_database TO role\_name;                                                                                                                    |
+
+### MySQL
+| Requirement                                | Permission                                                    | Example                                                                                                                                                                                              |
+| ---------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Configure tasks (fetch metadata: database list, table list, table field list) | SHOW DATABASES&#xA;SHOW TABLES (directly granting the more general SELECT permission is also possible)&#xA;SELECT | Grant the user permission to query the database list: GRANT SHOW DATABASES ON . TO 'username'@'host';&#xA;&#xA;Grant the user permission to query the table list and table details (SELECT includes the SHOW TABLES permission): GRANT SELECT ON database\_name.table\_name TO 'username'@'host'; |
+| Sync binlog logs                               | RELOAD&#xA;REPLICATION SLAVE&#xA;REPLICATION CLIENT           | GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON . TO 'username'@'host';                                                                                          |
+| Sync historical full data                                 | SELECT                                                        | Grant the user permission to query the table:&#xA;  GRANT SELECT ON database\_name.table\_name TO 'username'@'host';                                                                                                          |
+
+## Known Limitations
+
+* The current solution requires setting the data refresh frequency, with a minimum support of 1 minute. This means that the end-to-end data freshness delay will be at least over 1 minute, and second-level delay is not yet achieved (will be provided in future versions).
+* Schema Evolution, currently does not support changing field types, and does not support automatic table addition.
+* In multi-table real-time sync tasks, if there are data with the same primary key in different source tables, the sync result will be abnormal.
+* MySQL sync, unsupported field types:
+
+| Field Type | Behavior after Sync |
+| ---- | ----- |
+| year | Value does not correspond |
+
+* PostgreSQL sync, unsupported field types:
+
+| Field Type     | Behavior after Sync          |
+| -------- | -------------- |
+| varbit   | Value does not correspond          |
+| bytea    | Value does not correspond          |
+| TIMETZ   | Value does not correspond          |
+| interval | Value does not correspond          |
+| NAME     | Value does not correspond          |
+| NUMERIC  | Precision does not correspond, target end precision will be higher |
+| decimal  | Precision does not match, the target end precision will be higher |
+
+^