@clickzetta/cz-cli-darwin-arm64 0.5.16 → 0.5.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/cz-cli +0 -0
- package/bin/skills/lakehouse-doc-en/SKILL.md +6 -11
- package/bin/skills/lakehouse-doc-en/references/AIGateway.md +58 -13
- package/bin/skills/lakehouse-doc-en/references/Computation.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/DataSource_Amazon_DocumentDB.md +3 -1
- package/bin/skills/lakehouse-doc-en/references/Foreach.md +14 -14
- package/bin/skills/lakehouse-doc-en/references/JDBC-Driver.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/LakehouseAI-overview.md +21 -8
- package/bin/skills/lakehouse-doc-en/references/LakehouseDataGPT-tour.md +4 -9
- package/bin/skills/lakehouse-doc-en/references/LakehouseStudio-tour.md +14 -19
- package/bin/skills/lakehouse-doc-en/references/Lakehouse_Zilliz_MakeDataReadyforBIandAI.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/Logstash.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/Migrate_Spark_DataEngineeringBestPractices_Project_to_Lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/Notebook.md +17 -17
- package/bin/skills/lakehouse-doc-en/references/RemoteFunction-as-udf.md +14 -14
- package/bin/skills/lakehouse-doc-en/references/SQL_External_Catalog_Guide.md +1 -9
- package/bin/skills/lakehouse-doc-en/references/SUMMARY.md +59 -29
- package/bin/skills/lakehouse-doc-en/references/WINDOWFUNCTION.md +99 -57
- package/bin/skills/lakehouse-doc-en/references/Zettapark_Data_Engineering_Demo.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/access-control-configuration.md +1 -8
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-2-5-1.0.md +16 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-3-29-1.0.2.md +14 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-3-8-1.0.1.md +16 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-4-28-1.1.md +29 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-12-1.1.1.md +18 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-15-1.2.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-21-1.3.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-28-1.4.md +10 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-6-3-1.5.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/alicloud-arn-externalid.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/answer-accuracy-improve.md +120 -103
- package/bin/skills/lakehouse-doc-en/references/application-list.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/approval-list.md +16 -17
- package/bin/skills/lakehouse-doc-en/references/batch-load-parquet-file-into-lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/batch_sync.md +9 -9
- package/bin/skills/lakehouse-doc-en/references/batch_sync_Sop.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/batchloadparquetfileintoLakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/bulkloadv1-python-sdk.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/chart-auto-refresh-guide.md +12 -6
- package/bin/skills/lakehouse-doc-en/references/clickzetta-sample-data.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/code_approval.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/composite_task.md +31 -42
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_environment_and_data_generate.md +6 -9
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_javasdk_bulkload_realtime.md +4 -10
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_kafka_realtime_sync.md +1 -10
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_local_file_into_table_by_studio.md +0 -6
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_batchload_public_network.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_python_node.md +2 -7
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_realtime_cdc_public_network.md +13 -18
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_sql_insert.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/concepts.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/config-datasource.md +5 -7
- package/bin/skills/lakehouse-doc-en/references/connect-with-cli.md +116 -72
- package/bin/skills/lakehouse-doc-en/references/connect-with-cz-cli.md +151 -0
- package/bin/skills/lakehouse-doc-en/references/continue-job.md +9 -17
- package/bin/skills/lakehouse-doc-en/references/create-api-connection.md +315 -286
- package/bin/skills/lakehouse-doc-en/references/create-catalog-connection.md +1 -0
- package/bin/skills/lakehouse-doc-en/references/create-dynamic-table.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/create-external-catalog.md +85 -22
- package/bin/skills/lakehouse-doc-en/references/create-table-ddl.md +45 -0
- package/bin/skills/lakehouse-doc-en/references/creating_alicloud_privatelinkendpoint.md +4 -6
- package/bin/skills/lakehouse-doc-en/references/creating_alicloud_privatelinkservice.md +4 -7
- package/bin/skills/lakehouse-doc-en/references/creating_tencentcloud_privatelinkendpoint.md +2 -7
- package/bin/skills/lakehouse-doc-en/references/creating_tencentcloud_privatelinkservice.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/cz-cli-agent.md +15 -10
- package/bin/skills/lakehouse-doc-en/references/cz-cli-datasource.md +0 -8
- package/bin/skills/lakehouse-doc-en/references/cz-cli-sql.md +2 -45
- package/bin/skills/lakehouse-doc-en/references/cz-cli.md +53 -42
- package/bin/skills/lakehouse-doc-en/references/dashboard-version-management-guide.md +12 -4
- package/bin/skills/lakehouse-doc-en/references/data-integration-intro.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/data-integration.md +29 -27
- package/bin/skills/lakehouse-doc-en/references/data-load-summary.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/data-quality.md +25 -25
- package/bin/skills/lakehouse-doc-en/references/data-sharing.md +31 -54
- package/bin/skills/lakehouse-doc-en/references/data-sources.md +45 -45
- package/bin/skills/lakehouse-doc-en/references/data_catalog.md +23 -25
- package/bin/skills/lakehouse-doc-en/references/data_privacy.md +5 -2
- package/bin/skills/lakehouse-doc-en/references/data_sharing_between_accounts_guide.md +0 -4
- package/bin/skills/lakehouse-doc-en/references/data_visualization.md +4 -15
- package/bin/skills/lakehouse-doc-en/references/dataagent.md +39 -7
- package/bin/skills/lakehouse-doc-en/references/databricks-delta-to-lakehouse-migration.md +168 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-dlt-to-lakehouse-migration.md +331 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-external-catalog-practice.md +367 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-jobs-to-studio-migration.md +199 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-notebook-to-studio-migration.md +350 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-uc-governance-to-lakehouse-migration.md +327 -0
- package/bin/skills/lakehouse-doc-en/references/datagpt-model-config.md +34 -0
- package/bin/skills/lakehouse-doc-en/references/datagpt_data_source.md +50 -37
- package/bin/skills/lakehouse-doc-en/references/datagpt_introduction.md +55 -79
- package/bin/skills/lakehouse-doc-en/references/datagpt_quickstart.md +50 -64
- package/bin/skills/lakehouse-doc-en/references/datalake-acceleration.md +75 -2
- package/bin/skills/lakehouse-doc-en/references/dbt-databricks-to-clickzetta-migration.md +242 -0
- package/bin/skills/lakehouse-doc-en/references/dynamic-mask.md +30 -30
- package/bin/skills/lakehouse-doc-en/references/dynamic-table-bestpractice.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/dynamic-table-introduce.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/dynamic_table_summary.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/eco_integration/streamlit.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/eco_integration/superset.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/ecosystem-all.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/ecosystem.md +145 -0
- package/bin/skills/lakehouse-doc-en/references/external-catalog-summary.md +33 -38
- package/bin/skills/lakehouse-doc-en/references/external-function-combo-practice.md +466 -0
- package/bin/skills/lakehouse-doc-en/references/f6fc6447ee.md +7 -9
- package/bin/skills/lakehouse-doc-en/references/federation-query.md +56 -6
- package/bin/skills/lakehouse-doc-en/references/finebi-mysql.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/get-started-with-sample-data.md +10 -11
- package/bin/skills/lakehouse-doc-en/references/gitfolder.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/grant-privileges.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/iceberg-rest-catalog-databricks.md +166 -0
- package/bin/skills/lakehouse-doc-en/references/ide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/if_else_task.md +59 -57
- package/bin/skills/lakehouse-doc-en/references/input_output.md +10 -7
- package/bin/skills/lakehouse-doc-en/references/jobprofile-bestpractices.md +60 -64
- package/bin/skills/lakehouse-doc-en/references/kafka-connection.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/key-concepts.md +146 -117
- package/bin/skills/lakehouse-doc-en/references/lakehouse-ai-gateway-cz-cli.md +317 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-ai-sql-analysis.md +345 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-dqc-guide.md +300 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-medallion-sql-dt-guide.md +543 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-multi-cloud-acceleration.md +274 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-multimodal-ai-pipeline.md +198 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-quick-experience_guide.md +49 -52
- package/bin/skills/lakehouse-doc-en/references/lakehouse-volume-pipe-acceleration-guide.md +380 -0
- package/bin/skills/lakehouse-doc-en/references/langchain-plug-installation.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/management.md +4 -9
- package/bin/skills/lakehouse-doc-en/references/medallion-lakehouse-from-scratch.md +2 -1
- package/bin/skills/lakehouse-doc-en/references/metrics_answer_build.md +58 -21
- package/bin/skills/lakehouse-doc-en/references/migrate-spark-data-engineering-best-practices-to-lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/mindsdb.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/monitoring_and_alerting.md +65 -60
- package/bin/skills/lakehouse-doc-en/references/monitoring_item_specification.md +33 -33
- package/bin/skills/lakehouse-doc-en/references/multitable_batch_sync.md +16 -16
- package/bin/skills/lakehouse-doc-en/references/multitable_realtime_sync.md +65 -72
- package/bin/skills/lakehouse-doc-en/references/multitable_realtime_sync_sop.md +54 -52
- package/bin/skills/lakehouse-doc-en/references/navicat-mysql.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/om-dynamic-table.md +71 -66
- package/bin/skills/lakehouse-doc-en/references/om-vcluster.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-create-session.md +79 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-generate-auth-token.md +63 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-overview.md +96 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-quick-start.md +286 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-response-guide.md +264 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-safe-question-poll.md +201 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-text2insight-query.md +99 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-text2insight-stop.md +74 -0
- package/bin/skills/lakehouse-doc-en/references/overview.md +6 -7
- package/bin/skills/lakehouse-doc-en/references/permission-application.md +5 -5
- package/bin/skills/lakehouse-doc-en/references/pipe-introduction.md +1 -0
- package/bin/skills/lakehouse-doc-en/references/pipe-kafka-table-stream.md +72 -70
- package/bin/skills/lakehouse-doc-en/references/pipe-kafka.md +105 -110
- package/bin/skills/lakehouse-doc-en/references/pipe-overview.md +40 -40
- package/bin/skills/lakehouse-doc-en/references/pipe-storage-object.md +43 -48
- package/bin/skills/lakehouse-doc-en/references/pipe-summary.md +14 -4
- package/bin/skills/lakehouse-doc-en/references/pipe-syntax.md +58 -151
- package/bin/skills/lakehouse-doc-en/references/practice_python_task.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/pricing-ai-gateway.md +181 -0
- package/bin/skills/lakehouse-doc-en/references/pricing-lakehouse.md +316 -0
- package/bin/skills/lakehouse-doc-en/references/pricing.md +44 -288
- package/bin/skills/lakehouse-doc-en/references/private-link-general.md +0 -2
- package/bin/skills/lakehouse-doc-en/references/pyspark-to-zettapark-migration-f1.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python-igs.md +7 -3
- package/bin/skills/lakehouse-doc-en/references/python-sample-put-github-rt-events.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python-task.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector_advanced.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector_examples.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/python_sdk_guide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python_shell_datasource.md +11 -9
- package/bin/skills/lakehouse-doc-en/references/quick_start_batch_sync_data.md +9 -18
- package/bin/skills/lakehouse-doc-en/references/quick_start_bi_analysis.md +8 -25
- package/bin/skills/lakehouse-doc-en/references/quick_start_create_workspace.md +4 -6
- package/bin/skills/lakehouse-doc-en/references/quick_start_data_quality.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quick_start_etl.md +16 -20
- package/bin/skills/lakehouse-doc-en/references/quick_start_monitoring_and_alerting.md +10 -18
- package/bin/skills/lakehouse-doc-en/references/quick_start_sql_query.md +7 -10
- package/bin/skills/lakehouse-doc-en/references/quick_start_upload_data.md +5 -7
- package/bin/skills/lakehouse-doc-en/references/quick_start_user_management.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quick_start_workspace.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/quick_start_workspace_user.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quickstart.md +69 -56
- package/bin/skills/lakehouse-doc-en/references/quickstart_datashare_between_companies.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/quickstart_envirment_for_team.md +0 -24
- package/bin/skills/lakehouse-doc-en/references/realtime-pipeline-selection-guide.md +1 -2
- package/bin/skills/lakehouse-doc-en/references/realtime-sales-dashboard-with-dynamic-table.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/realtime_sync.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/release-note-2026-05-19.md +5 -3
- package/bin/skills/lakehouse-doc-en/references/revoke-privileges.md +3 -1
- package/bin/skills/lakehouse-doc-en/references/roles.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/row-filter.md +165 -0
- package/bin/skills/lakehouse-doc-en/references/row_level_permission.md +30 -19
- package/bin/skills/lakehouse-doc-en/references/scheduled_task.md +28 -21
- package/bin/skills/lakehouse-doc-en/references/security_overview.md +99 -21
- package/bin/skills/lakehouse-doc-en/references/set-command.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/setup.md +13 -15
- package/bin/skills/lakehouse-doc-en/references/show-grants.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/snowflake-dynamic-tables-to-lakehouse.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/spark-connector-summary.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/sql_functions/context_functions/current_vcluster.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/sso-configuration.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/streaming_pipeline_with_dynamic_table.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/studio-incremental-sync-practice.md +27 -23
- package/bin/skills/lakehouse-doc-en/references/studio-shell-task.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/supported-cloud-platforms.md +32 -0
- package/bin/skills/lakehouse-doc-en/references/table_rendering.md +18 -12
- package/bin/skills/lakehouse-doc-en/references/task-develop.md +89 -91
- package/bin/skills/lakehouse-doc-en/references/task_development.md +19 -17
- package/bin/skills/lakehouse-doc-en/references/task_group.md +16 -14
- package/bin/skills/lakehouse-doc-en/references/task_instance.md +21 -21
- package/bin/skills/lakehouse-doc-en/references/task_param.md +38 -35
- package/bin/skills/lakehouse-doc-en/references/task_param_reference.md +81 -79
- package/bin/skills/lakehouse-doc-en/references/task_scheduling_dependency.md +20 -21
- package/bin/skills/lakehouse-doc-en/references/tencentcloud_arn_and_externalid.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/trial-account-quotas-and-limits.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/tutorial_connect_to_lakehouse.md +69 -0
- package/bin/skills/lakehouse-doc-en/references/tutorials.md +4 -1
- package/bin/skills/lakehouse-doc-en/references/unique-key.md +167 -0
- package/bin/skills/lakehouse-doc-en/references/usageandbillingview.md +138 -0
- package/bin/skills/lakehouse-doc-en/references/use-dbt-dev.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/use-java-sdk-realtime-uploaddata.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/use-java-sdk-upload-data-local.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/use-models.md +128 -0
- package/bin/skills/lakehouse-doc-en/references/use-mysql-client.md +81 -81
- package/bin/skills/lakehouse-doc-en/references/use-python-sdk-upload-data.md +10 -12
- package/bin/skills/lakehouse-doc-en/references/user-identification.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/user_permission_grand_guide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/using-udf-in-dynamic-table.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/vc_cache.md +18 -22
- package/bin/skills/lakehouse-doc-en/references/vcluster_size_description.md +33 -31
- package/bin/skills/lakehouse-doc-en/references/virtual-cluster.md +43 -45
- package/bin/skills/lakehouse-doc-en/references/web-job-history.md +94 -108
- package/bin/skills/lakehouse-doc-en/references/web_search.md +16 -7
- package/bin/skills/lakehouse-doc-en/references/zettapark-data-engineering-demo.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/zettapark-dataframe-guide.md +144 -70
- package/bin/skills/lakehouse-doc-en/references/zettapark-dynamic-table-guide.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-etl-guide.md +73 -33
- package/bin/skills/lakehouse-doc-en/references/zettapark-feature-engineering.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-functions-guide.md +75 -46
- package/bin/skills/lakehouse-doc-en/references/zettapark-quick-start.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-stream-guide.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/zettapark-volume-guide.md +93 -29
- package/package.json +1 -1
- package/bin/skills/lakehouse-doc-en/references/CLAUDE.md +0 -606
- package/bin/skills/lakehouse-doc-en/references/modelprice.md +0 -155
|
@@ -41,24 +41,24 @@ The source database requires certain parameter settings and permission grants to
|
|
|
41
41
|
|
|
42
42
|
Note: Modifying the following parameters requires restarting the PostgreSQL server to take effect.
|
|
43
43
|
|
|
44
|
-
| |
|
|
45
|
-
| -------------------------- |
|
|
46
|
-
| Configuration | Description
|
|
47
|
-
| wal\_level | WAL level. `logical` is required for real-time sync. `replica` supports WAL archiving and replication. `minimal` keeps only what is needed to recover from a crash.
|
|
48
|
-
| max\_replication\_slots | Maximum number of replication slots allowed on the server.
|
|
49
|
-
| max\_wal\_senders | Maximum number of WAL sender processes that can run simultaneously, corresponding to the number of concurrent real-time sync tasks.
|
|
50
|
-
| max\_slot\_wal\_keep\_size | Size of WAL retained per slot. `-1` means unlimited.
|
|
51
|
-
| wal\_sender\_timeout | Replication connections idle longer than this value will be terminated.
|
|
44
|
+
| | | |
|
|
45
|
+
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
|
|
46
|
+
| Configuration | Description | Default Value (Unit) |
|
|
47
|
+
| wal\_level | WAL level. `logical` is required for real-time sync. `replica` supports WAL archiving and replication. `minimal` keeps only what is needed to recover from a crash. | replica |
|
|
48
|
+
| max\_replication\_slots | Maximum number of replication slots allowed on the server. | 10 |
|
|
49
|
+
| max\_wal\_senders | Maximum number of WAL sender processes that can run simultaneously, corresponding to the number of concurrent real-time sync tasks. | 10 |
|
|
50
|
+
| max\_slot\_wal\_keep\_size | Size of WAL retained per slot. `-1` means unlimited. | -1 (MB) |
|
|
51
|
+
| wal\_sender\_timeout | Replication connections idle longer than this value will be terminated. | 60000 (ms) |
|
|
52
52
|
|
|
53
53
|
**MySQL**
|
|
54
54
|
|
|
55
|
-
| |
|
|
56
|
-
| ----------------------------- |
|
|
57
|
-
| Attribute | Description
|
|
58
|
-
| log\_bin | Whether binlog is enabled.
|
|
59
|
-
| binlog\_format | Binlog format. `statement` records SQL (compact but can cause sync errors with non-deterministic functions). `row` records full before/after row images (accurate but higher volume). `mixed` lets MySQL choose automatically.
|
|
60
|
-
| binlog\_row\_image | Whether full before/after row images are recorded.
|
|
61
|
-
| binlog\_expire\_logs\_seconds | Binlog automatic cleanup interval.
|
|
55
|
+
| | | | |
|
|
56
|
+
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- | ----------------------------------------------- |
|
|
57
|
+
| Attribute | Description | Required Setting | Query Method |
|
|
58
|
+
| log\_bin | Whether binlog is enabled. | ON | SHOW GLOBAL VARIABLES LIKE 'log\_bin' |
|
|
59
|
+
| binlog\_format | Binlog format. `statement` records SQL (compact but can cause sync errors with non-deterministic functions). `row` records full before/after row images (accurate but higher volume). `mixed` lets MySQL choose automatically. | ROW | SHOW GLOBAL VARIABLES LIKE 'binlog\_format' |
|
|
60
|
+
| binlog\_row\_image | Whether full before/after row images are recorded. | FULL (record all fields in both images) | SHOW GLOBAL VARIABLES LIKE 'binlog\_row\_image' |
|
|
61
|
+
| binlog\_expire\_logs\_seconds | Binlog automatic cleanup interval. | Configure based on business needs; 86400 seconds (1 day) or more recommended. | |
|
|
62
62
|
|
|
63
63
|
#### Database Permission Configuration
|
|
64
64
|
|
|
@@ -68,11 +68,11 @@ Appropriate permissions must be configured on each source database to ensure cha
|
|
|
68
68
|
|
|
69
69
|
When executing grant SQL statements, ensure the executing account itself has the ability to grant those permissions — using an administrator account is recommended. Execute the grants for all scenarios listed below to ensure the task runs smoothly.
|
|
70
70
|
|
|
71
|
-
**Scenario: Task configuration (fetching metadata: schema list, table list, field list)
|
|
71
|
+
**Scenario: Task configuration (fetching metadata: schema list, table list, field list**)
|
|
72
72
|
|
|
73
73
|
Required permissions:
|
|
74
74
|
|
|
75
|
-
> SELECT (on
|
|
75
|
+
> SELECT (on information\_schema and the tables to be inspected)
|
|
76
76
|
|
|
77
77
|
Grant statements:
|
|
78
78
|
|
|
@@ -100,7 +100,7 @@ Grant statements:
|
|
|
100
100
|
CREATE ROLE <name> REPLICATION LOGIN;
|
|
101
101
|
```
|
|
102
102
|
|
|
103
|
-
**Scenario: Sync historical full data (optional)
|
|
103
|
+
**Scenario: Sync historical full data (optional**)
|
|
104
104
|
|
|
105
105
|
* Required permissions:
|
|
106
106
|
|
|
@@ -138,7 +138,7 @@ Grant statement:
|
|
|
138
138
|
|
|
139
139
|
When executing grant SQL statements, ensure the executing account has the GRANT OPTION privilege — using a superuser such as root is recommended. Execute the grants for all scenarios listed below.
|
|
140
140
|
|
|
141
|
-
**Scenario: Task configuration (fetching metadata: database list, table list, field list)
|
|
141
|
+
**Scenario: Task configuration (fetching metadata: database list, table list, field list**)
|
|
142
142
|
|
|
143
143
|
Required permissions:
|
|
144
144
|
|
|
@@ -210,13 +210,13 @@ Follow these steps in order:
|
|
|
210
210
|
|
|
211
211
|
* Extended fields currently support: source server name, database name, schema name, and table name.
|
|
212
212
|
|
|
213
|
-
|
|
213
|
+
^
|
|
214
214
|
|
|
215
215
|
### General: If the same primary key exists in different sharded source tables, how do I avoid write conflicts in the target table?
|
|
216
216
|
|
|
217
217
|
* Enable the extended fields feature in the task configuration, and include those extended fields in the composite primary key of the target table.
|
|
218
218
|
|
|
219
|
-
|
|
219
|
+
^
|
|
220
220
|
|
|
221
221
|
### General: If the sharded source tables have mostly — but not exactly — the same field structure, how do I configure the sync task?
|
|
222
222
|
|
|
@@ -314,7 +314,7 @@ There are several options:
|
|
|
314
314
|
|
|
315
315
|
* `step1.taskmanager.memory.task.off-heap.size` — for example, set to `500M` (default: `256M`)
|
|
316
316
|
|
|
317
|
-
|
|
317
|
+
^
|
|
318
318
|
|
|
319
319
|
### Emergency Maintenance: The task has failed and I cannot resolve it on my own — how do I get help from Singdata?
|
|
320
320
|
|
|
@@ -334,31 +334,31 @@ There are several options:
|
|
|
334
334
|
|
|
335
335
|
After a task starts, it goes through three phases: initialization, full sync, and incremental sync. You can view the status of these phases in the instance monitoring area.
|
|
336
336
|
|
|
337
|
-
|
|
337
|
+
^
|
|
338
338
|
|
|
339
339
|
#### Metric Monitoring
|
|
340
340
|
|
|
341
|
-
|
|
|
342
|
-
|
|
|
343
|
-
| Metric
|
|
344
|
-
| Data read
|
|
345
|
-
| Data written
|
|
346
|
-
| Avg. read rate
|
|
347
|
-
| Avg. write rate
|
|
348
|
-
| Failover count
|
|
341
|
+
| | |
|
|
342
|
+
| --------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
|
343
|
+
| Metric | Description |
|
|
344
|
+
| Data read | Number of records read from the source during the measurement period. |
|
|
345
|
+
| Data written | Number of records written to the target during the measurement period. |
|
|
346
|
+
| Avg. read rate | Average read rate during the measurement period (total records read / period duration). |
|
|
347
|
+
| Avg. write rate | Average write rate during the measurement period (total records written / period duration). |
|
|
348
|
+
| Failover count | Number of failovers during the measurement period. This reflects the operational stability of the sync service itself. |
|
|
349
349
|
|
|
350
|
-
|
|
350
|
+
^
|
|
351
351
|
|
|
352
352
|
#### Per-Table Sync Progress
|
|
353
353
|
|
|
354
|
-
| |
|
|
355
|
-
| -------------------- |
|
|
356
|
-
| Metric | Description
|
|
357
|
-
| Latest read position | The write time of the most recent record written to the target table, used as a proxy for the current read position.
|
|
358
|
-
| Latest update time | The last time a record was written to the target table.
|
|
359
|
-
| Data latency | The time between a transaction committing on the source and the data becoming visible on the target.
|
|
354
|
+
| | |
|
|
355
|
+
| -------------------- | -------------------------------------------------------------------------------------------------------------------- |
|
|
356
|
+
| Metric | Description |
|
|
357
|
+
| Latest read position | The write time of the most recent record written to the target table, used as a proxy for the current read position. |
|
|
358
|
+
| Latest update time | The last time a record was written to the target table. |
|
|
359
|
+
| Data latency | The time between a transaction committing on the source and the data becoming visible on the target. |
|
|
360
360
|
|
|
361
|
-
|
|
361
|
+
^
|
|
362
362
|
|
|
363
363
|
### Configuring Task Monitoring and Alerts
|
|
364
364
|
|
|
@@ -381,25 +381,25 @@ In the monitoring and alerting module, you can configure monitoring rules to tra
|
|
|
381
381
|
|
|
382
382
|
3. Add a new webhook configuration in the product — select Lark or WeCom as the channel and enter the bot's webhook URL.
|
|
383
383
|
|
|
384
|
-
|
|
384
|
+
^
|
|
385
385
|
|
|
386
386
|
4. Enable the webhook in the notification policy.
|
|
387
387
|
|
|
388
|
-
|
|
388
|
+
^
|
|
389
389
|
|
|
390
390
|
5. In the monitoring rule, select the notification policy with the webhook enabled, then choose the webhook configuration you just created.
|
|
391
391
|
|
|
392
|
-
|
|
392
|
+
^
|
|
393
393
|
|
|
394
394
|
#### Configuring Exception Monitoring for Real-Time Data Integration Tasks
|
|
395
395
|
|
|
396
396
|
1. **Task failover alert**: create a new monitoring rule and select "Multi-table real-time sync job failover" as the monitoring event. You can add filter attributes such as workspace or task name. If no filter is added, all multi-table real-time sync tasks in the instance are monitored by default.
|
|
397
397
|
|
|
398
|
-
|
|
398
|
+
^
|
|
399
399
|
|
|
400
400
|
2. **Task stop alert**: create a new monitoring rule and select "Multi-table real-time sync task run failure" as the monitoring event. You can add filter attributes such as workspace or task name. If no filter is added, all multi-table real-time sync tasks in the instance are monitored by default.
|
|
401
401
|
|
|
402
|
-
|
|
402
|
+
^
|
|
403
403
|
|
|
404
404
|
#### Configuring In-Task Exception Monitoring
|
|
405
405
|
|
|
@@ -409,21 +409,21 @@ In the monitoring and alerting module, you can configure monitoring rules to tra
|
|
|
409
409
|
|
|
410
410
|
* You can configure an alert for when a table is added to the blacklist, covering two scenarios: Schema Evolution failure and a single field value exceeding the 10 MB size limit.
|
|
411
411
|
|
|
412
|
-
|
|
412
|
+
^
|
|
413
413
|
|
|
414
414
|
* Schema Evolution failure alert: create a new monitoring rule and select "Multi-table real-time sync target table change failure."
|
|
415
415
|
|
|
416
|
-
|
|
416
|
+
^
|
|
417
417
|
|
|
418
418
|
#### Configuring Maximum Latency Monitoring for Real-Time Data Integration Tasks
|
|
419
419
|
|
|
420
420
|
1. **End-to-end sync latency**: create a new monitoring rule and select "Multi-table real-time sync latency." You can add filter attributes such as workspace or task name. If no filter is added, all multi-table real-time sync tasks in the instance are monitored by default.
|
|
421
421
|
|
|
422
|
-
|
|
422
|
+
^
|
|
423
423
|
|
|
424
424
|
2. **Read checkpoint latency**: create a new monitoring rule and select "Multi-table real-time sync read checkpoint latency." You can add filter attributes such as workspace or task name. If no filter is added, all multi-table real-time sync tasks in the instance are monitored by default.
|
|
425
425
|
|
|
426
|
-
|
|
426
|
+
^
|
|
427
427
|
|
|
428
428
|
## Common Error Troubleshooting
|
|
429
429
|
|
|
@@ -443,11 +443,11 @@ In the monitoring and alerting module, you can configure monitoring rules to tra
|
|
|
443
443
|
|
|
444
444
|
* Query the current binlog file and position with `SHOW MASTER STATUS`:
|
|
445
445
|
|
|
446
|
-
|
|
446
|
+
^
|
|
447
447
|
|
|
448
448
|
* Restart the sync task using the file and position from the result.
|
|
449
449
|
|
|
450
|
-
|
|
450
|
+
^
|
|
451
451
|
|
|
452
452
|
* If you need to recover the data that was missed, resync the affected tables.
|
|
453
453
|
|
|
@@ -507,7 +507,7 @@ In the monitoring and alerting module, you can configure monitoring rules to tra
|
|
|
507
507
|
* Cause
|
|
508
508
|
|
|
509
509
|
* The total encoded length of the primary key fields in the source table exceeds the default limit of 128 bytes.
|
|
510
|
-
* In a merge scenario, multiple extended fields (e.g.,
|
|
510
|
+
* In a merge scenario, multiple extended fields (e.g., server\_id, database, schema, table) are included in the composite primary key to prevent conflicts across sharded tables — and the total encoded key length exceeds 128 bytes.
|
|
511
511
|
|
|
512
512
|
* Resolution
|
|
513
513
|
|
|
@@ -567,7 +567,7 @@ In the monitoring and alerting module, you can configure monitoring rules to tra
|
|
|
567
567
|
|
|
568
568
|
* Problem
|
|
569
569
|
|
|
570
|
-
*
|
|
570
|
+
* A table's status automatically changes to "sync stopped." The tooltip on the table object shows errors such as `pk column different`, `pk column type mismatch`, or `invalid modify column`.
|
|
571
571
|
|
|
572
572
|
* Cause
|
|
573
573
|
|
|
@@ -586,3 +586,5 @@ In the monitoring and alerting module, you can configure monitoring rules to tra
|
|
|
586
586
|
* To prevent write conflicts on the target, only source tables with primary key (PK) fields are supported. Tables without a primary key cannot be synced.
|
|
587
587
|
* The sync task will automatically create target tables. To ensure stable task execution and data correctness, avoid manually creating, modifying, or deleting target tables unless absolutely necessary.
|
|
588
588
|
* Schema Evolution supports adding and removing columns on the source. Changing column types and automatically adding new tables are not currently supported.
|
|
589
|
+
|
|
590
|
+
^
|
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
# Navicat Connection Guide
|
|
2
|
+
|
|
1
3
|
Navicat (full name Navicat Premium) is a powerful database management tool widely used for database development, management, and maintenance. This article describes how to use Navicat to connect to Lakehouse via the MySQL protocol.
|
|
2
4
|
|
|
3
5
|
# Prerequisites
|
|
@@ -8,26 +8,26 @@ Think of a Dynamic Table as an "automatically running data processing pipeline"
|
|
|
8
8
|
|
|
9
9
|
## Comparison with Other Table Types
|
|
10
10
|
|
|
11
|
-
| Aspect
|
|
12
|
-
|
|
13
|
-
| Definition
|
|
14
|
-
| Data Storage
|
|
15
|
-
| Data Freshness
|
|
16
|
-
| Update Mechanism
|
|
17
|
-
| Query Optimization | Does not rely on optimizer auto-rewriting
|
|
18
|
-
| DML Support
|
|
19
|
-
| Primary Use
|
|
20
|
-
| Operations
|
|
11
|
+
| Aspect | Dynamic Table | Materialized View | View | Table |
|
|
12
|
+
| ------------------ | ---------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------- | ------------------------------ |
|
|
13
|
+
| Definition | Efficient tool focused on data processing | Special view that pre-computes and stores query results | Stores no data; only saves the query definition | General-purpose storage object |
|
|
14
|
+
| Data Storage | Yes | Yes | No | Yes |
|
|
15
|
+
| Data Freshness | Adjustable; emphasizes processing flexibility | Requires near-real-time updates (ensures accurate rewriting) | Always latest data on each query | Determined by write timing |
|
|
16
|
+
| Update Mechanism | Incremental computation; only processes changed data | Periodic full/incremental refresh | Real-time computation on each query | Manual DML |
|
|
17
|
+
| Query Optimization | Does not rely on optimizer auto-rewriting | **Optimizer automatically identifies and uses pre-stored results** | Recomputed on every query | — |
|
|
18
|
+
| DML Support | No | No | No | Yes |
|
|
19
|
+
| Primary Use | Data processing pipelines (ODS → DWD → DWS) | Query acceleration, result reuse | Logic encapsulation for simple queries | Raw data storage |
|
|
20
|
+
| Operations | Supports adding columns, version rollback | No complex operations | No complex operations | Flexible |
|
|
21
21
|
|
|
22
22
|
**When to use a Dynamic Table**: When you need to automatically compute and store results based on upstream tables. The typical scenario is the ODS → DWD → DWS data processing pipeline. Incremental computation only processes changed data, saving significant compute resources compared to full refreshes.
|
|
23
23
|
|
|
24
24
|
**When not to use a Dynamic Table**:
|
|
25
25
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
26
|
+
* Only need transparent acceleration of existing queries → use a Materialized View (the optimizer auto-rewrites queries to use pre-computed results)
|
|
27
|
+
* Only need logic encapsulation without storing data → use a View
|
|
28
|
+
* Data source is in an external database → use a sync job to write into a regular table; Dynamic Tables can only consume Lakehouse-internal tables
|
|
29
|
+
* Need precise minute-level cron scheduling → use a Studio SQL job
|
|
30
|
+
* Query contains heavy ORDER BY or complex window functions → incremental computation is limited; use a regular View + scheduling
|
|
31
31
|
|
|
32
32
|
## Incremental Computation Principles
|
|
33
33
|
|
|
@@ -36,9 +36,9 @@ Dynamic Tables work based on the Lakehouse's MVCC version management mechanism:
|
|
|
36
36
|
1. **Version awareness**: On each refresh, the system records the source table's last version offset
|
|
37
37
|
2. **Delta capture**: Compares the current version with the last version to identify INSERT/UPDATE/DELETE changes
|
|
38
38
|
3. **Incremental execution**: Executes computation only on changed data; different operators handle changes differently:
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
39
|
+
* Filter/Project: processes only changed rows
|
|
40
|
+
* Join: joins changed rows with historical data from the right table
|
|
41
|
+
* Aggregate: merges changed rows with historical aggregation results
|
|
42
42
|
4. **Result merge**: Merges incremental results into the Dynamic Table via MERGE INTO
|
|
43
43
|
|
|
44
44
|
The system automatically selects incremental or full refresh mode. When the SQL contains operators that don't support incremental processing (e.g., ORDER BY), when the source table change volume is too large, or on the first refresh, the system automatically falls back to full computation.
|
|
@@ -47,11 +47,11 @@ The system automatically selects incremental or full refresh mode. When the SQL
|
|
|
47
47
|
|
|
48
48
|
Dynamic Tables support three scheduling approaches:
|
|
49
49
|
|
|
50
|
-
| Scheduling Approach
|
|
51
|
-
|
|
52
|
-
| DDL-defined refresh interval (`REFRESH INTERVAL`) | Simple scenarios, quick deployment
|
|
53
|
-
| Lakehouse Studio scheduling
|
|
54
|
-
| Third-party scheduling engine
|
|
50
|
+
| Scheduling Approach | Suitable Scenario | Advantages | Disadvantages |
|
|
51
|
+
| ------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
|
|
52
|
+
| DDL-defined refresh interval (`REFRESH INTERVAL`) | Simple scenarios, quick deployment | Simple to use; no external tools needed | No upstream/downstream dependency support; minimum interval is 1 minute |
|
|
53
|
+
| Lakehouse Studio scheduling | Multi-layer DT pipelines requiring dependency control | Visual configuration; supports task dependencies (trigger B after A completes); failure/timeout alerts | Minimum interval is 1 minute |
|
|
54
|
+
| Third-party scheduling engine | Existing scheduling infrastructure requiring flexible control | No interval restrictions; integrates with existing scheduling systems | Introduces external dependencies; requires self-maintenance |
|
|
55
55
|
|
|
56
56
|
> ⚠️ **Note**: The refresh interval must be greater than the duration of a single refresh run; otherwise jobs will queue up. Use `SHOW DYNAMIC TABLE REFRESH HISTORY` to check refresh durations.
|
|
57
57
|
|
|
@@ -109,9 +109,10 @@ SELECT * FROM dws_category_sales;
|
|
|
109
109
|
**Symptom**: Refresh status shows `QUEUED`; data lag keeps growing.
|
|
110
110
|
|
|
111
111
|
**Solution**:
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
112
|
+
|
|
113
|
+
* Use `SHOW DYNAMIC TABLE REFRESH HISTORY` to check refresh durations
|
|
114
|
+
* The refresh interval should be 1.5–2x the single refresh duration
|
|
115
|
+
* If refresh duration keeps growing, incremental computation may have degraded to full computation
|
|
115
116
|
|
|
116
117
|
### Issue 2: Non-Deterministic Functions Cause Inconsistent Results
|
|
117
118
|
|
|
@@ -120,42 +121,44 @@ SELECT * FROM dws_category_sales;
|
|
|
120
121
|
**Symptom**: Function return values differ on each refresh, potentially causing incremental computation results to deviate from expectations.
|
|
121
122
|
|
|
122
123
|
**Solution**:
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
124
|
+
|
|
125
|
+
* Avoid non-deterministic functions in DT definitions
|
|
126
|
+
* For day-based filtering, use parameterized DT + `SESSION_CONFIGS()['dt.args.bizdate']`; see [Dynamic Table Parameterized Definitions](dynamic-table-parameters.md)
|
|
127
|
+
* If you need to record refresh time, use `CURRENT_TIMESTAMP()` in downstream queries rather than in the DT definition
|
|
126
128
|
|
|
127
129
|
### Issue 3: Multi-Layer Pipeline Latency Accumulation
|
|
128
130
|
|
|
129
|
-
**Problem**:
|
|
131
|
+
**Problem**: DT\_A (5-minute refresh) → DT\_B (1-minute refresh); expected DT\_B latency is 1 minute.
|
|
130
132
|
|
|
131
|
-
**Symptom**:
|
|
133
|
+
**Symptom**: DT\_B actual latency is at least 5 minutes.
|
|
132
134
|
|
|
133
135
|
**Solution**:
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
136
|
+
|
|
137
|
+
* The upstream DT's refresh frequency determines the minimum latency achievable downstream
|
|
138
|
+
* Use Studio task dependencies (trigger B after A completes) to avoid polling waits
|
|
139
|
+
* Total pipeline latency = sum of refresh intervals across all layers
|
|
137
140
|
|
|
138
141
|
## Usage Restrictions
|
|
139
142
|
|
|
140
|
-
|
|
141
|
-
|
|
143
|
+
* **Non-deterministic functions not supported**: DT definitions cannot use `RANDOM()`, `CURRENT_TIMESTAMP()`, `CURRENT_DATE()`, or other non-deterministic functions; otherwise incremental refresh results are unpredictable. For time-based filtering, use parameterized DT (`SESSION_CONFIGS()['dt.args.bizdate']`); see [Dynamic Table Parameterized Definitions](dynamic-table-parameters.md).
|
|
144
|
+
* **Direct data modification not supported**: You cannot run `UPDATE`, `DELETE`, or `TRUNCATE` on a Dynamic Table. Data in a Dynamic Table can only be updated through the refresh mechanism.
|
|
142
145
|
|
|
143
146
|
## Cost Implications
|
|
144
147
|
|
|
145
148
|
### Compute Cost
|
|
146
149
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
150
|
+
* Each refresh consumes VCluster CRU resources
|
|
151
|
+
* Higher refresh frequency means greater CRU consumption:
|
|
152
|
+
* `1 DAY`: 1 refresh per day — lowest cost
|
|
153
|
+
* `1 HOUR`: 24 refreshes per day
|
|
154
|
+
* `1 MINUTE`: 1440 refreshes per day — evaluate carefully
|
|
155
|
+
* Incremental refresh saves significantly more resources than full refresh (verify with actual testing)
|
|
153
156
|
|
|
154
157
|
### Storage Cost
|
|
155
158
|
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
+
* Dynamic Tables store computed results and consume storage space
|
|
160
|
+
* Time Travel is supported; the default retention is 1 day of historical versions, which increases storage
|
|
161
|
+
* Time Travel retention can be adjusted with the following command (range: 0–90 days):
|
|
159
162
|
|
|
160
163
|
```sql
|
|
161
164
|
ALTER TABLE dws_category_sales SET PROPERTIES ('data_retention_days'='7');
|
|
@@ -182,23 +185,23 @@ SHOW DYNAMIC TABLE REFRESH HISTORY WHERE name = 'dws_category_sales' LIMIT 10;
|
|
|
182
185
|
|
|
183
186
|
Field descriptions:
|
|
184
187
|
|
|
185
|
-
| Field
|
|
186
|
-
|
|
187
|
-
| `workspace_name`
|
|
188
|
-
| `schema_name`
|
|
189
|
-
| `name`
|
|
190
|
-
| `virtual_cluster`
|
|
191
|
-
| `start_time`
|
|
192
|
-
| `end_time`
|
|
193
|
-
| `duration`
|
|
194
|
-
| `state`
|
|
195
|
-
| `refresh_trigger`
|
|
196
|
-
| `suspended_reason` | Reason scheduling is suspended (null when not suspended)
|
|
197
|
-
| `refresh_mode`
|
|
198
|
-
| `error_message`
|
|
199
|
-
| `source_tables`
|
|
200
|
-
| `stats`
|
|
201
|
-
| `job_id`
|
|
188
|
+
| Field | Description |
|
|
189
|
+
| ------------------ | ----------------------------------------------------------------------------------------------------------------- |
|
|
190
|
+
| `workspace_name` | Workspace name |
|
|
191
|
+
| `schema_name` | Schema name |
|
|
192
|
+
| `name` | Dynamic Table name |
|
|
193
|
+
| `virtual_cluster` | Cluster used for the refresh |
|
|
194
|
+
| `start_time` | Refresh start time |
|
|
195
|
+
| `end_time` | Refresh end time |
|
|
196
|
+
| `duration` | Refresh duration |
|
|
197
|
+
| `state` | Job state: `setup` / `resuming cluster` / `queued` / `running` / `SUCCEED` / `FAILED` |
|
|
198
|
+
| `refresh_trigger` | Trigger type: `MANUAL` (user-triggered, including Studio scheduling) / `LH_SCHEDULED` (Lakehouse auto-scheduling) |
|
|
199
|
+
| `suspended_reason` | Reason scheduling is suspended (null when not suspended) |
|
|
200
|
+
| `refresh_mode` | Refresh mode: `INCREMENTAL` / `FULL` / `NO_DATA` (no changes) |
|
|
201
|
+
| `error_message` | Error message on failure |
|
|
202
|
+
| `source_tables` | List of source tables the Dynamic Table depends on (JSON format) |
|
|
203
|
+
| `stats` | Incremental refresh stats: `rows_inserted` / `rows_deleted` (values are string type) |
|
|
204
|
+
| `job_id` | Job ID; click to view the Job Profile and incremental execution plan |
|
|
202
205
|
|
|
203
206
|
### Modifying and Dropping
|
|
204
207
|
|
|
@@ -217,7 +220,9 @@ UNDROP TABLE dws_category_sales;
|
|
|
217
220
|
|
|
218
221
|
## Related Documentation
|
|
219
222
|
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
223
|
+
* [Dynamic Table Overview](dynamic_table_summary.md) — Core concepts and working principles
|
|
224
|
+
* [CREATE DYNAMIC TABLE](create-dynamic-table.md) — Complete syntax
|
|
225
|
+
* [Incremental Computation](incremental-computing.md) — Incremental refresh principles
|
|
226
|
+
* [Real-Time Pipeline Selection Guide](realtime-pipeline-selection-guide.md) — Choosing between Pipe, Stream, and Dynamic Table
|
|
227
|
+
|
|
228
|
+
^
|
|
@@ -63,6 +63,8 @@ ALTER VCLUSTER my_gp_cluster SUSPEND;
|
|
|
63
63
|
SHOW VCLUSTERS;
|
|
64
64
|
```
|
|
65
65
|
|
|
66
|
+
> ⚠️ **Note**: VCluster names are automatically converted to **uppercase** when stored (unlike table names and schema names, which are lowercased). For example, a cluster created as `my_gp_cluster` will have the actual name `MY_GP_CLUSTER`. References are case-insensitive, but `SHOW VCLUSTERS` always displays uppercase names.
|
|
67
|
+
|
|
66
68
|
## Cost Implications
|
|
67
69
|
|
|
68
70
|
### Compute Cost
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# CreateSession - Create a Conversation Session
|
|
2
|
+
|
|
3
|
+
Create a new conversation session (Session) to organize one or more question-and-answer interactions.
|
|
4
|
+
|
|
5
|
+
## Interface Description
|
|
6
|
+
|
|
7
|
+
A Session represents a conversation context. After creation, a `sessionId` is returned. Subsequent question requests must be associated with a Session.
|
|
8
|
+
|
|
9
|
+
If the client already has a reusable `sessionId`, this step can be skipped and a question can be submitted directly.
|
|
10
|
+
|
|
11
|
+
### Usage Notes
|
|
12
|
+
|
|
13
|
+
- A single Session can contain multiple questions.
|
|
14
|
+
- A Session can be reused multiple times; you do not need to create a new one for every question.
|
|
15
|
+
|
|
16
|
+
## Request Method
|
|
17
|
+
|
|
18
|
+
```
|
|
19
|
+
POST /open/session/safe_new?tenantId={tenantId}&userId={userId}&loginToken={loginToken}
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
> 💡 **Note**: `tenantId`, `userId`, and `loginToken` must appear **both** in the URL query parameters and in the request body. Missing query parameters will cause the request to fail.
|
|
23
|
+
|
|
24
|
+
## Request Parameters
|
|
25
|
+
|
|
26
|
+
| Parameter | Location | Type | Required | Description |
|
|
27
|
+
|-----------|----------|------|----------|-------------|
|
|
28
|
+
| tenantId | Query + Body | Integer | Yes | Tenant ID |
|
|
29
|
+
| userId | Query + Body | Integer | Yes | User ID |
|
|
30
|
+
| domainId | Body | Integer | Yes | Data domain ID, corresponding to a dataset |
|
|
31
|
+
| title | Body | String | No | Session title, used to identify the Session |
|
|
32
|
+
| loginToken | Query + Body | String | Yes | Authentication Token obtained via GenerateAuthToken |
|
|
33
|
+
|
|
34
|
+
## Response Parameters
|
|
35
|
+
|
|
36
|
+
| Parameter | Type | Description |
|
|
37
|
+
|-----------|------|-------------|
|
|
38
|
+
| success | Boolean | Whether the request was successful |
|
|
39
|
+
| data | Integer | The sessionId of the newly created session |
|
|
40
|
+
|
|
41
|
+
## Request Example
|
|
42
|
+
|
|
43
|
+
```http
|
|
44
|
+
POST /open/session/safe_new
|
|
45
|
+
Content-Type: application/json
|
|
46
|
+
|
|
47
|
+
{
|
|
48
|
+
"tenantId": 10,
|
|
49
|
+
"userId": 1,
|
|
50
|
+
"domainId": 106,
|
|
51
|
+
"title": "Q1 Sales Analysis",
|
|
52
|
+
"loginToken": "eyJhbGciOiJIUzI1NiJ9..."
|
|
53
|
+
}
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Response Example
|
|
57
|
+
|
|
58
|
+
Successful response:
|
|
59
|
+
|
|
60
|
+
```json
|
|
61
|
+
{
|
|
62
|
+
"success": true,
|
|
63
|
+
"data": 4729
|
|
64
|
+
}
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The value `4729` in `data` is the newly created `sessionId`.
|
|
68
|
+
|
|
69
|
+
## Error Codes
|
|
70
|
+
|
|
71
|
+
| Error Code | Description |
|
|
72
|
+
|------------|-------------|
|
|
73
|
+
| success=false | Token is invalid, parameters are missing, or insufficient permissions |
|
|
74
|
+
|
|
75
|
+
## Related Documentation
|
|
76
|
+
|
|
77
|
+
- [GenerateAuthToken](open-api-generate-auth-token) — Previous step: obtain authentication Token
|
|
78
|
+
- [Text2InsightQuery](open-api-text2insight-query) — Next step: submit a data analysis request
|
|
79
|
+
- [Quick Start](open-api-quick-start) — Complete end-to-end example
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# GenerateAuthToken - Obtain Authentication Token
|
|
2
|
+
|
|
3
|
+
Obtain an authentication Token for subsequent API calls using an application secret key (appSecretKey).
|
|
4
|
+
|
|
5
|
+
## Interface Description
|
|
6
|
+
|
|
7
|
+
Before calling any business interface in Open Text2Insight, you must first use this interface to obtain a login Token. This Token serves as the identity credential for all subsequent requests.
|
|
8
|
+
|
|
9
|
+
### Usage Notes
|
|
10
|
+
|
|
11
|
+
- The appSecretKey is generated and distributed by the platform administrator in the backend.
|
|
12
|
+
- The Token has an expiration time; you must obtain a new one after it expires.
|
|
13
|
+
- Keep your appSecretKey secure to prevent leakage.
|
|
14
|
+
|
|
15
|
+
## Request Method
|
|
16
|
+
|
|
17
|
+
```
|
|
18
|
+
GET /open/api/v1/appSecretKey/generateAuthToken
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Request Parameters
|
|
22
|
+
|
|
23
|
+
| Parameter | Location | Type | Required | Description |
|
|
24
|
+
|-----------|----------|------|----------|-------------|
|
|
25
|
+
| appSecretKey | Query | String | Yes | Application secret key, assigned by the administrator |
|
|
26
|
+
|
|
27
|
+
## Response Parameters
|
|
28
|
+
|
|
29
|
+
| Parameter | Type | Description |
|
|
30
|
+
|-----------|------|-------------|
|
|
31
|
+
| success | Boolean | Whether the request was successful |
|
|
32
|
+
| data.token | String | Authentication Token, used for subsequent interface calls |
|
|
33
|
+
|
|
34
|
+
## Request Example
|
|
35
|
+
|
|
36
|
+
```http
|
|
37
|
+
GET /open/api/v1/appSecretKey/generateAuthToken?appSecretKey=your_secret_key_here
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Response Example
|
|
41
|
+
|
|
42
|
+
Successful response:
|
|
43
|
+
|
|
44
|
+
```json
|
|
45
|
+
{
|
|
46
|
+
"success": true,
|
|
47
|
+
"data": {
|
|
48
|
+
"token": "eyJhbGciOiJIUzI1NiJ9..."
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Error Codes
|
|
54
|
+
|
|
55
|
+
| Error Code | Description |
|
|
56
|
+
|------------|-------------|
|
|
57
|
+
| success=false | appSecretKey is invalid or has expired |
|
|
58
|
+
|
|
59
|
+
## Related Documentation
|
|
60
|
+
|
|
61
|
+
- [Open API Overview](open-api-overview) — Interface list and general description
|
|
62
|
+
- [Quick Start](open-api-quick-start) — Complete call flow after obtaining the Token
|
|
63
|
+
- [CreateSession](open-api-create-session) — Next step: create a session using the Token
|