@clickzetta/cz-cli-darwin-arm64 0.5.15 → 0.5.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/cz-cli +0 -0
- package/bin/skills/lakehouse-doc-en/SKILL.md +6 -11
- package/bin/skills/lakehouse-doc-en/references/AIGateway.md +58 -13
- package/bin/skills/lakehouse-doc-en/references/Computation.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/DataSource_Amazon_DocumentDB.md +3 -1
- package/bin/skills/lakehouse-doc-en/references/Foreach.md +14 -14
- package/bin/skills/lakehouse-doc-en/references/JDBC-Driver.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/LakehouseAI-overview.md +21 -8
- package/bin/skills/lakehouse-doc-en/references/LakehouseDataGPT-tour.md +4 -9
- package/bin/skills/lakehouse-doc-en/references/LakehouseStudio-tour.md +14 -19
- package/bin/skills/lakehouse-doc-en/references/Lakehouse_Zilliz_MakeDataReadyforBIandAI.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/Logstash.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/Migrate_Spark_DataEngineeringBestPractices_Project_to_Lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/Notebook.md +17 -17
- package/bin/skills/lakehouse-doc-en/references/RemoteFunction-as-udf.md +14 -14
- package/bin/skills/lakehouse-doc-en/references/SQL_External_Catalog_Guide.md +1 -9
- package/bin/skills/lakehouse-doc-en/references/SUMMARY.md +59 -29
- package/bin/skills/lakehouse-doc-en/references/WINDOWFUNCTION.md +99 -57
- package/bin/skills/lakehouse-doc-en/references/Zettapark_Data_Engineering_Demo.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/access-control-configuration.md +1 -8
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-2-5-1.0.md +16 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-3-29-1.0.2.md +14 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-3-8-1.0.1.md +16 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-4-28-1.1.md +29 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-12-1.1.1.md +18 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-15-1.2.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-21-1.3.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-28-1.4.md +10 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-6-3-1.5.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/alicloud-arn-externalid.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/answer-accuracy-improve.md +120 -103
- package/bin/skills/lakehouse-doc-en/references/application-list.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/approval-list.md +16 -17
- package/bin/skills/lakehouse-doc-en/references/batch-load-parquet-file-into-lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/batch_sync.md +9 -9
- package/bin/skills/lakehouse-doc-en/references/batch_sync_Sop.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/batchloadparquetfileintoLakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/bulkloadv1-python-sdk.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/chart-auto-refresh-guide.md +12 -6
- package/bin/skills/lakehouse-doc-en/references/clickzetta-sample-data.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/code_approval.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/composite_task.md +31 -42
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_environment_and_data_generate.md +6 -9
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_javasdk_bulkload_realtime.md +4 -10
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_kafka_realtime_sync.md +1 -10
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_local_file_into_table_by_studio.md +0 -6
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_batchload_public_network.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_python_node.md +2 -7
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_realtime_cdc_public_network.md +13 -18
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_sql_insert.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/concepts.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/config-datasource.md +5 -7
- package/bin/skills/lakehouse-doc-en/references/connect-with-cli.md +116 -72
- package/bin/skills/lakehouse-doc-en/references/connect-with-cz-cli.md +151 -0
- package/bin/skills/lakehouse-doc-en/references/continue-job.md +9 -17
- package/bin/skills/lakehouse-doc-en/references/create-api-connection.md +315 -286
- package/bin/skills/lakehouse-doc-en/references/create-catalog-connection.md +1 -0
- package/bin/skills/lakehouse-doc-en/references/create-dynamic-table.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/create-external-catalog.md +85 -22
- package/bin/skills/lakehouse-doc-en/references/create-table-ddl.md +45 -0
- package/bin/skills/lakehouse-doc-en/references/creating_alicloud_privatelinkendpoint.md +4 -6
- package/bin/skills/lakehouse-doc-en/references/creating_alicloud_privatelinkservice.md +4 -7
- package/bin/skills/lakehouse-doc-en/references/creating_tencentcloud_privatelinkendpoint.md +2 -7
- package/bin/skills/lakehouse-doc-en/references/creating_tencentcloud_privatelinkservice.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/cz-cli-agent.md +15 -10
- package/bin/skills/lakehouse-doc-en/references/cz-cli-datasource.md +0 -8
- package/bin/skills/lakehouse-doc-en/references/cz-cli-sql.md +2 -45
- package/bin/skills/lakehouse-doc-en/references/cz-cli.md +53 -42
- package/bin/skills/lakehouse-doc-en/references/dashboard-version-management-guide.md +12 -4
- package/bin/skills/lakehouse-doc-en/references/data-integration-intro.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/data-integration.md +29 -27
- package/bin/skills/lakehouse-doc-en/references/data-load-summary.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/data-quality.md +25 -25
- package/bin/skills/lakehouse-doc-en/references/data-sharing.md +31 -54
- package/bin/skills/lakehouse-doc-en/references/data-sources.md +45 -45
- package/bin/skills/lakehouse-doc-en/references/data_catalog.md +23 -25
- package/bin/skills/lakehouse-doc-en/references/data_privacy.md +5 -2
- package/bin/skills/lakehouse-doc-en/references/data_sharing_between_accounts_guide.md +0 -4
- package/bin/skills/lakehouse-doc-en/references/data_visualization.md +4 -15
- package/bin/skills/lakehouse-doc-en/references/dataagent.md +39 -7
- package/bin/skills/lakehouse-doc-en/references/databricks-delta-to-lakehouse-migration.md +168 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-dlt-to-lakehouse-migration.md +331 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-external-catalog-practice.md +367 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-jobs-to-studio-migration.md +199 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-notebook-to-studio-migration.md +350 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-uc-governance-to-lakehouse-migration.md +327 -0
- package/bin/skills/lakehouse-doc-en/references/datagpt-model-config.md +34 -0
- package/bin/skills/lakehouse-doc-en/references/datagpt_data_source.md +50 -37
- package/bin/skills/lakehouse-doc-en/references/datagpt_introduction.md +55 -79
- package/bin/skills/lakehouse-doc-en/references/datagpt_quickstart.md +50 -64
- package/bin/skills/lakehouse-doc-en/references/datalake-acceleration.md +75 -2
- package/bin/skills/lakehouse-doc-en/references/dbt-databricks-to-clickzetta-migration.md +242 -0
- package/bin/skills/lakehouse-doc-en/references/dynamic-mask.md +30 -30
- package/bin/skills/lakehouse-doc-en/references/dynamic-table-bestpractice.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/dynamic-table-introduce.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/dynamic_table_summary.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/eco_integration/streamlit.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/eco_integration/superset.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/ecosystem-all.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/ecosystem.md +145 -0
- package/bin/skills/lakehouse-doc-en/references/external-catalog-summary.md +33 -38
- package/bin/skills/lakehouse-doc-en/references/external-function-combo-practice.md +466 -0
- package/bin/skills/lakehouse-doc-en/references/f6fc6447ee.md +7 -9
- package/bin/skills/lakehouse-doc-en/references/federation-query.md +56 -6
- package/bin/skills/lakehouse-doc-en/references/finebi-mysql.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/get-started-with-sample-data.md +10 -11
- package/bin/skills/lakehouse-doc-en/references/gitfolder.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/grant-privileges.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/iceberg-rest-catalog-databricks.md +166 -0
- package/bin/skills/lakehouse-doc-en/references/ide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/if_else_task.md +59 -57
- package/bin/skills/lakehouse-doc-en/references/input_output.md +10 -7
- package/bin/skills/lakehouse-doc-en/references/jobprofile-bestpractices.md +60 -64
- package/bin/skills/lakehouse-doc-en/references/kafka-connection.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/key-concepts.md +146 -117
- package/bin/skills/lakehouse-doc-en/references/lakehouse-ai-gateway-cz-cli.md +317 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-ai-sql-analysis.md +345 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-dqc-guide.md +300 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-medallion-sql-dt-guide.md +543 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-multi-cloud-acceleration.md +274 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-multimodal-ai-pipeline.md +198 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-quick-experience_guide.md +49 -52
- package/bin/skills/lakehouse-doc-en/references/lakehouse-volume-pipe-acceleration-guide.md +380 -0
- package/bin/skills/lakehouse-doc-en/references/langchain-plug-installation.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/management.md +4 -9
- package/bin/skills/lakehouse-doc-en/references/medallion-lakehouse-from-scratch.md +2 -1
- package/bin/skills/lakehouse-doc-en/references/metrics_answer_build.md +58 -21
- package/bin/skills/lakehouse-doc-en/references/migrate-spark-data-engineering-best-practices-to-lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/mindsdb.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/monitoring_and_alerting.md +65 -60
- package/bin/skills/lakehouse-doc-en/references/monitoring_item_specification.md +33 -33
- package/bin/skills/lakehouse-doc-en/references/multitable_batch_sync.md +16 -16
- package/bin/skills/lakehouse-doc-en/references/multitable_realtime_sync.md +65 -72
- package/bin/skills/lakehouse-doc-en/references/multitable_realtime_sync_sop.md +54 -52
- package/bin/skills/lakehouse-doc-en/references/navicat-mysql.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/om-dynamic-table.md +71 -66
- package/bin/skills/lakehouse-doc-en/references/om-vcluster.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-create-session.md +79 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-generate-auth-token.md +63 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-overview.md +96 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-quick-start.md +286 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-response-guide.md +264 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-safe-question-poll.md +201 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-text2insight-query.md +99 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-text2insight-stop.md +74 -0
- package/bin/skills/lakehouse-doc-en/references/overview.md +6 -7
- package/bin/skills/lakehouse-doc-en/references/permission-application.md +5 -5
- package/bin/skills/lakehouse-doc-en/references/pipe-introduction.md +1 -0
- package/bin/skills/lakehouse-doc-en/references/pipe-kafka-table-stream.md +72 -70
- package/bin/skills/lakehouse-doc-en/references/pipe-kafka.md +105 -110
- package/bin/skills/lakehouse-doc-en/references/pipe-overview.md +40 -40
- package/bin/skills/lakehouse-doc-en/references/pipe-storage-object.md +43 -48
- package/bin/skills/lakehouse-doc-en/references/pipe-summary.md +14 -4
- package/bin/skills/lakehouse-doc-en/references/pipe-syntax.md +58 -151
- package/bin/skills/lakehouse-doc-en/references/practice_python_task.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/pricing-ai-gateway.md +181 -0
- package/bin/skills/lakehouse-doc-en/references/pricing-lakehouse.md +316 -0
- package/bin/skills/lakehouse-doc-en/references/pricing.md +44 -288
- package/bin/skills/lakehouse-doc-en/references/private-link-general.md +0 -2
- package/bin/skills/lakehouse-doc-en/references/pyspark-to-zettapark-migration-f1.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python-igs.md +7 -3
- package/bin/skills/lakehouse-doc-en/references/python-sample-put-github-rt-events.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python-task.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector_advanced.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector_examples.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/python_sdk_guide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python_shell_datasource.md +11 -9
- package/bin/skills/lakehouse-doc-en/references/quick_start_batch_sync_data.md +9 -18
- package/bin/skills/lakehouse-doc-en/references/quick_start_bi_analysis.md +8 -25
- package/bin/skills/lakehouse-doc-en/references/quick_start_create_workspace.md +4 -6
- package/bin/skills/lakehouse-doc-en/references/quick_start_data_quality.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quick_start_etl.md +16 -20
- package/bin/skills/lakehouse-doc-en/references/quick_start_monitoring_and_alerting.md +10 -18
- package/bin/skills/lakehouse-doc-en/references/quick_start_sql_query.md +7 -10
- package/bin/skills/lakehouse-doc-en/references/quick_start_upload_data.md +5 -7
- package/bin/skills/lakehouse-doc-en/references/quick_start_user_management.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quick_start_workspace.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/quick_start_workspace_user.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quickstart.md +69 -56
- package/bin/skills/lakehouse-doc-en/references/quickstart_datashare_between_companies.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/quickstart_envirment_for_team.md +0 -24
- package/bin/skills/lakehouse-doc-en/references/realtime-pipeline-selection-guide.md +1 -2
- package/bin/skills/lakehouse-doc-en/references/realtime-sales-dashboard-with-dynamic-table.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/realtime_sync.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/release-note-2026-05-19.md +5 -3
- package/bin/skills/lakehouse-doc-en/references/revoke-privileges.md +3 -1
- package/bin/skills/lakehouse-doc-en/references/roles.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/row-filter.md +165 -0
- package/bin/skills/lakehouse-doc-en/references/row_level_permission.md +30 -19
- package/bin/skills/lakehouse-doc-en/references/scheduled_task.md +28 -21
- package/bin/skills/lakehouse-doc-en/references/security_overview.md +99 -21
- package/bin/skills/lakehouse-doc-en/references/set-command.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/setup.md +13 -15
- package/bin/skills/lakehouse-doc-en/references/show-grants.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/snowflake-dynamic-tables-to-lakehouse.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/spark-connector-summary.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/sql_functions/context_functions/current_vcluster.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/sso-configuration.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/streaming_pipeline_with_dynamic_table.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/studio-incremental-sync-practice.md +27 -23
- package/bin/skills/lakehouse-doc-en/references/studio-shell-task.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/supported-cloud-platforms.md +32 -0
- package/bin/skills/lakehouse-doc-en/references/table_rendering.md +18 -12
- package/bin/skills/lakehouse-doc-en/references/task-develop.md +89 -91
- package/bin/skills/lakehouse-doc-en/references/task_development.md +19 -17
- package/bin/skills/lakehouse-doc-en/references/task_group.md +16 -14
- package/bin/skills/lakehouse-doc-en/references/task_instance.md +21 -21
- package/bin/skills/lakehouse-doc-en/references/task_param.md +38 -35
- package/bin/skills/lakehouse-doc-en/references/task_param_reference.md +81 -79
- package/bin/skills/lakehouse-doc-en/references/task_scheduling_dependency.md +20 -21
- package/bin/skills/lakehouse-doc-en/references/tencentcloud_arn_and_externalid.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/trial-account-quotas-and-limits.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/tutorial_connect_to_lakehouse.md +69 -0
- package/bin/skills/lakehouse-doc-en/references/tutorials.md +4 -1
- package/bin/skills/lakehouse-doc-en/references/unique-key.md +167 -0
- package/bin/skills/lakehouse-doc-en/references/usageandbillingview.md +138 -0
- package/bin/skills/lakehouse-doc-en/references/use-dbt-dev.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/use-java-sdk-realtime-uploaddata.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/use-java-sdk-upload-data-local.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/use-models.md +128 -0
- package/bin/skills/lakehouse-doc-en/references/use-mysql-client.md +81 -81
- package/bin/skills/lakehouse-doc-en/references/use-python-sdk-upload-data.md +10 -12
- package/bin/skills/lakehouse-doc-en/references/user-identification.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/user_permission_grand_guide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/using-udf-in-dynamic-table.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/vc_cache.md +18 -22
- package/bin/skills/lakehouse-doc-en/references/vcluster_size_description.md +33 -31
- package/bin/skills/lakehouse-doc-en/references/virtual-cluster.md +43 -45
- package/bin/skills/lakehouse-doc-en/references/web-job-history.md +94 -108
- package/bin/skills/lakehouse-doc-en/references/web_search.md +16 -7
- package/bin/skills/lakehouse-doc-en/references/zettapark-data-engineering-demo.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/zettapark-dataframe-guide.md +144 -70
- package/bin/skills/lakehouse-doc-en/references/zettapark-dynamic-table-guide.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-etl-guide.md +73 -33
- package/bin/skills/lakehouse-doc-en/references/zettapark-feature-engineering.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-functions-guide.md +75 -46
- package/bin/skills/lakehouse-doc-en/references/zettapark-quick-start.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-stream-guide.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/zettapark-volume-guide.md +93 -29
- package/package.json +1 -1
- package/bin/skills/lakehouse-doc-en/references/CLAUDE.md +0 -606
- package/bin/skills/lakehouse-doc-en/references/modelprice.md +0 -155
|
@@ -24,24 +24,27 @@ doc_github_clean (standardized fields, ready for direct query and analysis)
|
|
|
24
24
|
|
|
25
25
|
---
|
|
26
26
|
|
|
27
|
-
##
|
|
27
|
+
## Shell Task: Pull Raw Data
|
|
28
28
|
|
|
29
29
|
### Script
|
|
30
30
|
|
|
31
|
+
The Shell layer sets task parameters and calls embedded Python:
|
|
32
|
+
|
|
31
33
|
```bash
|
|
32
34
|
#!/bin/bash
|
|
33
|
-
# Task parameter: biz_date = $[yyyy-MM-dd]
|
|
34
35
|
BIZ_DATE='${biz_date}'
|
|
35
36
|
echo "Fetch date: $BIZ_DATE"
|
|
37
|
+
```
|
|
36
38
|
|
|
37
|
-
python3 - << PYEOF
|
|
39
|
+
The embedded Python code handles table creation, API fetching, and writing to the raw layer (executed via `python3 - << PYEOF ... PYEOF` heredoc):
|
|
40
|
+
|
|
41
|
+
```python
|
|
38
42
|
import urllib.request, json
|
|
39
43
|
from clickzetta_dbutils import get_active_lakehouse_engine
|
|
40
44
|
from sqlalchemy import text
|
|
41
45
|
|
|
42
46
|
biz_date = '$BIZ_DATE'
|
|
43
47
|
|
|
44
|
-
# ── 1. Create table (idempotent) ──────────────────────────────────────────
|
|
45
48
|
engine = get_active_lakehouse_engine(schema="doc_connector_demo")
|
|
46
49
|
with engine.connect() as conn:
|
|
47
50
|
conn.execute(text("CREATE SCHEMA IF NOT EXISTS doc_connector_demo"))
|
|
@@ -57,14 +60,12 @@ with engine.connect() as conn:
|
|
|
57
60
|
)
|
|
58
61
|
"""))
|
|
59
62
|
|
|
60
|
-
# ── 2. Fetch GitHub Releases API ──────────────────────────────────────────
|
|
61
63
|
url = "https://api.github.com/repos/clickzetta/dbt-clickzetta/releases?per_page=10"
|
|
62
64
|
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
|
|
63
65
|
with urllib.request.urlopen(req, timeout=15) as r:
|
|
64
66
|
releases = json.loads(r.read())
|
|
65
67
|
print(f"API returned {len(releases)} records")
|
|
66
68
|
|
|
67
|
-
# ── 3. Idempotent write to raw layer ──────────────────────────────────────
|
|
68
69
|
with engine.connect() as conn:
|
|
69
70
|
conn.execute(text(f"DELETE FROM doc_connector_demo.doc_github_raw WHERE load_date = '{biz_date}'"))
|
|
70
71
|
for rel in releases:
|
|
@@ -79,7 +80,6 @@ with engine.connect() as conn:
|
|
|
79
80
|
))
|
|
80
81
|
print(f"Wrote {len(releases)} raw records, load_date={biz_date}")
|
|
81
82
|
|
|
82
|
-
# ── 4. Verify ─────────────────────────────────────────────────────────────
|
|
83
83
|
with engine.connect() as conn:
|
|
84
84
|
result = conn.execute(text(
|
|
85
85
|
f"SELECT tag_name, published_at FROM doc_connector_demo.doc_github_raw "
|
|
@@ -87,7 +87,6 @@ with engine.connect() as conn:
|
|
|
87
87
|
))
|
|
88
88
|
for row in result:
|
|
89
89
|
print(f" {row[0]:15s} {row[1]}")
|
|
90
|
-
PYEOF
|
|
91
90
|
```
|
|
92
91
|
|
|
93
92
|
### Creating and Executing the Task
|
|
@@ -106,7 +105,7 @@ PYEOF
|
|
|
106
105
|
cz-cli task create github_raw_fetch --type shell --profile <your-profile>
|
|
107
106
|
cz-cli task save-content github_raw_fetch --file github_raw_fetch.sh \
|
|
108
107
|
--params '{"biz_date": "$[yyyy-MM-dd]"}' --profile <your-profile>
|
|
109
|
-
cz-cli task save-config github_raw_fetch --vcluster
|
|
108
|
+
cz-cli task save-config github_raw_fetch --vcluster DEFAULT --retry-count 1 --profile <your-profile>
|
|
110
109
|
cz-cli task save-cron github_raw_fetch --cron "0 1 * * *" --profile <your-profile>
|
|
111
110
|
cz-cli task online github_raw_fetch -y --profile <your-profile>
|
|
112
111
|
cz-cli task execute github_raw_fetch --param "biz_date=2024-12-01" --profile <your-profile>
|
|
@@ -125,10 +124,12 @@ Wrote 10 raw records, load_date=2024-12-01
|
|
|
125
124
|
|
|
126
125
|
---
|
|
127
126
|
|
|
128
|
-
##
|
|
127
|
+
## Python Task: Clean and Standardize
|
|
129
128
|
|
|
130
129
|
### Script
|
|
131
130
|
|
|
131
|
+
Task parameters and Session creation:
|
|
132
|
+
|
|
132
133
|
```python
|
|
133
134
|
from clickzetta_dbutils import get_active_lakehouse_engine
|
|
134
135
|
from clickzetta.zettapark.session import Session
|
|
@@ -136,12 +137,9 @@ from clickzetta.zettapark import functions as F
|
|
|
136
137
|
from urllib.parse import urlparse, parse_qs
|
|
137
138
|
import re
|
|
138
139
|
|
|
139
|
-
# ── Task parameters ───────────────────────────────────────────────────────
|
|
140
|
-
# biz_date = $[yyyy-MM-dd]
|
|
141
140
|
biz_date = '${biz_date}'
|
|
142
141
|
print(f"Cleaning date: {biz_date}")
|
|
143
142
|
|
|
144
|
-
# ── 1. Create ZettaPark Session ───────────────────────────────────────────
|
|
145
143
|
engine = get_active_lakehouse_engine(schema="doc_connector_demo")
|
|
146
144
|
url_str = str(engine.url)
|
|
147
145
|
parsed = urlparse(url_str.replace('clickzetta://', 'https://'))
|
|
@@ -154,25 +152,25 @@ session = Session.builder.configs({
|
|
|
154
152
|
"magic_token": params['magic_token'][0],
|
|
155
153
|
"workspace": parsed.path.lstrip('/'),
|
|
156
154
|
"schema": params.get('schema', ['public'])[0],
|
|
157
|
-
"vcluster": params.get('virtualcluster', ['
|
|
155
|
+
"vcluster": params.get('virtualcluster', ['DEFAULT'])[0],
|
|
158
156
|
}).getOrCreate()
|
|
159
157
|
|
|
160
158
|
print(f"Session ready: {session.get_current_catalog()}.{session.get_current_schema()}")
|
|
159
|
+
```
|
|
161
160
|
|
|
162
|
-
|
|
161
|
+
Read from the raw layer and convert to pandas for cleaning:
|
|
162
|
+
|
|
163
|
+
```python
|
|
163
164
|
raw = session.table("doc_connector_demo.doc_github_raw").filter(
|
|
164
165
|
F.col("load_date") == biz_date
|
|
165
166
|
)
|
|
166
167
|
print(f"Raw layer record count: {raw.count()}")
|
|
167
168
|
|
|
168
|
-
# ── 3. Convert to pandas for cleaning ────────────────────────────────────
|
|
169
169
|
df = raw.to_pandas()
|
|
170
170
|
|
|
171
|
-
# Deduplication: keep only one record per tag_name
|
|
172
171
|
df = df.drop_duplicates(subset=['tag_name'], keep='first')
|
|
173
172
|
print(f"After deduplication: {len(df)} records")
|
|
174
173
|
|
|
175
|
-
# Parse version number: v1.7.5 → major=1, minor=7, patch=5
|
|
176
174
|
def parse_version(tag):
|
|
177
175
|
m = re.match(r'v?(\d+)\.(\d+)\.(\d+)', tag)
|
|
178
176
|
if m:
|
|
@@ -183,14 +181,16 @@ df[['major', 'minor', 'patch']] = df['tag_name'].apply(
|
|
|
183
181
|
lambda t: parse_version(t)
|
|
184
182
|
).apply(lambda x: x if x else (None, None, None)).tolist()
|
|
185
183
|
|
|
186
|
-
# Filter out invalid versions (tags that don't match the vX.Y.Z format)
|
|
187
184
|
df = df.dropna(subset=['major'])
|
|
188
185
|
df['major'] = df['major'].astype(int)
|
|
189
186
|
df['minor'] = df['minor'].astype(int)
|
|
190
187
|
df['patch'] = df['patch'].astype(int)
|
|
191
188
|
print(f"Valid versions: {len(df)} records")
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Create table, write back to clean layer, and print summary:
|
|
192
192
|
|
|
193
|
-
|
|
193
|
+
```python
|
|
194
194
|
session.sql("""
|
|
195
195
|
CREATE TABLE IF NOT EXISTS doc_connector_demo.doc_github_clean (
|
|
196
196
|
load_date STRING,
|
|
@@ -212,7 +212,6 @@ result_df = session.create_dataframe(
|
|
|
212
212
|
result_df.write.mode("append").save_as_table("doc_connector_demo.doc_github_clean")
|
|
213
213
|
print(f"Wrote to clean layer: {len(df)} records, load_date={biz_date}")
|
|
214
214
|
|
|
215
|
-
# ── 5. Summary output ─────────────────────────────────────────────────────
|
|
216
215
|
summary = (
|
|
217
216
|
session.table("doc_connector_demo.doc_github_clean")
|
|
218
217
|
.filter(F.col("load_date") == biz_date)
|
|
@@ -238,14 +237,19 @@ session.close()
|
|
|
238
237
|
|
|
239
238
|
**cz-cli**
|
|
240
239
|
|
|
240
|
+
First get the task_id of `github_raw_fetch` (needed for configuring the dependency):
|
|
241
|
+
|
|
241
242
|
```bash
|
|
242
|
-
# First get the task_id of github_raw_fetch (needed for configuring the dependency)
|
|
243
243
|
cz-cli task list --profile <your-profile>
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
Then create the task and configure the dependency:
|
|
244
247
|
|
|
248
|
+
```bash
|
|
245
249
|
cz-cli task create github_clean --type python --profile <your-profile>
|
|
246
250
|
cz-cli task save-content github_clean --file github_clean.py \
|
|
247
251
|
--params '{"biz_date": "$[yyyy-MM-dd]"}' --profile <your-profile>
|
|
248
|
-
cz-cli task save-config github_clean --vcluster
|
|
252
|
+
cz-cli task save-config github_clean --vcluster DEFAULT --retry-count 1 \
|
|
249
253
|
--deps replace \
|
|
250
254
|
--dep-tasks '[{"taskId": <github_raw_fetch_task_id>, "taskName": "github_raw_fetch"}]' \
|
|
251
255
|
--profile <your-profile>
|
|
@@ -138,7 +138,7 @@ cz-cli task save-content shell_etl --file shell_etl.sh \
|
|
|
138
138
|
--profile <your-profile>
|
|
139
139
|
|
|
140
140
|
# Configure scheduling
|
|
141
|
-
cz-cli task save-config shell_etl --vcluster
|
|
141
|
+
cz-cli task save-config shell_etl --vcluster DEFAULT --retry-count 1 --profile <your-profile>
|
|
142
142
|
cz-cli task save-cron shell_etl --cron "0 1 * * *" --profile <your-profile>
|
|
143
143
|
|
|
144
144
|
# Publish and run a one-time execution to verify
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Supported Cloud Platforms and Regions
|
|
2
|
+
|
|
3
|
+
## Region Support
|
|
4
|
+
|
|
5
|
+
Singdata Lakehouse, as a SaaS-based data management and analysis service, fully leverages cloud infrastructure to provide users with efficient and convenient service experiences. We are committed to meeting the data connectivity and integration needs of customers and business systems across different cloud service providers and regions.
|
|
6
|
+
|
|
7
|
+
Currently, Singdata Lakehouse provides services in the following cloud service providers and regions, with plans to expand further:
|
|
8
|
+
|
|
9
|
+
| Cloud Provider | Region | Region Code |
|
|
10
|
+
| -------------- | ------------------------ | ----------------------- |
|
|
11
|
+
| Alibaba Cloud | Singapore (Singapore) | ap-southeast-1-alicloud |
|
|
12
|
+
| AWS | Asia Pacific (Singapore) | ap-southeast-1-aws |
|
|
13
|
+
|
|
14
|
+
## Service Domains
|
|
15
|
+
|
|
16
|
+
When you register a Singdata Lakehouse account, the system automatically assigns a unique account name. When managing your account, you need to use this account name to log in to the Account Center. In the Account Center, administrators can create and open Lakehouse service instances for you in specified cloud service providers and regions. Please note that Lakehouse service instance names are globally unique.
|
|
17
|
+
|
|
18
|
+
| Service | Sub-service | Domain |
|
|
19
|
+
| ------------------- | -------------------------- | -------------------------------------------------------------------- |
|
|
20
|
+
| Account Console | Account Management Center | \<account\_name>.\<region\_id>.accounts.singdata.com |
|
|
21
|
+
| Product Web Console | Lakehouse Instance Console | \<instance\_name>.\<region\_id>.app.singdata.com |
|
|
22
|
+
| | Lakehouse Workspace List | \<instance\_name>.app.lakehouse.singdata..com/workspace |
|
|
23
|
+
| Lakehouse JDBC URL | | jdbc\:clickzetta://\<instance\_name>.\<region\_id>.api.singdata.com/ |
|
|
24
|
+
|
|
25
|
+
JDBC Domain & Service Endpoint Detailed List
|
|
26
|
+
|
|
27
|
+
| Cloud Provider | Region | region\_id | JDBC Domain | Endpoint |
|
|
28
|
+
| -------------- | ------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- |
|
|
29
|
+
| Alibaba Cloud | Singapore (Singapore) | cn-shanghai-alicloud | jdbc\:clickzetta://\<instance\_name>.ap-southeast-1-alicloud.api.singdata.com/\<workspace\_name>?username=\<user\_name>\&password=\&schema=public\&virtualCluster=DEFAULT | ap-southeast-1-alicloud.app.singdata.com |
|
|
30
|
+
| AWS | Asia Pacific (Singapore) | cn-north-1-aws | jdbc\:clickzetta://\<instance\_name>.ap-southeast-1-aws.api.singdata.com/\<workspace\_name>?username=\<user\_name>\&password=\&schema=public\&virtualCluster=DEFAULT | cn-north-1-aws.api.clickzetta.com |
|
|
31
|
+
|
|
32
|
+
Note: For creating and obtaining the account name \<account\_name> and instance name \<instance\_name>, please refer to [Getting Started](logging-in.md).
|
|
@@ -2,33 +2,39 @@
|
|
|
2
2
|
|
|
3
3
|
## Feature Overview
|
|
4
4
|
|
|
5
|
-
The Table Rendering feature allows users to describe table styles using natural language. DataGPT will automatically generate data tables
|
|
5
|
+
The Table Rendering feature allows users to describe table styles using natural language. The Data Analytics Agent (DataGPT) will automatically generate data tables with complex layouts such as grouping, aggregation, color marking, fixed headers, and dual-layer headers. No manual formatting is needed — just describe the table style you want as if chatting.
|
|
6
6
|
|
|
7
7
|
## Usage
|
|
8
8
|
|
|
9
|
-
In the AI dialog, describe the table content and style you need, and the Agent will automatically generate the corresponding table.
|
|
9
|
+
In the AI dialog box, describe the table content and style you need, and the Agent will automatically generate the corresponding table.
|
|
10
10
|
|
|
11
11
|
### Example
|
|
12
12
|
|
|
13
13
|
"Help me analyze the sales performance of different brands. Group by brand, differentiate between B-end and C-end, include full-year summary data and monthly information, and add commentary. Numeric columns should support sorting, with sorting limited to within each brand only."
|
|
14
14
|
|
|
15
|
-

|
|
16
16
|
|
|
17
17
|
**Output Report**:
|
|
18
|
-

|
|
19
19
|
|
|
20
20
|
You can optimize table details by continuing the conversation, for example:
|
|
21
21
|
|
|
22
|
-
1.
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
Sorting:
|
|
26
|
-

|
|
22
|
+
1. Use a compact table style with minimum row height, reduced padding, no text wrapping, and a tight layout similar to Excel.
|
|
23
|
+
|
|
24
|
+

|
|
27
25
|
|
|
28
26
|
 **Notes**
|
|
29
27
|
|
|
30
|
-
1
|
|
28
|
+
1\. The table style is automatically generated by the Agent based on natural language descriptions. The more specific the description, the better the generated result matches expectations.
|
|
29
|
+
|
|
30
|
+
2\. For complex layout requirements, it is recommended to fully describe all style requirements (grouping method, aggregation dimensions, color rules, etc.) in a single query.
|
|
31
|
+
|
|
32
|
+
3\. To adjust a generated table style, you can modify it by appending a follow-up description (e.g., "Change the background color to blue").
|
|
33
|
+
|
|
34
|
+
## Related Documentation
|
|
31
35
|
|
|
32
|
-
|
|
36
|
+
* [Chart Auto-Refresh Configuration](chart-auto-refresh-guide.md) — Set up auto-refresh for generated charts
|
|
37
|
+
* [Dashboard Version Management](dashboard-version-management-guide.md) — Manage dashboard versions across multiple revisions
|
|
38
|
+
* [Conversational Data Analytics (Analytics Agent)](datagpt_introduction.md) — Return to the feature overview
|
|
33
39
|
|
|
34
|
-
|
|
40
|
+
^
|
|
@@ -8,15 +8,15 @@ Click "Development" in the left navigation menu to enter the main interface.
|
|
|
8
8
|
|
|
9
9
|
## Interface Overview
|
|
10
10
|
|
|
11
|
-

|
|
12
12
|
|
|
13
|
-
| No. | Function
|
|
14
|
-
|
|
|
15
|
-
| 1
|
|
16
|
-
| 2
|
|
17
|
-
| 3
|
|
18
|
-
| 4
|
|
19
|
-
| 5
|
|
13
|
+
| No. | Function |
|
|
14
|
+
| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
15
|
+
| 1 | Function switch |
|
|
16
|
+
| 2 | Switch workspace: This section displays the name of the current data development project space and the region you are in. You can click the dropdown icon to switch to other project spaces. |
|
|
17
|
+
| 3 | Development directory tree: The directory tree is used to manage task code in an orderly manner. You can create directory trees according to business needs to manage tasks by category and hierarchy. |
|
|
18
|
+
| 4 | SQL editing area: The main operation interface for development. Different types of editing areas have different operation interfaces. |
|
|
19
|
+
| 5 | Run results: Presents the results after running operations in the SQL editing area. |
|
|
20
20
|
|
|
21
21
|
## Directory Tree
|
|
22
22
|
|
|
@@ -61,20 +61,20 @@ Supported operations on the directory tree:
|
|
|
61
61
|
|
|
62
62
|

|
|
63
63
|
|
|
64
|
-
| Status Legend
|
|
65
|
-
| --------------------------------------------------------------------------------- |
|
|
66
|
-
|  | After task submission, the saved version differs from the submitted version.
|
|
64
|
+
| Status Legend | Description |
|
|
65
|
+
| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
|
|
66
|
+
|  | After task submission, the saved version differs from the submitted version. |
|
|
67
67
|
|  | After task submission, the server-side saved version matches the submitted version. |
|
|
68
|
-
|  | After task submission, click the offline operation.
|
|
69
|
-
|  | Other status types.
|
|
68
|
+
|  | After task submission, click the offline operation. |
|
|
69
|
+
|  | Other status types. |
|
|
70
70
|
|
|
71
|
-
| Icon Legend
|
|
72
|
-
| --------------------------------------------------------------------------------- |
|
|
73
|
-
|  | Real-time sync
|
|
74
|
-
|  | Offline sync
|
|
75
|
-
|  | SQL script
|
|
76
|
-
|  | Shell script
|
|
77
|
-
|  | Python script
|
|
71
|
+
| Icon Legend | Description |
|
|
72
|
+
| --------------------------------------------------------------------------------- | -------------- |
|
|
73
|
+
|  | Real-time sync |
|
|
74
|
+
|  | Offline sync |
|
|
75
|
+
|  | SQL script |
|
|
76
|
+
|  | Shell script |
|
|
77
|
+
|  | Python script |
|
|
78
78
|
|
|
79
79
|
## Task Development Process
|
|
80
80
|
|
|
@@ -86,7 +86,7 @@ Currently, two types of task nodes are supported: data integration and data deve
|
|
|
86
86
|
|
|
87
87
|
Click "New" and select the specific development type. A popup will appear. Enter the task name, select the specific directory level where the task should be saved, and then enter the development interface.
|
|
88
88
|
|
|
89
|
-
|
|
89
|
+
^
|
|
90
90
|
|
|
91
91
|
### Operation Bar Description
|
|
92
92
|
|
|
@@ -102,45 +102,43 @@ After clicking the task directory tree on the left or creating a new task node,
|
|
|
102
102
|
|
|
103
103
|
Tab color status mapping:
|
|
104
104
|
|
|
105
|
-
| Status Icon | Description
|
|
106
|
-
|
|
|
107
|
-
| Green
|
|
108
|
-
| Red
|
|
109
|
-
| Blue
|
|
110
|
-
| Gray
|
|
105
|
+
| Status Icon | Description |
|
|
106
|
+
| ----------- | -------------- |
|
|
107
|
+
| Green | Run successful |
|
|
108
|
+
| Red | Run failed |
|
|
109
|
+
| Blue | Running |
|
|
110
|
+
| Gray | No run |
|
|
111
111
|
|
|
112
112
|
#### SQL Function Area
|
|
113
113
|
|
|
114
114
|

|
|
115
115
|
|
|
116
|
-
| No. | Function Name
|
|
117
|
-
|
|
|
118
|
-
| 1
|
|
119
|
-
| 2
|
|
120
|
-
| 3
|
|
121
|
-
| 4
|
|
122
|
-
| 5
|
|
123
|
-
| 6
|
|
124
|
-
| 7
|
|
125
|
-
| 8
|
|
126
|
-
| 9
|
|
127
|
-
| 10
|
|
116
|
+
| No. | Function Name | Description |
|
|
117
|
+
| --- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
118
|
+
| 1 | Save | Save the task, including the current node code and related configurations. |
|
|
119
|
+
| 2 | Format | Format the written code to make the syntax structure look clean and concise. |
|
|
120
|
+
| 3 | Parameters | Not yet online. |
|
|
121
|
+
| 4 | Versions | Click Versions to view the submitted and saved versions of the current task. Supports code viewing and code rollback between versions. |
|
|
122
|
+
| 5 | Scheduling | Click to open a sidebar panel for scheduling configuration. See Scheduling Settings for details. |
|
|
123
|
+
| 6 | Submit | The "Submit" function is only needed for tasks that require scheduled dispatch. After "Submit", the task is published to the Operations Center and runs according to the configured schedule. Before submission, scheduling configuration is required. See Scheduling Settings for details. |
|
|
124
|
+
| 7 | Operations | Click to enter the "Operations" center. |
|
|
125
|
+
| 8 | Task flow tips | Hover over the task flow tips to display the online flowchart for the development task. |
|
|
126
|
+
| 9 | Cluster filter | The "default" shown in the figure indicates the Virtual Cluster used for task execution. Click to switch to another Virtual Cluster. To add a new Virtual Cluster, go to "Compute". |
|
|
127
|
+
| 10 | Run/Stop | Run/Stop the current node's code. When running SQL code, you can select and run a portion of the code. |
|
|
128
128
|
|
|
129
129
|
#### SQL Editing Area
|
|
130
130
|
|
|
131
|
-

|
|
132
|
-
|
|
133
131
|
* Schema switching: Click the dropdown to switch schemas in the current workspace. The default is "public".
|
|
134
132
|
* Shortcuts: Various capabilities supported in the Studio development editing area. See Others: Common Shortcut Operations for details.
|
|
135
133
|
* SQL editing area: The SQL editor in Studio provides the following features to improve data development and analysis efficiency.
|
|
136
134
|
|
|
137
|
-
| Feature
|
|
138
|
-
|
|
|
139
|
-
| Code folding
|
|
140
|
-
| Real-time syntax error prompts | Prompt users of syntax errors found during code writing to help avoid mistakes.
|
|
141
|
-
| Syntax highlighting
|
|
142
|
-
| Intelligent completion
|
|
143
|
-
| Partial code execution
|
|
135
|
+
| Feature | Description |
|
|
136
|
+
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
137
|
+
| Code folding | Collapse code blocks to reduce reading distractions. |
|
|
138
|
+
| Real-time syntax error prompts | Prompt users of syntax errors found during code writing to help avoid mistakes. |
|
|
139
|
+
| Syntax highlighting | Use different colors or fonts to highlight keywords and syntax structures in the editor or IDE to enhance readability. |
|
|
140
|
+
| Intelligent completion | Automatically complete keywords, function names, variable names, etc., in code based on context and known information to improve coding efficiency and precision. |
|
|
141
|
+
| Partial code execution | Run only a portion of the code instead of the entire program for quick testing of small code segments or debugging. |
|
|
144
142
|
|
|
145
143
|
### Scheduling Settings
|
|
146
144
|
|
|
@@ -148,46 +146,44 @@ After the task has been tested and everything works correctly, if you want to ru
|
|
|
148
146
|
|
|
149
147
|
#### Basic Information
|
|
150
148
|
|
|
151
|
-
| Parameter
|
|
152
|
-
|
|
|
153
|
-
| Owner
|
|
154
|
-
| Description
|
|
155
|
-
| Run attributes
|
|
156
|
-
| Cluster
|
|
157
|
-
| Schema
|
|
158
|
-
| Task priority
|
|
159
|
-
| Parameter configuration
|
|
149
|
+
| Parameter | Description |
|
|
150
|
+
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
151
|
+
| Owner | Required. Only one member allowed, defaults to the task creator. Can be changed to another workspace member as needed. |
|
|
152
|
+
| Description | Optional. Provides a detailed description of the task for future reference and management. |
|
|
153
|
+
| Run attributes | Required. Normal scheduling: Runs according to the user's scheduling rules. Dry-run scheduling: When a task's logic does not need to run temporarily but you don't want to disrupt the data pipeline, set it to dry-run; the task will be marked as successful. Paused: After setting the task to "Paused" and publishing to the Operations Center, the task status is "Paused" and no task instances are generated, but backfill operations are still possible. |
|
|
154
|
+
| Cluster | Required. Defines the scheduling resource group used when the task is published to production. |
|
|
155
|
+
| Schema | Required. Defines the prefix schema used when the task runs in production. |
|
|
156
|
+
| Task priority | Optional. Sets the scheduling priority for Lakehouse SQL tasks, supporting 10 levels (0-9), where higher numbers mean higher priority. |
|
|
157
|
+
| Parameter configuration | Click "Add Parameter" to add a new parameter. Click "Load Parameters in Code" to automatically load parameters already used in the code. In code, parameters are referenced as: '${bizdate}'. Note: more system built-in parameters will be supported in future versions. |
|
|
160
158
|
|
|
161
159
|
#### Scheduling Time
|
|
162
160
|
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
|
166
|
-
|
|
|
167
|
-
| Scheduling
|
|
168
|
-
|
|
|
169
|
-
|
|
|
170
|
-
|
|
|
171
|
-
|
|
|
172
|
-
|
|
|
173
|
-
|
|
|
174
|
-
| Expiry time | The date from which the task expires. |
|
|
175
|
-
| Preview scheduling time | Click to preview the actual run times after configuration. |
|
|
161
|
+
| Parameter | Description |
|
|
162
|
+
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
163
|
+
| Scheduling cycle | Daily: Runs every day. Monthly specified day: User selects specific days each month. Weekly specified day: User selects specific days each week. |
|
|
164
|
+
| Scheduling frequency | Execute once: Only the start scheduling time needs to be configured. Execute multiple times: Also requires scheduling interval, start time, end time, and exclusion time settings. |
|
|
165
|
+
| Scheduling interval | When executing multiple times, users can customize the interval between each execution, from 1 minute to 12 hours. |
|
|
166
|
+
| Start scheduling time | The time when daily instances begin executing. |
|
|
167
|
+
| End scheduling time | The time when daily instances stop executing. |
|
|
168
|
+
| Exclusion time settings | No instances will be generated during the specified exclusion time ranges. |
|
|
169
|
+
| Effective time | The date from which the task takes effect. |
|
|
170
|
+
| Expiry time | The date from which the task expires. |
|
|
171
|
+
| Preview scheduling time | Click to preview the actual run times after configuration. |
|
|
176
172
|
|
|
177
173
|
#### Instance Information
|
|
178
174
|
|
|
179
175
|
Studio supports two instance generation modes: **Effective next day** and **Effective after publishing**.
|
|
180
176
|
|
|
181
|
-
| Parameter
|
|
182
|
-
|
|
|
183
|
-
| Instance generation mode
|
|
184
|
-
| Instance retry on error
|
|
185
|
-
| Auto retry count
|
|
186
|
-
| Instance timeout duration
|
|
187
|
-
| Preview scheduling time
|
|
188
|
-
| Parameter configuration
|
|
189
|
-
| Scheduling wait duration
|
|
190
|
-
| Delayed run skip duration | Based on the difference between the task instance's actual start time and the configured scheduled time. If the difference exceeds this setting, the instance is skipped as a dry run and marked as successful.
|
|
177
|
+
| Parameter | Description |
|
|
178
|
+
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
179
|
+
| Instance generation mode | Effective after publishing: Instances are generated immediately after publishing and run according to the configured schedule. Effective next day: Instances run on the following day. |
|
|
180
|
+
| Instance retry on error | Defines whether a task instance can be retried, considering data idempotency. Set as needed. Can retry after success or failure. Cannot retry after success, can retry after failure. Cannot retry after success or failure. |
|
|
181
|
+
| Auto retry count | Custom number of retries. |
|
|
182
|
+
| Instance timeout duration | Not enabled / Custom. If a task instance runs longer than the set duration, it will be marked as failed. Changes only apply to new instances after the task is submitted. |
|
|
183
|
+
| Preview scheduling time | Click to preview the actual run times after configuration. |
|
|
184
|
+
| Parameter configuration | Click "Add Parameter" to add a new parameter. Click "Load Parameters in Code" to automatically load parameters already used in the code. In code, parameters are referenced as: '${bizdate}'. Note: More system built-in parameters will be supported in future versions. |
|
|
185
|
+
| Scheduling wait duration | If a task instance reaches its scheduled time and waits longer than this setting (still waiting for upstream tasks to finish), the system will mark it as failed to prevent scheduling backlogs. |
|
|
186
|
+
| Delayed run skip duration | Based on the difference between the task instance's actual start time and the configured scheduled time. If the difference exceeds this setting, the instance is skipped as a dry run and marked as successful. |
|
|
191
187
|
|
|
192
188
|
#### Scheduling Dependencies
|
|
193
189
|
|
|
@@ -195,7 +191,7 @@ Complex production tasks typically have upstream and downstream dependencies. Fo
|
|
|
195
191
|
|
|
196
192
|
Dependency strategy: Currently supports default strategy, forward dependency, and forward nearest dependency. For specific instance generation methods for each strategy, see [Task Scheduling Dependencies](task_scheduling_dependency.md).
|
|
197
193
|
|
|
198
|
-

|
|
199
195
|
|
|
200
196
|
#### Task Output
|
|
201
197
|
|
|
@@ -205,7 +201,7 @@ Method 1: Automatically generate output table names through code auto-parsing me
|
|
|
205
201
|
|
|
206
202
|
Method 2: Manually add by searching table names.
|
|
207
203
|
|
|
208
|
-
|
|
204
|
+
^
|
|
209
205
|
|
|
210
206
|
For more information about task scheduling, refer to: [Task Scheduling Dependencies](task_scheduling_dependency.md), [Task Scheduling and Instance Execution](f6fc6447ee.md).
|
|
211
207
|
|
|
@@ -223,7 +219,7 @@ Click to jump to the "Operations" center to view task operations.
|
|
|
223
219
|
|
|
224
220
|
Run history provides up to 20 recent run result records for the current task tab within 7 days.
|
|
225
221
|
|
|
226
|
-
|
|
222
|
+
^
|
|
227
223
|
|
|
228
224
|
### Run Results
|
|
229
225
|
|
|
@@ -236,13 +232,13 @@ Use the expand and collapse buttons to adjust the log display area.
|
|
|
236
232
|
|
|
237
233
|
Use the refresh button to force refresh the logs.
|
|
238
234
|
|
|
239
|
-
|
|
235
|
+
^
|
|
240
236
|
|
|
241
237
|
#### Data
|
|
242
238
|
|
|
243
239
|
After the task completes, results (such as from SELECT statements) are displayed in the "Data" tab:
|
|
244
240
|
|
|
245
|
-
|
|
241
|
+
^
|
|
246
242
|
|
|
247
243
|
## Others: Common Shortcut Operations
|
|
248
244
|
|
|
@@ -269,9 +265,11 @@ The following shortcuts are supported in the code editor to improve editing effi
|
|
|
269
265
|
|
|
270
266
|
## Related Documents
|
|
271
267
|
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
268
|
+
* [Task Parameters](task_param.md) — Use dynamic parameters (dates, timestamps, etc.) in SQL tasks
|
|
269
|
+
* [Task Parameter Syntax Reference](task_param_reference.md) — Complete syntax for system built-in parameters, time expressions, and time functions
|
|
270
|
+
* [Task Parameter Scenario Examples](task_param_examples.md) — Complete business scenarios such as daily, monthly, and weekly reports
|
|
271
|
+
* [Workflow (Composite Task)](composite_task.md) — Orchestrate multiple SQL tasks into a DAG
|
|
272
|
+
* [Task Groups](task_group.md) — Share parameters across tasks and manage them uniformly
|
|
273
|
+
* [Python Tasks](python-task.md) — Use parameters in Python scripts
|
|
274
|
+
|
|
275
|
+
^
|