@clickzetta/cz-cli-darwin-arm64 0.5.16 → 0.5.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/cz-cli +0 -0
- package/bin/skills/lakehouse-doc-en/SKILL.md +6 -11
- package/bin/skills/lakehouse-doc-en/references/AIGateway.md +58 -13
- package/bin/skills/lakehouse-doc-en/references/Computation.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/DataSource_Amazon_DocumentDB.md +3 -1
- package/bin/skills/lakehouse-doc-en/references/Foreach.md +14 -14
- package/bin/skills/lakehouse-doc-en/references/JDBC-Driver.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/LakehouseAI-overview.md +21 -8
- package/bin/skills/lakehouse-doc-en/references/LakehouseDataGPT-tour.md +4 -9
- package/bin/skills/lakehouse-doc-en/references/LakehouseStudio-tour.md +14 -19
- package/bin/skills/lakehouse-doc-en/references/Lakehouse_Zilliz_MakeDataReadyforBIandAI.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/Logstash.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/Migrate_Spark_DataEngineeringBestPractices_Project_to_Lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/Notebook.md +17 -17
- package/bin/skills/lakehouse-doc-en/references/RemoteFunction-as-udf.md +14 -14
- package/bin/skills/lakehouse-doc-en/references/SQL_External_Catalog_Guide.md +1 -9
- package/bin/skills/lakehouse-doc-en/references/SUMMARY.md +59 -29
- package/bin/skills/lakehouse-doc-en/references/WINDOWFUNCTION.md +99 -57
- package/bin/skills/lakehouse-doc-en/references/Zettapark_Data_Engineering_Demo.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/access-control-configuration.md +1 -8
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-2-5-1.0.md +16 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-3-29-1.0.2.md +14 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-3-8-1.0.1.md +16 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-4-28-1.1.md +29 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-12-1.1.1.md +18 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-15-1.2.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-21-1.3.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-5-28-1.4.md +10 -0
- package/bin/skills/lakehouse-doc-en/references/aigw-2026-6-3-1.5.md +9 -0
- package/bin/skills/lakehouse-doc-en/references/alicloud-arn-externalid.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/answer-accuracy-improve.md +120 -103
- package/bin/skills/lakehouse-doc-en/references/application-list.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/approval-list.md +16 -17
- package/bin/skills/lakehouse-doc-en/references/batch-load-parquet-file-into-lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/batch_sync.md +9 -9
- package/bin/skills/lakehouse-doc-en/references/batch_sync_Sop.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/batchloadparquetfileintoLakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/bulkloadv1-python-sdk.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/chart-auto-refresh-guide.md +12 -6
- package/bin/skills/lakehouse-doc-en/references/clickzetta-sample-data.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/code_approval.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/composite_task.md +31 -42
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_environment_and_data_generate.md +6 -9
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_javasdk_bulkload_realtime.md +4 -10
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_kafka_realtime_sync.md +1 -10
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_local_file_into_table_by_studio.md +0 -6
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_batchload_public_network.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_python_node.md +2 -7
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_realtime_cdc_public_network.md +13 -18
- package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_studio_sql_insert.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/concepts.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/config-datasource.md +5 -7
- package/bin/skills/lakehouse-doc-en/references/connect-with-cli.md +116 -72
- package/bin/skills/lakehouse-doc-en/references/connect-with-cz-cli.md +151 -0
- package/bin/skills/lakehouse-doc-en/references/continue-job.md +9 -17
- package/bin/skills/lakehouse-doc-en/references/create-api-connection.md +315 -286
- package/bin/skills/lakehouse-doc-en/references/create-catalog-connection.md +1 -0
- package/bin/skills/lakehouse-doc-en/references/create-dynamic-table.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/create-external-catalog.md +85 -22
- package/bin/skills/lakehouse-doc-en/references/create-table-ddl.md +45 -0
- package/bin/skills/lakehouse-doc-en/references/creating_alicloud_privatelinkendpoint.md +4 -6
- package/bin/skills/lakehouse-doc-en/references/creating_alicloud_privatelinkservice.md +4 -7
- package/bin/skills/lakehouse-doc-en/references/creating_tencentcloud_privatelinkendpoint.md +2 -7
- package/bin/skills/lakehouse-doc-en/references/creating_tencentcloud_privatelinkservice.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/cz-cli-agent.md +15 -10
- package/bin/skills/lakehouse-doc-en/references/cz-cli-datasource.md +0 -8
- package/bin/skills/lakehouse-doc-en/references/cz-cli-sql.md +2 -45
- package/bin/skills/lakehouse-doc-en/references/cz-cli.md +53 -42
- package/bin/skills/lakehouse-doc-en/references/dashboard-version-management-guide.md +12 -4
- package/bin/skills/lakehouse-doc-en/references/data-integration-intro.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/data-integration.md +29 -27
- package/bin/skills/lakehouse-doc-en/references/data-load-summary.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/data-quality.md +25 -25
- package/bin/skills/lakehouse-doc-en/references/data-sharing.md +31 -54
- package/bin/skills/lakehouse-doc-en/references/data-sources.md +45 -45
- package/bin/skills/lakehouse-doc-en/references/data_catalog.md +23 -25
- package/bin/skills/lakehouse-doc-en/references/data_privacy.md +5 -2
- package/bin/skills/lakehouse-doc-en/references/data_sharing_between_accounts_guide.md +0 -4
- package/bin/skills/lakehouse-doc-en/references/data_visualization.md +4 -15
- package/bin/skills/lakehouse-doc-en/references/dataagent.md +39 -7
- package/bin/skills/lakehouse-doc-en/references/databricks-delta-to-lakehouse-migration.md +168 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-dlt-to-lakehouse-migration.md +331 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-external-catalog-practice.md +367 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-jobs-to-studio-migration.md +199 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-notebook-to-studio-migration.md +350 -0
- package/bin/skills/lakehouse-doc-en/references/databricks-uc-governance-to-lakehouse-migration.md +327 -0
- package/bin/skills/lakehouse-doc-en/references/datagpt-model-config.md +34 -0
- package/bin/skills/lakehouse-doc-en/references/datagpt_data_source.md +50 -37
- package/bin/skills/lakehouse-doc-en/references/datagpt_introduction.md +55 -79
- package/bin/skills/lakehouse-doc-en/references/datagpt_quickstart.md +50 -64
- package/bin/skills/lakehouse-doc-en/references/datalake-acceleration.md +75 -2
- package/bin/skills/lakehouse-doc-en/references/dbt-databricks-to-clickzetta-migration.md +242 -0
- package/bin/skills/lakehouse-doc-en/references/dynamic-mask.md +30 -30
- package/bin/skills/lakehouse-doc-en/references/dynamic-table-bestpractice.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/dynamic-table-introduce.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/dynamic_table_summary.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/eco_integration/streamlit.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/eco_integration/superset.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/ecosystem-all.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/ecosystem.md +145 -0
- package/bin/skills/lakehouse-doc-en/references/external-catalog-summary.md +33 -38
- package/bin/skills/lakehouse-doc-en/references/external-function-combo-practice.md +466 -0
- package/bin/skills/lakehouse-doc-en/references/f6fc6447ee.md +7 -9
- package/bin/skills/lakehouse-doc-en/references/federation-query.md +56 -6
- package/bin/skills/lakehouse-doc-en/references/finebi-mysql.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/get-started-with-sample-data.md +10 -11
- package/bin/skills/lakehouse-doc-en/references/gitfolder.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/grant-privileges.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/iceberg-rest-catalog-databricks.md +166 -0
- package/bin/skills/lakehouse-doc-en/references/ide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/if_else_task.md +59 -57
- package/bin/skills/lakehouse-doc-en/references/input_output.md +10 -7
- package/bin/skills/lakehouse-doc-en/references/jobprofile-bestpractices.md +60 -64
- package/bin/skills/lakehouse-doc-en/references/kafka-connection.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/key-concepts.md +146 -117
- package/bin/skills/lakehouse-doc-en/references/lakehouse-ai-gateway-cz-cli.md +317 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-ai-sql-analysis.md +345 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-dqc-guide.md +300 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-medallion-sql-dt-guide.md +543 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-multi-cloud-acceleration.md +274 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-multimodal-ai-pipeline.md +198 -0
- package/bin/skills/lakehouse-doc-en/references/lakehouse-quick-experience_guide.md +49 -52
- package/bin/skills/lakehouse-doc-en/references/lakehouse-volume-pipe-acceleration-guide.md +380 -0
- package/bin/skills/lakehouse-doc-en/references/langchain-plug-installation.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/management.md +4 -9
- package/bin/skills/lakehouse-doc-en/references/medallion-lakehouse-from-scratch.md +2 -1
- package/bin/skills/lakehouse-doc-en/references/metrics_answer_build.md +58 -21
- package/bin/skills/lakehouse-doc-en/references/migrate-spark-data-engineering-best-practices-to-lakehouse.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/mindsdb.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/monitoring_and_alerting.md +65 -60
- package/bin/skills/lakehouse-doc-en/references/monitoring_item_specification.md +33 -33
- package/bin/skills/lakehouse-doc-en/references/multitable_batch_sync.md +16 -16
- package/bin/skills/lakehouse-doc-en/references/multitable_realtime_sync.md +65 -72
- package/bin/skills/lakehouse-doc-en/references/multitable_realtime_sync_sop.md +54 -52
- package/bin/skills/lakehouse-doc-en/references/navicat-mysql.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/om-dynamic-table.md +71 -66
- package/bin/skills/lakehouse-doc-en/references/om-vcluster.md +2 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-create-session.md +79 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-generate-auth-token.md +63 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-overview.md +96 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-quick-start.md +286 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-response-guide.md +264 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-safe-question-poll.md +201 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-text2insight-query.md +99 -0
- package/bin/skills/lakehouse-doc-en/references/open-api-text2insight-stop.md +74 -0
- package/bin/skills/lakehouse-doc-en/references/overview.md +6 -7
- package/bin/skills/lakehouse-doc-en/references/permission-application.md +5 -5
- package/bin/skills/lakehouse-doc-en/references/pipe-introduction.md +1 -0
- package/bin/skills/lakehouse-doc-en/references/pipe-kafka-table-stream.md +72 -70
- package/bin/skills/lakehouse-doc-en/references/pipe-kafka.md +105 -110
- package/bin/skills/lakehouse-doc-en/references/pipe-overview.md +40 -40
- package/bin/skills/lakehouse-doc-en/references/pipe-storage-object.md +43 -48
- package/bin/skills/lakehouse-doc-en/references/pipe-summary.md +14 -4
- package/bin/skills/lakehouse-doc-en/references/pipe-syntax.md +58 -151
- package/bin/skills/lakehouse-doc-en/references/practice_python_task.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/pricing-ai-gateway.md +181 -0
- package/bin/skills/lakehouse-doc-en/references/pricing-lakehouse.md +316 -0
- package/bin/skills/lakehouse-doc-en/references/pricing.md +44 -288
- package/bin/skills/lakehouse-doc-en/references/private-link-general.md +0 -2
- package/bin/skills/lakehouse-doc-en/references/pyspark-to-zettapark-migration-f1.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python-igs.md +7 -3
- package/bin/skills/lakehouse-doc-en/references/python-sample-put-github-rt-events.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python-task.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector_advanced.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/python_reference/connector_examples.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/python_sdk_guide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/python_shell_datasource.md +11 -9
- package/bin/skills/lakehouse-doc-en/references/quick_start_batch_sync_data.md +9 -18
- package/bin/skills/lakehouse-doc-en/references/quick_start_bi_analysis.md +8 -25
- package/bin/skills/lakehouse-doc-en/references/quick_start_create_workspace.md +4 -6
- package/bin/skills/lakehouse-doc-en/references/quick_start_data_quality.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quick_start_etl.md +16 -20
- package/bin/skills/lakehouse-doc-en/references/quick_start_monitoring_and_alerting.md +10 -18
- package/bin/skills/lakehouse-doc-en/references/quick_start_sql_query.md +7 -10
- package/bin/skills/lakehouse-doc-en/references/quick_start_upload_data.md +5 -7
- package/bin/skills/lakehouse-doc-en/references/quick_start_user_management.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quick_start_workspace.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/quick_start_workspace_user.md +8 -8
- package/bin/skills/lakehouse-doc-en/references/quickstart.md +69 -56
- package/bin/skills/lakehouse-doc-en/references/quickstart_datashare_between_companies.md +0 -5
- package/bin/skills/lakehouse-doc-en/references/quickstart_envirment_for_team.md +0 -24
- package/bin/skills/lakehouse-doc-en/references/realtime-pipeline-selection-guide.md +1 -2
- package/bin/skills/lakehouse-doc-en/references/realtime-sales-dashboard-with-dynamic-table.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/realtime_sync.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/release-note-2026-05-19.md +5 -3
- package/bin/skills/lakehouse-doc-en/references/revoke-privileges.md +3 -1
- package/bin/skills/lakehouse-doc-en/references/roles.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/row-filter.md +165 -0
- package/bin/skills/lakehouse-doc-en/references/row_level_permission.md +30 -19
- package/bin/skills/lakehouse-doc-en/references/scheduled_task.md +28 -21
- package/bin/skills/lakehouse-doc-en/references/security_overview.md +99 -21
- package/bin/skills/lakehouse-doc-en/references/set-command.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/setup.md +13 -15
- package/bin/skills/lakehouse-doc-en/references/show-grants.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/snowflake-dynamic-tables-to-lakehouse.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/spark-connector-summary.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/sql_functions/context_functions/current_vcluster.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/sso-configuration.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/streaming_pipeline_with_dynamic_table.md +0 -1
- package/bin/skills/lakehouse-doc-en/references/studio-incremental-sync-practice.md +27 -23
- package/bin/skills/lakehouse-doc-en/references/studio-shell-task.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/supported-cloud-platforms.md +32 -0
- package/bin/skills/lakehouse-doc-en/references/table_rendering.md +18 -12
- package/bin/skills/lakehouse-doc-en/references/task-develop.md +89 -91
- package/bin/skills/lakehouse-doc-en/references/task_development.md +19 -17
- package/bin/skills/lakehouse-doc-en/references/task_group.md +16 -14
- package/bin/skills/lakehouse-doc-en/references/task_instance.md +21 -21
- package/bin/skills/lakehouse-doc-en/references/task_param.md +38 -35
- package/bin/skills/lakehouse-doc-en/references/task_param_reference.md +81 -79
- package/bin/skills/lakehouse-doc-en/references/task_scheduling_dependency.md +20 -21
- package/bin/skills/lakehouse-doc-en/references/tencentcloud_arn_and_externalid.md +1 -5
- package/bin/skills/lakehouse-doc-en/references/trial-account-quotas-and-limits.md +1 -3
- package/bin/skills/lakehouse-doc-en/references/tutorial_connect_to_lakehouse.md +69 -0
- package/bin/skills/lakehouse-doc-en/references/tutorials.md +4 -1
- package/bin/skills/lakehouse-doc-en/references/unique-key.md +167 -0
- package/bin/skills/lakehouse-doc-en/references/usageandbillingview.md +138 -0
- package/bin/skills/lakehouse-doc-en/references/use-dbt-dev.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/use-java-sdk-realtime-uploaddata.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/use-java-sdk-upload-data-local.md +3 -3
- package/bin/skills/lakehouse-doc-en/references/use-models.md +128 -0
- package/bin/skills/lakehouse-doc-en/references/use-mysql-client.md +81 -81
- package/bin/skills/lakehouse-doc-en/references/use-python-sdk-upload-data.md +10 -12
- package/bin/skills/lakehouse-doc-en/references/user-identification.md +2 -3
- package/bin/skills/lakehouse-doc-en/references/user_permission_grand_guide.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/using-udf-in-dynamic-table.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/vc_cache.md +18 -22
- package/bin/skills/lakehouse-doc-en/references/vcluster_size_description.md +33 -31
- package/bin/skills/lakehouse-doc-en/references/virtual-cluster.md +43 -45
- package/bin/skills/lakehouse-doc-en/references/web-job-history.md +94 -108
- package/bin/skills/lakehouse-doc-en/references/web_search.md +16 -7
- package/bin/skills/lakehouse-doc-en/references/zettapark-data-engineering-demo.md +1 -1
- package/bin/skills/lakehouse-doc-en/references/zettapark-dataframe-guide.md +144 -70
- package/bin/skills/lakehouse-doc-en/references/zettapark-dynamic-table-guide.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-etl-guide.md +73 -33
- package/bin/skills/lakehouse-doc-en/references/zettapark-feature-engineering.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-functions-guide.md +75 -46
- package/bin/skills/lakehouse-doc-en/references/zettapark-quick-start.md +2 -2
- package/bin/skills/lakehouse-doc-en/references/zettapark-stream-guide.md +4 -4
- package/bin/skills/lakehouse-doc-en/references/zettapark-volume-guide.md +93 -29
- package/package.json +1 -1
- package/bin/skills/lakehouse-doc-en/references/CLAUDE.md +0 -606
- package/bin/skills/lakehouse-doc-en/references/modelprice.md +0 -155
|
@@ -7,7 +7,7 @@ Federation Query lets you query data in external data systems (Hive, Databricks,
|
|
|
7
7
|
## Supported External Data Sources
|
|
8
8
|
|
|
9
9
|
| External System | Connection Method | Typical Use Case |
|
|
10
|
-
|
|
10
|
+
|-----------------|-------------------|------------------|
|
|
11
11
|
| Apache Hive | Hive Metastore URIs | In-place acceleration of existing Hive warehouses, replacing Presto/Trino |
|
|
12
12
|
| Databricks Unity Catalog | Databricks API | Cross-platform federated analytics without moving Databricks data |
|
|
13
13
|
| Iceberg REST Catalog | Iceberg REST API | Query any data lake compatible with the Iceberg REST protocol |
|
|
@@ -17,9 +17,58 @@ Federation Query lets you query data in external data systems (Hive, Databricks,
|
|
|
17
17
|
|
|
18
18
|
## Core Concepts
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
There are three independent approaches to access external data, each suited to different scenarios:
|
|
21
21
|
|
|
22
|
-
|
|
22
|
+
### External Catalog (Recommended)
|
|
23
|
+
|
|
24
|
+
Maps an external data system (Hive/Databricks/Snowflake/Iceberg REST) as a top-level catalog; the Schemas and Tables underneath automatically correspond to the external system's structure:
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
External Catalog ← Top-level catalog, maps the external system
|
|
28
|
+
└── External Schema ← Corresponds to a Schema/Database in the external system
|
|
29
|
+
└── External Table ← Corresponds to an actual table in the external system
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Queries use three-level naming: `external_catalog.schema.table`
|
|
33
|
+
|
|
34
|
+
```sql
|
|
35
|
+
SELECT * FROM databricks_catalog.table_types_demo.orders_external;
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Supports: Hive, Databricks Unity Catalog, Iceberg REST (including Snowflake Open Catalog)
|
|
39
|
+
|
|
40
|
+
### External Schema (Standalone)
|
|
41
|
+
|
|
42
|
+
Without going through an External Catalog, directly mounts an external Hive Database into the **current Workspace's internal Catalog**, using two-level naming `schema.table` (equivalent to `<current internal catalog>.schema.table`). Direct HMS mapping — all tables under the entire Database are immediately queryable, and newly added tables are automatically visible without per-table definitions.
|
|
43
|
+
|
|
44
|
+
```sql
|
|
45
|
+
SELECT * FROM hive_orders.order_detail LIMIT 100;
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Supports: Hive (OSS/COS/GCS/HDFS). Read-only; DML is not supported.
|
|
49
|
+
|
|
50
|
+
### External Table (Standalone)
|
|
51
|
+
|
|
52
|
+
Creates a single table pointing to external storage under an ordinary Schema in the **current Workspace's internal Catalog**, using two-level naming `schema.table`. Unlike External Schema: column names and types can be customized, and renaming and modifying comments are supported.
|
|
53
|
+
|
|
54
|
+
```sql
|
|
55
|
+
CREATE EXTERNAL TABLE my_schema.delta_orders
|
|
56
|
+
LOCATION 's3://bucket/orders/'
|
|
57
|
+
USING DELTA;
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Supports: Kafka, Delta Lake, Hudi. Read-only; DML is not supported.
|
|
61
|
+
|
|
62
|
+
### Comparison of the Three Approaches
|
|
63
|
+
|
|
64
|
+
| | External Catalog | External Schema (Standalone) | External Table (Standalone) |
|
|
65
|
+
|---|---|---|---|
|
|
66
|
+
| Catalog location | Independent external Catalog | Current Workspace's internal Catalog | Current Workspace's internal Catalog |
|
|
67
|
+
| Naming | Three-level `catalog.schema.table` | Two-level `schema.table` | Two-level `schema.table` |
|
|
68
|
+
| Use case | Cross-platform federated analytics | Mount an entire Hive database into the workspace | Custom mapping for a single external table |
|
|
69
|
+
| Supported sources | Hive, Databricks, Iceberg REST, Snowflake | Hive | Kafka, Delta Lake, Hudi |
|
|
70
|
+
| Schema definition | Auto-mapped from external system | Auto-mapped from HMS | Manually define column names and types |
|
|
71
|
+
| New external tables visible | Requires re-mapping | Automatically visible | Must be created one by one |
|
|
23
72
|
|
|
24
73
|
Selection guide: [External Catalog vs External Schema](org-hierarchy.md)
|
|
25
74
|
|
|
@@ -58,10 +107,10 @@ JOIN my_lakehouse_table u ON o.user_id = u.id;
|
|
|
58
107
|
## This Section
|
|
59
108
|
|
|
60
109
|
| Page | Description |
|
|
61
|
-
|
|
110
|
+
|------|-------------|
|
|
62
111
|
| [External Object User Guide](external_object_user_guide.md) | Complete operations for creating, querying, and managing External Catalog / Schema / Table |
|
|
63
112
|
| [Query Snowflake OpenCatalog Iceberg Tables](query-snowflake-open-catalog-iceberg-table.md) | Federated queries on Snowflake-managed Iceberg data via Iceberg REST API |
|
|
64
|
-
| [Databricks
|
|
113
|
+
| [Databricks Unity Catalog Federation Query Practice](databricks-external-catalog-practice.md) | Full step-by-step setup guide with verified results and common error troubleshooting |
|
|
65
114
|
|
|
66
115
|
---
|
|
67
116
|
|
|
@@ -71,4 +120,5 @@ JOIN my_lakehouse_table u ON o.user_id = u.id;
|
|
|
71
120
|
- [External Catalog Federation Query Guide](external-catalog-concept.md) — Detailed operation examples and architecture principles
|
|
72
121
|
- [Create External Catalog](create-external-catalog.md) — DDL syntax reference
|
|
73
122
|
- [Create Hive Catalog](create-hive-catalog.md) — Hive connection configuration details
|
|
74
|
-
- [In-Place Lake Acceleration Guide](lakehouse-acceleration-guide.md) — Complete guide for replacing Spark/Hive and Presto/Trino without moving data
|
|
123
|
+
- [In-Place Lake Acceleration Guide](lakehouse-acceleration-guide.md) — Complete guide for replacing Spark/Hive and Presto/Trino without moving data
|
|
124
|
+
- [Databricks Unity Catalog Federation Query Practice](databricks-external-catalog-practice.md) — Full step-by-step setup guide with verified results
|
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
# FineBI Connection Guide
|
|
2
|
+
|
|
1
3
|
FineBI is a Business Intelligence product launched by FanRuan Software Co., Ltd. FineBI is a new-generation big data analytics BI tool designed to help business personnel fully understand and utilize their data. This article describes how to use the MySQL protocol to connect to Lakehouse.
|
|
2
4
|
|
|
3
5
|
# Prerequisites
|
|
@@ -5,14 +5,14 @@
|
|
|
5
5
|
Through this tutorial, you will learn how to use the built-in sample dataset of the Lakehouse platform to quickly use SQL for query analysis and evaluate SQL functionality and performance without preparing data in advance.
|
|
6
6
|
|
|
7
7
|
> This tutorial leverages the Lakehouse Tutorial, providing an online tutorial guide and script import function. After logging into the Lakehouse Web console, you can enter the tutorial through the Lakehouse Tutorial entry and follow the online guide to complete the tutorial.
|
|
8
|
-
|
|
8
|
+
|
|
9
9
|
|
|
10
10
|
### Import Script
|
|
11
11
|
|
|
12
12
|
Open the "Lakehouse Tutorial" page in the console and select the "Quick Start Query Analysis with Sample Data" course. Follow the prompts on the page to import the script files needed for this course.
|
|
13
13
|
|
|
14
14
|
View the "Tutorial\_Run\_TPCH\_Queries\_USING\_SQL" directory in the "Development Module".
|
|
15
|
-
|
|
15
|
+
|
|
16
16
|
|
|
17
17
|
### Sample Dataset
|
|
18
18
|
|
|
@@ -32,34 +32,33 @@ Through the above steps, you will be able to evaluate the performance of SQL que
|
|
|
32
32
|
## Preparation
|
|
33
33
|
|
|
34
34
|
First, after logging into the Lakehouse Web console and entering the specified workspace, you can access the data module to check if the relevant tables under "clickzetta\_sample\_data.tpch\_100g" exist in the data object list under data management.
|
|
35
|
-
|
|
35
|
+
|
|
36
36
|
|
|
37
37
|
Next, we will temporarily create an independent compute cluster for query analysis for this test. You can create a new cluster through the "Compute → Clusters" menu in the Lakehouse Web console by following the page wizard.
|
|
38
|
-
|
|
38
|
+
|
|
39
39
|
|
|
40
40
|
At the same time, you can also create a cluster through SQL commands. When operating through SQL commands, you can control cluster creation, scaling, pausing/resuming, and destruction actions without leaving the SQL development context, often improving the efficiency of computing resource operations during Ad-hoc or ETL development processes.
|
|
41
41
|
|
|
42
42
|
This tutorial creates the compute cluster needed for analyzing the TPCH dataset by running the \[Tutorials/Tutorial\_Run\_TPCH\_Queries\_USING\_SQL/Step01.Preparation] SQL script task in the "Development" module.
|
|
43
|
-
|
|
43
|
+
|
|
44
44
|
|
|
45
45
|
## TPC-H Query on Sample Data
|
|
46
46
|
|
|
47
47
|
Open the \[Tutorials/Tutorial\_Run\_TPCH\_Queries\_USING\_SQL/Step02.Run\_TPCH\_Queries] SQL script file in the "Development" module. You will see that the 22 TPC-H query statements have been entered. Select the test cluster just created from the \[Cluster] dropdown list, then select all scripts in the task and click the \[Run] button to perform serial queries.
|
|
48
|
-
|
|
48
|
+
|
|
49
49
|
|
|
50
50
|
After execution, you can view the runtime of this task through the current SQL Editor's run history.
|
|
51
51
|
|
|
52
52
|
If you want to perform performance testing, you can execute the queries more than twice consecutively so that the compute cluster can fully cache the data and achieve optimal performance. The following is the result of the second run, showing a significant performance improvement after the compute cluster caches the data compared to the first run without caching.
|
|
53
|
-
|
|
53
|
+
|
|
54
54
|
|
|
55
55
|
If you want to see the execution details of each of the 22 queries, you can access Compute → Job History, filter the query history by the query tag "tpch100g\_benchmark" and view it.
|
|
56
56
|
|
|
57
|
-

|
|
58
57
|
|
|
59
58
|
## Query After Expanding Cluster Specifications
|
|
60
59
|
|
|
61
60
|
Through the Compute → Cluster Management page, you can modify the size of the test cluster you just created, changing it from Large to XLarge, where the XLarge size is twice that of Large.
|
|
62
|
-
|
|
61
|
+
|
|
63
62
|
|
|
64
63
|
Or execute the following command in the SQL script to modify:
|
|
65
64
|
|
|
@@ -69,11 +68,11 @@ alter vcluster TPCH_100GB SET VCLUSTER_SIZE = 'XLARGE';
|
|
|
69
68
|
```
|
|
70
69
|
|
|
71
70
|
After modification, use the resized cluster to perform the query test again.
|
|
72
|
-
|
|
71
|
+
|
|
73
72
|
After the new expanded compute nodes are fully cached, the performance will continue to improve.
|
|
74
73
|
By observing the runtime of the job, it can be seen that with the same data scale and query tasks, by increasing the size of the compute cluster, the overall runtime of the task is greatly reduced. After two executions, as the data is cached, query performance can be improved.
|
|
75
74
|
|
|
76
75
|
## Environment Cleanup
|
|
77
76
|
|
|
78
77
|
Open the "Development" module \[Tutorial\_Run\_TPCH\_Queries\_USING\_SQL->Step03.Clean\_Up] SQL script file, and execute the script to delete the test cluster for this tutorial.
|
|
79
|
-
|
|
78
|
+
|
|
@@ -7,7 +7,6 @@ If the address of the Git repository you need to associate is not public, you ne
|
|
|
7
7
|
Operation entry: Left navigation menu -> More -> Associate Git
|
|
8
8
|
|
|
9
9
|
Click Add to create an account link.
|
|
10
|
-

|
|
11
10
|
|
|
12
11
|
Display Name: A user-defined name for the currently managed account.
|
|
13
12
|
|
|
@@ -18,7 +17,6 @@ Access Token: The access token created by the user in GitLab. Note that if you n
|
|
|
18
17
|
## Create Git Folder
|
|
19
18
|
|
|
20
19
|
On the development page, click "New" and select the "Git Folder" type to create a special folder type associated with a Git repository.
|
|
21
|
-

|
|
22
20
|
|
|
23
21
|
**Type**: Git Folders support two types. Only Git Folders of the same type can be created within the same workspace.
|
|
24
22
|
|
|
@@ -42,7 +40,6 @@ After creating a Git Folder, the content of the remote repository will be cloned
|
|
|
42
40
|
> Singdata Studio will only clone content that complies with the Singdata task script specification. If there is content in the remote repository that does not comply with the specification, it will be automatically filtered out.
|
|
43
41
|
|
|
44
42
|
Click the "Git" button on the toolbar to open the Git operation panel. Through this panel, you can implement version control, collaboration, and CI/CD with the remote Git repository. The Git panel supports operations including: Commit and Push, Pull, Branch Merge, and visual diff comparison during commits.
|
|
45
|
-

|
|
46
43
|
|
|
47
44
|
1. The branch you are currently using. This is the branch you selected for cloning when creating the Git Folder. Switching to other branches is currently not supported.
|
|
48
45
|
2. Commit your work to the working branch and push the updated branch to the remote Git repository.
|
|
@@ -55,3 +52,5 @@ Click the "Git" button on the toolbar to open the Git operation panel. Through t
|
|
|
55
52
|
|
|
56
53
|
6. Add a commit message, and optionally add a detailed description of the changes.
|
|
57
54
|
7. Display the current branch history.
|
|
55
|
+
|
|
56
|
+
^
|
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
# Accessing Databricks Iceberg Tables via Iceberg REST Catalog
|
|
2
|
+
|
|
3
|
+
Singdata Lakehouse accesses Iceberg-format tables in Databricks Unity Catalog through the Iceberg REST Catalog protocol. Metadata is synchronized in real time, and data remains in the original S3 storage without any migration.
|
|
4
|
+
|
|
5
|
+

|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Prerequisites
|
|
10
|
+
|
|
11
|
+
- Databricks workspace: Unity Catalog must be enabled
|
|
12
|
+
- Singdata Lakehouse instance: must be on the same cloud platform (both AWS) as the Databricks data storage (S3)
|
|
13
|
+
- Iceberg table data stored in an **S3 bucket you control**, with cross-account read permissions configured for the Lakehouse
|
|
14
|
+
- Databricks Service Principal: OAuth Client ID and Secret already obtained
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Databricks-Side Preparation
|
|
19
|
+
|
|
20
|
+
### 1. Obtain OAuth Credentials
|
|
21
|
+
|
|
22
|
+
Go to `https://accounts.cloud.databricks.com` → **Service principals** → create or select a Service Principal → **Credentials & secrets** → record the Application ID (Client ID) and Secret.
|
|
23
|
+
|
|
24
|
+
### 2. Grant Catalog Permissions
|
|
25
|
+
|
|
26
|
+
```sql
|
|
27
|
+
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<application-id>`;
|
|
28
|
+
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<application-id>`;
|
|
29
|
+
GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO `<application-id>`;
|
|
30
|
+
GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<application-id>`;
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### 3. Enable Metastore External Data Access
|
|
34
|
+
|
|
35
|
+
Databricks Workspace → **Catalog** → gear icon → **Metastore** → **External data access** → enable.
|
|
36
|
+
|
|
37
|
+
### 4. Iceberg Table Format Requirements (Critical)
|
|
38
|
+
|
|
39
|
+
> ⚠️ **Note**: Only Iceberg-format tables can be queried via Iceberg REST Catalog. Tables in Delta, Parquet, CSV, and other formats are visible in `SHOW TABLES`, but `SELECT` will return `table not found`.
|
|
40
|
+
|
|
41
|
+
| Table Format | SHOW TABLES | SELECT | Notes |
|
|
42
|
+
|--------------|-------------|--------|-------|
|
|
43
|
+
| **Iceberg** | ✅ | ✅ | The only supported type |
|
|
44
|
+
| Delta | ✅ | ❌ | Metadata visible, cannot query |
|
|
45
|
+
| Parquet / CSV / JSON | ✅ | ❌ | Same as above |
|
|
46
|
+
|
|
47
|
+
Creating an Iceberg table in Databricks:
|
|
48
|
+
|
|
49
|
+
```sql
|
|
50
|
+
CREATE TABLE catalog.schema.my_iceberg_table
|
|
51
|
+
USING ICEBERG
|
|
52
|
+
LOCATION 's3://your-bucket/path/';
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### 5. S3 Storage Requirements
|
|
56
|
+
|
|
57
|
+
Iceberg table data must be in an **S3 bucket you control**. Tables in Databricks-managed storage (`s3://dbstorage-prod-*`) have accessible metadata, but data queries will return `S3 403 Forbidden`.
|
|
58
|
+
|
|
59
|
+
The Lakehouse AWS account also needs read access to the S3 bucket. You can obtain the Lakehouse AWS account ID through technical support.
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## Create a Catalog Connection
|
|
64
|
+
|
|
65
|
+
```sql
|
|
66
|
+
CREATE CATALOG CONNECTION IF NOT EXISTS databricks_iceberg_conn
|
|
67
|
+
TYPE ICEBERG_REST
|
|
68
|
+
URI = 'https://<workspace>.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest'
|
|
69
|
+
OAUTH_SERVER_URI = 'https://<workspace>.cloud.databricks.com/oidc/v1/token'
|
|
70
|
+
ACCESS_REGION = '<s3-bucket-region>'
|
|
71
|
+
OAUTH_CLIENT_ID = '<oauth-client-id>'
|
|
72
|
+
OAUTH_CLIENT_SECRET = '<oauth-client-secret>'
|
|
73
|
+
OAUTH_SCOPE = 'all-apis'
|
|
74
|
+
WAREHOUSE = '<databricks-catalog-name>';
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
### Parameter Reference
|
|
78
|
+
|
|
79
|
+
| Parameter | Required | Description |
|
|
80
|
+
|-----------|----------|-------------|
|
|
81
|
+
| `TYPE` | Yes | Fixed as `ICEBERG_REST` — do not add `=` |
|
|
82
|
+
| `URI` | Yes | Iceberg REST API endpoint — note the **new path** `/iceberg-rest`, not the old path `/iceberg` |
|
|
83
|
+
| `OAUTH_SERVER_URI` | **Yes** | Databricks OAuth token endpoint. Different path from URI — omitting it causes `Credential was not sent` |
|
|
84
|
+
| `ACCESS_REGION` | Yes | Region of the S3 bucket, not the Databricks workspace region |
|
|
85
|
+
| `OAUTH_CLIENT_ID` | Yes | Service Principal Application ID (UUID format) |
|
|
86
|
+
| `OAUTH_CLIENT_SECRET` | Yes | Service Principal OAuth Secret |
|
|
87
|
+
| `OAUTH_SCOPE` | Yes | Fixed as `all-apis` |
|
|
88
|
+
| `WAREHOUSE` | Yes | Catalog name in Databricks Unity Catalog (e.g., `workspace`, `main`) |
|
|
89
|
+
|
|
90
|
+
> ⚠️ **Note**: `TYPE = ICEBERG_REST` will cause a syntax error. The correct form omits `=`: `TYPE ICEBERG_REST`
|
|
91
|
+
|
|
92
|
+
---
|
|
93
|
+
|
|
94
|
+
## Create an External Catalog
|
|
95
|
+
|
|
96
|
+
```sql
|
|
97
|
+
CREATE EXTERNAL CATALOG databricks_iceberg_catalog
|
|
98
|
+
CONNECTION databricks_iceberg_conn;
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
> Note: Iceberg REST Catalog does not require `OPTIONS ('catalog' = '...')` — the warehouse was already specified in the connection above.
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
## Verification
|
|
106
|
+
|
|
107
|
+
```sql
|
|
108
|
+
-- Verify connectivity
|
|
109
|
+
SHOW SCHEMAS IN databricks_iceberg_catalog;
|
|
110
|
+
|
|
111
|
+
-- List tables
|
|
112
|
+
SHOW TABLES IN databricks_iceberg_catalog.<schema>;
|
|
113
|
+
|
|
114
|
+
-- Describe table structure
|
|
115
|
+
DESC TABLE databricks_iceberg_catalog.<schema>.<iceberg_table>;
|
|
116
|
+
|
|
117
|
+
-- Query data
|
|
118
|
+
SELECT * FROM databricks_iceberg_catalog.<schema>.<iceberg_table> LIMIT 10;
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
## Common Errors
|
|
124
|
+
|
|
125
|
+
### `Credential was not sent or was of an unsupported type`
|
|
126
|
+
|
|
127
|
+
`OAUTH_SERVER_URI` is not set. The Databricks OAuth token endpoint and the Iceberg REST endpoint are at different paths and must be specified separately.
|
|
128
|
+
|
|
129
|
+
### Legacy Iceberg endpoints deprecated
|
|
130
|
+
|
|
131
|
+
`URI` is using the old path `/iceberg`. Change to the new path `/iceberg-rest`.
|
|
132
|
+
|
|
133
|
+
### Must provide 'warehouse' parameter
|
|
134
|
+
|
|
135
|
+
`WAREHOUSE` is not set. Databricks' new Iceberg REST API requires the catalog name to be explicitly specified.
|
|
136
|
+
|
|
137
|
+
### S3 403 Forbidden
|
|
138
|
+
|
|
139
|
+
The Iceberg table data is in Databricks-managed storage (`s3://dbstorage-prod-*`), which Lakehouse does not have permission to read. You need to migrate the table data to a user-controlled S3 bucket and configure cross-account permissions.
|
|
140
|
+
|
|
141
|
+
### table or view not found (on SELECT)
|
|
142
|
+
|
|
143
|
+
The table is not in Iceberg format (it may be Delta / Parquet / CSV). `SHOW TABLES` displays all formats, but only Iceberg-format tables can be queried with `SELECT`.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## Management
|
|
148
|
+
|
|
149
|
+
```sql
|
|
150
|
+
-- View connection details
|
|
151
|
+
DESC CONNECTION databricks_iceberg_conn;
|
|
152
|
+
|
|
153
|
+
-- Drop Catalog
|
|
154
|
+
DROP CATALOG databricks_iceberg_catalog;
|
|
155
|
+
|
|
156
|
+
-- Drop connection
|
|
157
|
+
DROP CONNECTION databricks_iceberg_conn;
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
## Related Documentation
|
|
163
|
+
|
|
164
|
+
- [Create Catalog Connection](create-catalog-connection.md) — Full DDL syntax
|
|
165
|
+
- [Databricks Unity Catalog Federated Query Practice](databricks-external-catalog-practice.md) — TYPE DATABRICKS approach
|
|
166
|
+
- [External Catalog Federated Query](SQL_External_Catalog_Guide.md)
|
|
@@ -6,7 +6,7 @@ A branch task implements conditional routing in workflow scheduling. It dynamica
|
|
|
6
6
|
|
|
7
7
|
This article describes the structure and logic of branch tasks.
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
***
|
|
10
10
|
|
|
11
11
|
## Use Cases
|
|
12
12
|
|
|
@@ -14,7 +14,7 @@ If you are designing a workflow and need to decide how to proceed based on the o
|
|
|
14
14
|
|
|
15
15
|
## How It Works
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
^
|
|
18
18
|
|
|
19
19
|
1. **Upstream task output**: The upstream task (SQL, Python, Shell, etc.) passes a dataset to the downstream branch node via the `$[output]` parameter.
|
|
20
20
|
|
|
@@ -26,7 +26,7 @@ If you are designing a workflow and need to decide how to proceed based on the o
|
|
|
26
26
|
|
|
27
27
|
3. **Downstream task reference**: When a downstream task configures its dependency, it must specify which output name of the upstream branch node to associate with.
|
|
28
28
|
|
|
29
|
-
|
|
29
|
+
***
|
|
30
30
|
|
|
31
31
|
## Steps
|
|
32
32
|
|
|
@@ -51,7 +51,7 @@ The output identifier produced after the branch condition is evaluated. Used to
|
|
|
51
51
|
* When the branch condition is met: The downstream node associated with the corresponding output name is triggered.
|
|
52
52
|
* When the branch condition is not met: The downstream node associated with the corresponding output name is not executed, and its status is set to **Ended because branch condition was not met**.
|
|
53
53
|
|
|
54
|
-
Output name format: Supports Chinese characters, English letters, and underscores `_`. Maximum 128 characters. **Duplicate output names are not allowed within the same branch task
|
|
54
|
+
Output name format: Supports Chinese characters, English letters, and underscores `_`. Maximum 128 characters. **Duplicate output names are not allowed within the same branch task**.
|
|
55
55
|
|
|
56
56
|
**Branch Description**
|
|
57
57
|
|
|
@@ -79,82 +79,84 @@ Example: A branch node has two downstream nodes named `aaa` and `bbb`. When bran
|
|
|
79
79
|
|
|
80
80
|
**Currently not supported**: Mathematical operations, date/time functions, JSON Path / object property access, regular expression matching, null coalescing, and similar functions.
|
|
81
81
|
|
|
82
|
-
|
|
82
|
+
***
|
|
83
83
|
|
|
84
84
|
### Logical Functions
|
|
85
85
|
|
|
86
|
-
| Function Signature
|
|
87
|
-
|
|
|
88
|
-
| `and(arg1, arg2, ...)`
|
|
89
|
-
| `or(arg1, arg2, ...)`
|
|
90
|
-
| `not(arg)`
|
|
86
|
+
| Function Signature | Return Type | Description |
|
|
87
|
+
| ---------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
88
|
+
| `and(arg1, arg2, ...)` | `boolean` | Returns `true` when all arguments convert to `true`. Supports 1 or more arguments. Example: `and(true, bool(1), equals('a', 'a'))` |
|
|
89
|
+
| `or(arg1, arg2, ...)` | `boolean` | Returns `true` when any argument converts to `true`. Supports 1 or more arguments. |
|
|
90
|
+
| `not(arg)` | `boolean` | Returns the boolean negation of the argument. |
|
|
91
91
|
|
|
92
|
-
|
|
92
|
+
***
|
|
93
93
|
|
|
94
94
|
### Comparison Functions
|
|
95
95
|
|
|
96
96
|
When both sides can be converted to numbers, comparison is numeric; otherwise, lexicographic string comparison is used.
|
|
97
97
|
|
|
98
|
-
| Function Signature
|
|
99
|
-
|
|
|
100
|
-
| `equals(left, right)`
|
|
101
|
-
| `greater(left, right)`
|
|
102
|
-
| `greaterOrEquals(left, right)`
|
|
103
|
-
| `less(left, right)`
|
|
104
|
-
| `lessOrEquals(left, right)`
|
|
98
|
+
| Function Signature | Return Type | Description |
|
|
99
|
+
| ------------------------------ | ----------- | ------------------------------------------------------ |
|
|
100
|
+
| `equals(left, right)` | `boolean` | Returns `true` if the two normalized values are equal. |
|
|
101
|
+
| `greater(left, right)` | `boolean` | `left > right` |
|
|
102
|
+
| `greaterOrEquals(left, right)` | `boolean` | `left >= right` |
|
|
103
|
+
| `less(left, right)` | `boolean` | `left < right` |
|
|
104
|
+
| `lessOrEquals(left, right)` | `boolean` | `left <= right` |
|
|
105
105
|
|
|
106
|
-
|
|
106
|
+
***
|
|
107
107
|
|
|
108
108
|
### Collection Functions
|
|
109
109
|
|
|
110
110
|
Collection functions apply to arrays (`List` / `Collection` / `JSONArray`) and strings. Some also support dictionaries (`JSONObject` / `Map`).
|
|
111
111
|
|
|
112
|
-
| Function Signature
|
|
113
|
-
|
|
|
114
|
-
| `contains(container, expected)`
|
|
115
|
-
| `empty(value)`
|
|
116
|
-
| `first(value)`
|
|
117
|
-
| `last(value)`
|
|
118
|
-
| `length(value)`
|
|
119
|
-
| `skip(value, count)`
|
|
120
|
-
| `take(value, count)`
|
|
121
|
-
| `union(list1, list2, ...)`
|
|
122
|
-
| `intersection(list1, list2, ...)`
|
|
123
|
-
| `join(list, separator)`
|
|
124
|
-
|
|
125
|
-
|
|
112
|
+
| Function Signature | Return Type | Description |
|
|
113
|
+
| --------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
114
|
+
| `contains(container, expected)` | `boolean` | String: checks if the substring is contained. Collection: compares items by normalized value. JSONObject / Map: checks if the key exists. |
|
|
115
|
+
| `empty(value)` | `boolean` | Returns `true` for `null`, empty string, empty collection, or empty JSONObject / Map. |
|
|
116
|
+
| `first(value)` | First element | String: returns the first character. List / Collection: returns the first element. Returns `null` when empty. |
|
|
117
|
+
| `last(value)` | Last element | String: returns the last character. List / Collection: returns the last element. Returns `null` when empty. |
|
|
118
|
+
| `length(value)` | `integer` | String: returns character count. Collection / Map: returns element count. Returns `0` for `null`. |
|
|
119
|
+
| `skip(value, count)` | Same as input type | Skips the first `count` characters or elements. Returns the original value when `count < 0`, non-integer, or `value` is `null`. |
|
|
120
|
+
| `take(value, count)` | Same as input type | Keeps the first `count` characters or elements. Returns the original value when `count < 0`, non-integer, or `value` is `null`. |
|
|
121
|
+
| `union(list1, list2, ...)` | `List` | Merges multiple lists with deduplication, preserving first-occurrence order. All arguments must be convertible to lists, otherwise returns `null`. |
|
|
122
|
+
| `intersection(list1, list2, ...)` | `List` | Returns the intersection of multiple lists, ordered by the first list. All arguments must be convertible to lists, otherwise returns `null`. |
|
|
123
|
+
| `join(list, separator)` | `String` | Joins all list elements with the separator. The first argument must be convertible to a list, otherwise returns `null`. |
|
|
124
|
+
|
|
125
|
+
***
|
|
126
126
|
|
|
127
127
|
### String Functions
|
|
128
128
|
|
|
129
|
-
| Function Signature
|
|
130
|
-
|
|
|
131
|
-
| `concat(arg1, arg2, ...)`
|
|
132
|
-
| `guid()`
|
|
133
|
-
| `indexOf(text, search)`
|
|
134
|
-
| `lastIndexOf(text, search)`
|
|
135
|
-
| `replace(text, oldValue, newValue)`
|
|
136
|
-
| `split(text, separator)`
|
|
137
|
-
| `startsWith(text, prefix)`
|
|
138
|
-
| `endsWith(text, suffix)`
|
|
139
|
-
| `substring(text, start, length)`
|
|
140
|
-
| `toLower(value)`
|
|
141
|
-
| `toUpper(value)`
|
|
142
|
-
| `trim(value)`
|
|
143
|
-
| `string(value)`
|
|
144
|
-
|
|
145
|
-
|
|
129
|
+
| Function Signature | Return Type | Description |
|
|
130
|
+
| ----------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
131
|
+
| `concat(arg1, arg2, ...)` | `String` | Concatenates all arguments in order. `null` arguments are ignored. Supports 1 or more arguments. |
|
|
132
|
+
| `guid()` | `String` | Generates a random UUID string. Accepts no arguments. |
|
|
133
|
+
| `indexOf(text, search)` | `integer` | Returns the position of the first occurrence of the substring. Returns `-1` if not found. |
|
|
134
|
+
| `lastIndexOf(text, search)` | `integer` | Returns the position of the last occurrence of the substring. Returns `-1` if not found. |
|
|
135
|
+
| `replace(text, oldValue, newValue)` | `String` | Replaces all matching substrings. Returns empty string if `text` is `null`. Treats `oldValue` or `newValue` as empty string when `null`. |
|
|
136
|
+
| `split(text, separator)` | `List<String>` | Splits the string by the separator. Splits by single character when separator is empty. Returns empty list if `text` is `null`. |
|
|
137
|
+
| `startsWith(text, prefix)` | `boolean` | Checks if the string starts with the specified prefix. |
|
|
138
|
+
| `endsWith(text, suffix)` | `boolean` | Checks if the string ends with the specified suffix. |
|
|
139
|
+
| `substring(text, start, length)` | `String` | Extracts `length` characters starting from position `start`. Returns empty string when `start` is out of bounds, arguments are invalid, or negative. Auto-truncates at end. |
|
|
140
|
+
| `toLower(value)` | `String` | Converts to lowercase. Returns empty string for `null`. |
|
|
141
|
+
| `toUpper(value)` | `String` | Converts to uppercase. Returns empty string for `null`. |
|
|
142
|
+
| `trim(value)` | `String` | Trims leading and trailing whitespace. Returns empty string for `null`. |
|
|
143
|
+
| `string(value)` | `String` | Converts the value to a string. Returns empty string for `null`. |
|
|
144
|
+
|
|
145
|
+
***
|
|
146
146
|
|
|
147
147
|
### Conversion Functions
|
|
148
148
|
|
|
149
|
-
| Function Signature | Return Type | Description
|
|
150
|
-
| ------------------ | ----------- |
|
|
149
|
+
| Function Signature | Return Type | Description |
|
|
150
|
+
| ------------------ | ----------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
151
151
|
| `bool(value)` | `boolean` | Returns a boolean value according to the built-in boolean conversion rules (see the type description at the top of this document). |
|
|
152
152
|
|
|
153
|
-
|
|
153
|
+
***
|
|
154
154
|
|
|
155
155
|
## Related Documentation
|
|
156
156
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
157
|
+
* [Task Parameters](task_param.md) — Using dynamic parameter values in condition expressions
|
|
158
|
+
* [Task Parameter Syntax Reference](task_param_reference.md) — Full syntax for time expressions and built-in parameters
|
|
159
|
+
* [Composite Task](composite_task.md) — Incorporating branch tasks into a DAG
|
|
160
|
+
* [Task Development and Scheduling](task-develop.md) — Development and scheduling for SQL tasks
|
|
161
|
+
|
|
162
|
+
^
|
|
@@ -8,11 +8,11 @@ Context parameters, in parameter configuration, refer to configuring two special
|
|
|
8
8
|
|
|
9
9
|
## Usage Guide
|
|
10
10
|
|
|
11
|
-
* **Upstream Node (Provider)
|
|
11
|
+
* **Upstream Node (Provider**): Responsible for generating a value and using it as an output parameter.
|
|
12
12
|
|
|
13
|
-
* **Configure and Assign Output Parameter**: The system captures the query result of the last line of the node's code (e.g., `SELECT * from table_A;`) and **assigns** it to a built-in output parameter
|
|
13
|
+
* **Configure and Assign Output Parameter**: The system captures the query result of the last line of the node's code (e.g., `SELECT * from table_A;`) and **assigns** it to a built-in output parameter \`\`, then passes the value of this parameter to downstream nodes. The parameter value depends on the code execution result.
|
|
14
14
|
|
|
15
|
-
* **Downstream Node (Consumer)
|
|
15
|
+
* **Downstream Node (Consumer**): Receives and uses the value provided by the upstream node.
|
|
16
16
|
|
|
17
17
|
* **Establish Scheduling Dependency**: Configure a scheduling dependency on the upstream node.
|
|
18
18
|
* **Configure Input Parameter**: In the downstream node's **parameter configuration** area, add an input parameter and set its value source to the upstream node's output parameter.
|
|
@@ -21,12 +21,13 @@ Context parameters, in parameter configuration, refer to configuring two special
|
|
|
21
21
|
## Configuration Instructions
|
|
22
22
|
|
|
23
23
|
### Configure Output Parameters
|
|
24
|
-
|
|
24
|
+
|
|
25
|
+
^
|
|
25
26
|
|
|
26
27
|
If you wish to pass the current node's result to downstream, follow these steps:
|
|
27
28
|
|
|
28
29
|
1. Click the node, then Parameters > Output Parameters
|
|
29
|
-
2. Click "New", and the system will automatically fill in the built-in parameter
|
|
30
|
+
2. Click "New", and the system will automatically fill in the built-in parameter $\[output].
|
|
30
31
|
|
|
31
32
|
> If the task is of Python/Shell type, the output result defaults to the output of the last line, with result data split by delimiter.
|
|
32
33
|
>
|
|
@@ -47,7 +48,7 @@ Configure the input parameter.
|
|
|
47
48
|
* Upstream Dependency: From the currently configured upstream tasks, select those that have output parameters as the input parameter value source.
|
|
48
49
|
* Others: To add a task with output parameters from elsewhere, use "Others" to select it. Once selected, the dependency on that task will be automatically configured.
|
|
49
50
|
|
|
50
|
-
|
|
51
|
+
^
|
|
51
52
|
|
|
52
53
|
#### Reference Parameters
|
|
53
54
|
|
|
@@ -67,4 +68,6 @@ If the upstream node passes the assigned result to downstream, the parameter val
|
|
|
67
68
|
* Cell: `${param[i][j]}`.
|
|
68
69
|
* If the upstream is Python/Shell (one-dimensional array): Row: `${param[i]}`.
|
|
69
70
|
|
|
70
|
-
> *Indexes start from 0
|
|
71
|
+
> *Indexes start from 0*.
|
|
72
|
+
|
|
73
|
+
^
|