@clickzetta/cz-cli-darwin-arm64 0.3.87 → 0.3.88
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/cz-cli +0 -0
- package/bin/skills/clickzetta-dynamic-table/SKILL.md +169 -169
- package/bin/skills/clickzetta-dynamic-table/best-practices/dimension-table-join-guide.md +126 -126
- package/bin/skills/clickzetta-dynamic-table/best-practices/medallion-and-stream-patterns.md +25 -25
- package/bin/skills/clickzetta-dynamic-table/best-practices/non-partitioned-merge-into-warning.md +48 -48
- package/bin/skills/clickzetta-dynamic-table/best-practices/performance-optimization.md +51 -51
- package/bin/skills/clickzetta-dynamic-table/best-practices/scheduling-guide.md +59 -59
- package/bin/skills/clickzetta-dynamic-table/dt-creator/SKILL.md +8 -7
- package/bin/skills/clickzetta-dynamic-table/dt-creator/references/dt-declaration-strategy.md +99 -99
- package/bin/skills/clickzetta-dynamic-table/dt-creator/references/incremental-config-reference.md +188 -188
- package/bin/skills/clickzetta-dynamic-table/dt-creator/references/refresh-history-guide.md +117 -117
- package/bin/skills/clickzetta-dynamic-table/dt-creator/references/sql-limitations.md +29 -29
- package/bin/skills/clickzetta-dynamic-table/dynamic-table-alter/SKILL.md +80 -79
- package/bin/skills/clickzetta-dynamic-table/sql-to-dt/SKILL.md +15 -15
- package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-column-validation-rules.md +61 -61
- package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-conversion-rules.md +100 -100
- package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-placeholder-rules.md +64 -64
- package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-refresh-rules.md +32 -32
- package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-self-reference-rules.md +21 -21
- package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-workflow.md +71 -71
- package/bin/skills/clickzetta-sql-pipeline-manager/SKILL.md +203 -202
- package/bin/skills/clickzetta-sql-pipeline-manager/references/dynamic-table.md +62 -62
- package/bin/skills/clickzetta-sql-pipeline-manager/references/materialized-view.md +34 -34
- package/bin/skills/clickzetta-sql-pipeline-manager/references/pipe.md +61 -61
- package/bin/skills/clickzetta-sql-pipeline-manager/references/table-stream.md +41 -41
- package/bin/skills/clickzetta-table-stream-pipeline/SKILL.md +103 -101
- package/package.json +1 -1
|
@@ -1,160 +1,161 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dynamic-table-alter
|
|
3
3
|
description: |
|
|
4
|
-
|
|
5
|
-
rename_column
|
|
6
|
-
|
|
7
|
-
"
|
|
4
|
+
Modify the structure and properties of ClickZetta Dynamic Tables. Supports direct ALTER operations
|
|
5
|
+
(suspend, resume, rename_column, set_comment, set_column_comment, set/unset properties) as well as
|
|
6
|
+
CREATE OR REPLACE rebuild operations (modify refresh interval, compute cluster, add column, drop column,
|
|
7
|
+
change column type, change SQL definition). Triggers when the user says "modify dynamic table",
|
|
8
|
+
"add column to dynamic table", "change refresh interval", or "suspend dynamic table".
|
|
8
9
|
---
|
|
9
10
|
|
|
10
|
-
#
|
|
11
|
+
# Dynamic Table Modification Workflow
|
|
11
12
|
|
|
12
|
-
##
|
|
13
|
+
## Instructions
|
|
13
14
|
|
|
14
|
-
###
|
|
15
|
-
|
|
16
|
-
|
|
15
|
+
### Step 1: Confirm the Dynamic Table Exists and Retrieve Its Current Definition
|
|
16
|
+
Execute `SHOW CREATE TABLE schema_name.table_name` to get the current definition of the Dynamic Table.
|
|
17
|
+
If unsure whether it is a Dynamic Table, first use `SHOW TABLES WHERE is_dynamic` to view the list.
|
|
17
18
|
|
|
18
|
-
###
|
|
19
|
+
### Step 2: Determine the Operation Type and Choose the Execution Method
|
|
19
20
|
|
|
20
|
-
|
|
21
|
+
Dynamic Table modification operations fall into two categories:
|
|
21
22
|
|
|
22
|
-
**A.
|
|
23
|
+
**A. Direct ALTER operations** (6 types, can be executed directly):
|
|
23
24
|
|
|
24
|
-
1. **suspend** —
|
|
25
|
+
1. **suspend** — Pause the scheduled refresh:
|
|
25
26
|
```sql
|
|
26
27
|
ALTER DYNAMIC TABLE dt_name SUSPEND;
|
|
27
28
|
```
|
|
28
29
|
|
|
29
|
-
2. **resume** —
|
|
30
|
+
2. **resume** — Start the scheduled refresh:
|
|
30
31
|
```sql
|
|
31
32
|
ALTER DYNAMIC TABLE dt_name RESUME;
|
|
32
33
|
```
|
|
33
34
|
|
|
34
|
-
3. **set_comment** —
|
|
35
|
+
3. **set_comment** — Modify the table comment:
|
|
35
36
|
```sql
|
|
36
37
|
ALTER DYNAMIC TABLE dt_name SET COMMENT 'comment';
|
|
37
38
|
```
|
|
38
39
|
|
|
39
|
-
4. **rename_column** —
|
|
40
|
+
4. **rename_column** — Rename a column:
|
|
40
41
|
```sql
|
|
41
42
|
ALTER DYNAMIC TABLE dt_name RENAME COLUMN old_name TO new_name;
|
|
42
43
|
```
|
|
43
44
|
|
|
44
|
-
5. **set_column_comment** —
|
|
45
|
+
5. **set_column_comment** — Modify a column comment (note: use CHANGE COLUMN):
|
|
45
46
|
```sql
|
|
46
47
|
ALTER DYNAMIC TABLE dt_name CHANGE COLUMN column_name COMMENT 'comment';
|
|
47
48
|
```
|
|
48
49
|
|
|
49
|
-
6. **set/unset properties** —
|
|
50
|
+
6. **set/unset properties** — Modify table properties (currently reserved parameters):
|
|
50
51
|
```sql
|
|
51
|
-
--
|
|
52
|
+
-- Set a property
|
|
52
53
|
ALTER DYNAMIC TABLE dt_name SET PROPERTIES('key' = 'value');
|
|
53
|
-
--
|
|
54
|
+
-- Remove a property
|
|
54
55
|
ALTER DYNAMIC TABLE dt_name UNSET PROPERTIES('key');
|
|
55
56
|
```
|
|
56
57
|
|
|
57
|
-
**B. CREATE OR REPLACE
|
|
58
|
+
**B. CREATE OR REPLACE operations** (6 types, require rebuilding the Dynamic Table):
|
|
58
59
|
|
|
59
|
-
> ⚠️
|
|
60
|
+
> ⚠️ **The following operations do not support ALTER syntax.** Syntax like `ALTER DYNAMIC TABLE ... SET REFRESH INTERVAL` does not exist and will cause a syntax error. You must use `CREATE OR REPLACE DYNAMIC TABLE` to rebuild.
|
|
60
61
|
|
|
61
|
-
|
|
62
|
+
These operations involve changes to SQL query logic and cannot be completed via ALTER directly:
|
|
62
63
|
|
|
63
|
-
7.
|
|
64
|
-
8.
|
|
65
|
-
9.
|
|
66
|
-
10.
|
|
67
|
-
11.
|
|
68
|
-
12.
|
|
64
|
+
7. **Modify refresh interval** — ❌ `ALTER ... SET REFRESH INTERVAL` is not supported
|
|
65
|
+
8. **Modify compute cluster** — ❌ `ALTER ... SET VCLUSTER` is not supported
|
|
66
|
+
9. **Add column**
|
|
67
|
+
10. **Drop column**
|
|
68
|
+
11. **Modify column type**
|
|
69
|
+
12. **Modify SQL definition**
|
|
69
70
|
|
|
70
|
-
###
|
|
71
|
+
### Step 3: Execute CREATE OR REPLACE Rebuild (Type B operations only)
|
|
71
72
|
|
|
72
|
-
1.
|
|
73
|
-
> ⚠️ `SHOW CREATE TABLE`
|
|
74
|
-
2.
|
|
75
|
-
3.
|
|
76
|
-
4.
|
|
73
|
+
1. Execute `SHOW CREATE TABLE schema_name.table_name` to get the original DDL
|
|
74
|
+
> ⚠️ `SHOW CREATE TABLE` does not support LIMIT/WHERE clauses; execute it directly
|
|
75
|
+
2. Parse out: column definitions, REFRESH clause, AS SELECT clause, COMMENT, etc.
|
|
76
|
+
3. Modify the relevant parts according to the operation
|
|
77
|
+
4. Execute the rebuild SQL
|
|
77
78
|
|
|
78
|
-
|
|
79
|
-
-
|
|
80
|
-
-
|
|
81
|
-
-
|
|
79
|
+
**About full refresh triggers:**
|
|
80
|
+
- Simple drop column / add column (where the added column is simply passed through from the source table without participating in JOIN keys, GROUP keys, or other computations) → **incremental refresh**
|
|
81
|
+
- Changes involving computation logic (modifying WHERE conditions, modifying aggregation logic, new column participates in computation, etc.) → **full refresh**
|
|
82
|
+
- Compatible type changes (e.g., INT → BIGINT) → **incremental refresh**
|
|
82
83
|
|
|
83
|
-
###
|
|
84
|
-
|
|
84
|
+
### Step 4: Verify the Modification
|
|
85
|
+
Use `DESC TABLE dt_name` to confirm the modification took effect.
|
|
85
86
|
|
|
86
87
|
---
|
|
87
88
|
|
|
88
|
-
##
|
|
89
|
+
## Examples
|
|
89
90
|
|
|
90
|
-
###
|
|
91
|
+
### Example 1: Modify Refresh Interval
|
|
91
92
|
|
|
92
93
|
```sql
|
|
93
|
-
--
|
|
94
|
+
-- Original table
|
|
94
95
|
CREATE DYNAMIC TABLE dt_name
|
|
95
96
|
REFRESH INTERVAL 10 MINUTE vcluster DEFAULT
|
|
96
97
|
AS SELECT * FROM student02;
|
|
97
98
|
|
|
98
|
-
--
|
|
99
|
+
-- After modification (changed to 20 minutes)
|
|
99
100
|
CREATE OR REPLACE DYNAMIC TABLE dt_name
|
|
100
101
|
REFRESH INTERVAL 20 MINUTE vcluster DEFAULT
|
|
101
102
|
AS SELECT * FROM student02;
|
|
102
103
|
```
|
|
103
104
|
|
|
104
|
-
###
|
|
105
|
+
### Example 2: Modify Compute Cluster
|
|
105
106
|
|
|
106
107
|
```sql
|
|
107
|
-
--
|
|
108
|
+
-- Original table
|
|
108
109
|
CREATE DYNAMIC TABLE dt_name
|
|
109
110
|
REFRESH INTERVAL 10 MINUTE vcluster DEFAULT
|
|
110
111
|
AS SELECT * FROM student02;
|
|
111
112
|
|
|
112
|
-
--
|
|
113
|
+
-- After modification (changed to alter_vc cluster)
|
|
113
114
|
CREATE OR REPLACE DYNAMIC TABLE dt_name
|
|
114
115
|
REFRESH INTERVAL 10 MINUTE vcluster alter_vc
|
|
115
116
|
AS SELECT * FROM student02;
|
|
116
117
|
```
|
|
117
118
|
|
|
118
|
-
###
|
|
119
|
+
### Example 3: Add Column
|
|
119
120
|
|
|
120
121
|
```sql
|
|
121
|
-
--
|
|
122
|
+
-- Original table
|
|
122
123
|
CREATE DYNAMIC TABLE change_table (i, j)
|
|
123
124
|
AS SELECT * FROM dy_base_a;
|
|
124
125
|
|
|
125
|
-
--
|
|
126
|
+
-- Add column col (involves computation logic; next refresh will be a full refresh)
|
|
126
127
|
CREATE OR REPLACE DYNAMIC TABLE change_table (i, j, col)
|
|
127
128
|
AS SELECT i, j, j * 1 FROM dy_base_a;
|
|
128
129
|
|
|
129
130
|
REFRESH DYNAMIC TABLE change_table;
|
|
130
131
|
```
|
|
131
132
|
|
|
132
|
-
###
|
|
133
|
+
### Example 4: Drop Column
|
|
133
134
|
|
|
134
135
|
```sql
|
|
135
|
-
--
|
|
136
|
+
-- Original table has columns i, j
|
|
136
137
|
CREATE DYNAMIC TABLE change_table (i, j)
|
|
137
138
|
AS SELECT * FROM dy_base_a;
|
|
138
139
|
|
|
139
|
-
--
|
|
140
|
+
-- Drop column (simple pass-through; incremental refresh)
|
|
140
141
|
CREATE OR REPLACE DYNAMIC TABLE change_table (i)
|
|
141
142
|
AS SELECT i FROM dy_base_a;
|
|
142
143
|
```
|
|
143
144
|
|
|
144
|
-
###
|
|
145
|
+
### Example 5: Modify SQL Definition
|
|
145
146
|
|
|
146
147
|
```sql
|
|
147
|
-
--
|
|
148
|
+
-- Modify WHERE filter condition (full refresh)
|
|
148
149
|
CREATE OR REPLACE DYNAMIC TABLE change_table (i, j)
|
|
149
150
|
AS SELECT * FROM dy_base_a WHERE i > 3;
|
|
150
151
|
|
|
151
152
|
REFRESH DYNAMIC TABLE change_table;
|
|
152
153
|
```
|
|
153
154
|
|
|
154
|
-
###
|
|
155
|
+
### Example 6: Modify Column Type
|
|
155
156
|
|
|
156
157
|
```sql
|
|
157
|
-
-- INT → BIGINT
|
|
158
|
+
-- INT → BIGINT (compatible type; incremental refresh)
|
|
158
159
|
CREATE OR REPLACE DYNAMIC TABLE change_table (i, j)
|
|
159
160
|
AS SELECT CAST(i AS BIGINT), j FROM dy_base_a;
|
|
160
161
|
|
|
@@ -163,28 +164,28 @@ REFRESH DYNAMIC TABLE change_table;
|
|
|
163
164
|
|
|
164
165
|
---
|
|
165
166
|
|
|
166
|
-
##
|
|
167
|
+
## Platform-Specific Knowledge
|
|
167
168
|
|
|
168
|
-
- **CHANGE COLUMN
|
|
169
|
-
- **RENAME COLUMN
|
|
170
|
-
- **DML
|
|
171
|
-
- **REFRESH
|
|
172
|
-
- **CREATE OR REPLACE
|
|
173
|
-
- **
|
|
174
|
-
-
|
|
175
|
-
- **DROP
|
|
176
|
-
- **UNDROP
|
|
177
|
-
- **DESC
|
|
169
|
+
- **CHANGE COLUMN syntax**: set a column comment with `CHANGE COLUMN col COMMENT 'xxx'`, not `ALTER COLUMN`
|
|
170
|
+
- **RENAME COLUMN syntax**: `RENAME COLUMN old TO new`
|
|
171
|
+
- **DML restrictions**: Dynamic Tables do not support UPDATE/DELETE/MERGE by default (due to hidden column MV__KEY); to use DML, first execute `SET cz.sql.dt.allow.dml = true;`
|
|
172
|
+
- **REFRESH format**: `REFRESH INTERVAL <N> MINUTE vcluster <name>`, supports SECOND/MINUTE/HOUR/DAY
|
|
173
|
+
- **CREATE OR REPLACE risk**: changes involving computation logic will trigger a full refresh, which may take a long time for large tables
|
|
174
|
+
- **Schema prefix**: all ALTER/CREATE statements should include the schema prefix in the table name
|
|
175
|
+
- **Column definitions can omit types**: `CREATE DYNAMIC TABLE dt (i, j) AS SELECT ...` — types are inferred from SELECT
|
|
176
|
+
- **DROP syntax**: must use `DROP DYNAMIC TABLE dt_name`; `DROP TABLE dt_name` will cause an error
|
|
177
|
+
- **UNDROP syntax**: must use `UNDROP TABLE dt_name`; `UNDROP DYNAMIC TABLE dt_name` is not supported
|
|
178
|
+
- **DESC syntax**: use `DESC TABLE dt_name` for Dynamic Tables; do not write `DESC DYNAMIC TABLE dt_name EXTENDED` (EXTENDED is not supported)
|
|
178
179
|
|
|
179
|
-
##
|
|
180
|
+
## Troubleshooting
|
|
180
181
|
|
|
181
|
-
|
|
|
182
|
+
| Error | Cause | Solution |
|
|
182
183
|
|---|---|---|
|
|
183
|
-
| ALTER
|
|
184
|
-
| ALTER
|
|
185
|
-
| `DROP TABLE dt_name`
|
|
186
|
-
| `UNDROP DYNAMIC TABLE`
|
|
187
|
-
| `DESC DYNAMIC TABLE ... EXTENDED`
|
|
188
|
-
| UPDATE/DELETE
|
|
189
|
-
| CREATE OR REPLACE
|
|
190
|
-
| CREATE OR REPLACE
|
|
184
|
+
| ALTER reports "Syntax error at or near 'REFRESH'" | `ALTER ... SET REFRESH INTERVAL` syntax does not exist | Use `CREATE OR REPLACE DYNAMIC TABLE ... REFRESH INTERVAL ...` to rebuild |
|
|
185
|
+
| ALTER reports "unsupported operation" | Attempted a Type B ALTER operation on a Dynamic Table | Use CREATE OR REPLACE to rebuild |
|
|
186
|
+
| `DROP TABLE dt_name` fails | Dynamic Tables must use `DROP DYNAMIC TABLE` | Change to `DROP DYNAMIC TABLE dt_name` |
|
|
187
|
+
| `UNDROP DYNAMIC TABLE` fails | UNDROP does not support the DYNAMIC TABLE keyword | Change to `UNDROP TABLE dt_name` |
|
|
188
|
+
| `DESC DYNAMIC TABLE ... EXTENDED` fails | EXTENDED parameter is not supported | Change to `DESC TABLE dt_name` (without EXTENDED) |
|
|
189
|
+
| UPDATE/DELETE reports "MV__KEY" related error | Dynamic Tables have a hidden column MV__KEY; DML is disabled by default | First execute `SET cz.sql.dt.allow.dml = true;` |
|
|
190
|
+
| Data is empty after CREATE OR REPLACE | The AS SELECT clause references an incorrect source table or column | Verify that the SELECT clause returns data first |
|
|
191
|
+
| Full refresh triggered after CREATE OR REPLACE | The new column participates in computation logic (JOIN key, GROUP key, etc.) | Expected behavior; wait for the full refresh to complete |
|
|
@@ -1,27 +1,27 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: sql-to-dt
|
|
3
|
-
description:
|
|
3
|
+
description: Automatically converts CREATE TABLE DDL + INSERT OVERWRITE SQL from Hive/Spark or any batch processing system into Dynamic Table DDL and companion files (refresh, prev_refresh, backfill). Triggers when the user provides a DDL and INSERT OVERWRITE and requests conversion to DT, or when the user says "create dynamic table" and should be proactively guided to provide input. Triggers on: "convert to DT", "sql to dt", "convert to dynamic table", "INSERT OVERWRITE to DT", "DDL conversion", "create dynamic table"
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
# SQL → Dynamic Table
|
|
6
|
+
# SQL → Dynamic Table Automatic Conversion
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Converts ETL SQL (CREATE TABLE + INSERT OVERWRITE) from Hive/Spark or any batch processing system into Dynamic Table DDL and companion operation files.
|
|
9
9
|
|
|
10
|
-
##
|
|
10
|
+
## Usage
|
|
11
11
|
|
|
12
|
-
|
|
13
|
-
1. CREATE TABLE DDL
|
|
14
|
-
2. INSERT OVERWRITE SQL
|
|
12
|
+
Provide the following inputs:
|
|
13
|
+
1. CREATE TABLE DDL (table structure definition)
|
|
14
|
+
2. INSERT OVERWRITE SQL (ETL query logic)
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
The conversion tool will automatically handle: placeholder replacement, self-reference detection, core conversion, column validation, companion file generation, and post-conversion improvement suggestions.
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
For the detailed workflow, see #[[file:references/sql2dt-workflow.md]]
|
|
19
19
|
|
|
20
20
|
## references/
|
|
21
21
|
|
|
22
|
-
- **sql2dt-workflow.md** —
|
|
23
|
-
- **sql2dt-conversion-rules.md** —
|
|
24
|
-
- **sql2dt-placeholder-rules.md** —
|
|
25
|
-
- **sql2dt-self-reference-rules.md** —
|
|
26
|
-
- **sql2dt-column-validation-rules.md** —
|
|
27
|
-
- **sql2dt-refresh-rules.md** — Refresh
|
|
22
|
+
- **sql2dt-workflow.md** — Complete conversion workflow (6 steps: pre-processing, placeholder replacement, self-reference detection, core conversion, column validation, companion file generation)
|
|
23
|
+
- **sql2dt-conversion-rules.md** — Core DDL conversion rules (parse DDL, parse INSERT, assemble DT DDL, static partition injection)
|
|
24
|
+
- **sql2dt-placeholder-rules.md** — Placeholder replacement rules (${var} → SESSION_CONFIGS())
|
|
25
|
+
- **sql2dt-self-reference-rules.md** — Self-referencing table conversion rules
|
|
26
|
+
- **sql2dt-column-validation-rules.md** — Column validation rules (schema column count = SELECT column count)
|
|
27
|
+
- **sql2dt-refresh-rules.md** — Refresh and scheduling file generation rules
|
package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-column-validation-rules.md
CHANGED
|
@@ -1,118 +1,118 @@
|
|
|
1
|
-
# Dynamic Table
|
|
1
|
+
# Dynamic Table Column Validation and Consistency Rules
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
You are a SQL conversion expert. After generating the Dynamic Table DDL, you need to validate whether the columns defined in the schema match the columns produced by the SELECT query.
|
|
4
4
|
|
|
5
|
-
##
|
|
5
|
+
## Column Count Validation (Must Pass)
|
|
6
6
|
|
|
7
|
-
###
|
|
7
|
+
### Rule
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
The number of columns defined in the parentheses of the generated DDL must equal the number of columns produced by the SELECT query after AS.
|
|
10
10
|
|
|
11
11
|
```sql
|
|
12
12
|
CREATE OR REPLACE DYNAMIC TABLE t (
|
|
13
13
|
col1 BIGINT, -- 1
|
|
14
14
|
col2 STRING, -- 2
|
|
15
|
-
dt STRING -- 3 → schema
|
|
15
|
+
dt STRING -- 3 → schema column count = 3
|
|
16
16
|
)
|
|
17
17
|
AS
|
|
18
|
-
SELECT col1, col2, '2024-01-01' AS dt -- → SELECT
|
|
18
|
+
SELECT col1, col2, '2024-01-01' AS dt -- → SELECT column count = 3 ✓
|
|
19
19
|
FROM source;
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
### SELECT
|
|
22
|
+
### Counting SELECT Columns
|
|
23
23
|
|
|
24
|
-
1.
|
|
25
|
-
2.
|
|
26
|
-
3.
|
|
27
|
-
4.
|
|
24
|
+
1. Find the SELECT clause after AS
|
|
25
|
+
2. Find the top-level FROM (not inside a subquery/parentheses)
|
|
26
|
+
3. Count top-level commas between SELECT and FROM + 1 = column count
|
|
27
|
+
4. Top-level commas: commas not inside parentheses `()`, square brackets `[]`, or quotes `''`/`""`
|
|
28
28
|
|
|
29
|
-
### UNION ALL
|
|
29
|
+
### Column Count for UNION ALL
|
|
30
30
|
|
|
31
|
-
|
|
31
|
+
Use the column count of the first branch (all branches should have the same column count).
|
|
32
32
|
|
|
33
|
-
###
|
|
33
|
+
### Validation Failure
|
|
34
34
|
|
|
35
|
-
|
|
35
|
+
If schema column count ≠ SELECT column count, conversion fails with error:
|
|
36
36
|
```
|
|
37
|
-
Schema
|
|
37
|
+
Schema column count (N) != SELECT column count (M)
|
|
38
38
|
```
|
|
39
39
|
|
|
40
|
-
##
|
|
40
|
+
## Column Name Validation (Optional)
|
|
41
41
|
|
|
42
|
-
###
|
|
42
|
+
### Rule
|
|
43
43
|
|
|
44
|
-
|
|
44
|
+
Compare schema column names against inferred aliases from SELECT, position by position. Recommended to enable after column count validation passes, if most columns in SELECT have explicit aliases (AS or bare identifiers).
|
|
45
45
|
|
|
46
|
-
### SELECT
|
|
46
|
+
### Inferring SELECT Column Aliases
|
|
47
47
|
|
|
48
|
-
|
|
48
|
+
In order of priority from high to low:
|
|
49
49
|
|
|
50
|
-
1. **AS
|
|
51
|
-
2.
|
|
52
|
-
3.
|
|
53
|
-
4.
|
|
50
|
+
1. **AS alias**: `expression AS alias` → alias is `alias`
|
|
51
|
+
2. **Trailing identifier**: `table.column` → alias is `column`
|
|
52
|
+
3. **Bare identifier**: `column_name` → alias is `column_name`
|
|
53
|
+
4. **Cannot infer**: `func(a, b)` without AS → mark as `<expr>`, skip validation
|
|
54
54
|
|
|
55
|
-
###
|
|
55
|
+
### Comparison Rules
|
|
56
56
|
|
|
57
|
-
-
|
|
58
|
-
-
|
|
59
|
-
-
|
|
60
|
-
-
|
|
57
|
+
- Compare position by position (1st column vs 1st column, 2nd vs 2nd, ...)
|
|
58
|
+
- If a position is `<expr>` (cannot infer), skip that position
|
|
59
|
+
- Comparison is case-insensitive
|
|
60
|
+
- On mismatch, report error and list the specific misaligned columns
|
|
61
61
|
|
|
62
|
-
##
|
|
62
|
+
## Column Count After Static Partition Injection
|
|
63
63
|
|
|
64
|
-
|
|
64
|
+
After injecting static partition columns, the SELECT column count increases. Validation should be performed after injection.
|
|
65
65
|
|
|
66
|
-
###
|
|
66
|
+
### Avoid Duplicate Injection
|
|
67
67
|
|
|
68
|
-
|
|
68
|
+
Before injection, check whether SELECT already contains the partition column:
|
|
69
69
|
|
|
70
|
-
1.
|
|
71
|
-
2.
|
|
72
|
-
3.
|
|
70
|
+
1. Parse the final alias of each expression in SELECT
|
|
71
|
+
2. If the alias (case-insensitive) matches the partition column name → the column already exists; skip injection
|
|
72
|
+
3. Only inject partition columns not already present in SELECT
|
|
73
73
|
|
|
74
|
-
## UNION ALL
|
|
74
|
+
## UNION ALL Consistency
|
|
75
75
|
|
|
76
|
-
###
|
|
76
|
+
### Branch Column Count Consistency
|
|
77
77
|
|
|
78
|
-
|
|
78
|
+
All UNION ALL branches must have the same column count. If inconsistent, record a warning:
|
|
79
79
|
```
|
|
80
|
-
UNION
|
|
80
|
+
UNION branch column counts are inconsistent: [12, 13, 12]
|
|
81
81
|
```
|
|
82
82
|
|
|
83
|
-
###
|
|
83
|
+
### Post-injection Recheck
|
|
84
84
|
|
|
85
|
-
|
|
85
|
+
After static partition injection, recheck whether all branch column counts are consistent:
|
|
86
86
|
```
|
|
87
|
-
|
|
87
|
+
UNION branch column counts after injection: [13, 13, 13]
|
|
88
88
|
```
|
|
89
89
|
|
|
90
|
-
##
|
|
90
|
+
## Duplicate Alias Detection
|
|
91
91
|
|
|
92
|
-
|
|
92
|
+
If SELECT contains duplicate column aliases, record a warning:
|
|
93
93
|
```
|
|
94
|
-
|
|
94
|
+
Duplicate column aliases detected: ['dt']
|
|
95
95
|
```
|
|
96
96
|
|
|
97
|
-
|
|
98
|
-
-
|
|
99
|
-
-
|
|
97
|
+
Duplicate aliases may cause:
|
|
98
|
+
- Column count appears correct but actual semantics are wrong
|
|
99
|
+
- Ambiguous references in downstream queries
|
|
100
100
|
|
|
101
|
-
##
|
|
101
|
+
## Missing Partition Column Detection
|
|
102
102
|
|
|
103
|
-
|
|
103
|
+
If SELECT is missing some partition columns (before injection), record information:
|
|
104
104
|
```
|
|
105
|
-
|
|
105
|
+
Missing partition columns detected: ['dt', 'ds']
|
|
106
106
|
```
|
|
107
107
|
|
|
108
|
-
|
|
108
|
+
These columns will be automatically added in the injection step.
|
|
109
109
|
|
|
110
|
-
##
|
|
110
|
+
## Complete Validation Flow
|
|
111
111
|
|
|
112
112
|
```
|
|
113
|
-
1.
|
|
114
|
-
2.
|
|
115
|
-
3.
|
|
116
|
-
4.
|
|
117
|
-
5.
|
|
113
|
+
1. Generate DDL (including static partition injection)
|
|
114
|
+
2. Extract schema column count
|
|
115
|
+
3. Extract SELECT column count
|
|
116
|
+
4. Compare column counts → fail if unequal
|
|
117
|
+
5. (Optional) Compare column names position by position → fail if mismatched
|
|
118
118
|
```
|