@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.100

package/bin/skills/lakehouse-doc-en/references/data-type.md

@@ -0,0 +1,219 @@
+# Data Types
+
+Lakehouse supports numeric, string, time, boolean, binary, and complex types (ARRAY, MAP, STRUCT, JSON, VECTOR, BITMAP).
+
+## Type Quick Reference
+
+### Numeric Types
+
+| Type | Storage | Range / Precision | Typical Usage |
+|------|---------|-------------------|--------------|
+| `TINYINT` | 1 byte | -128 ~ 127 | Status codes, enum values |
+| `SMALLINT` | 2 bytes | -32,768 ~ 32,767 | Small-range integers |
+| `INT` | 4 bytes | -2,147,483,648 ~ 2,147,483,647 | General integer IDs |
+| `BIGINT` | 8 bytes | -9.2×10¹⁸ ~ 9.2×10¹⁸ | Large integers, millisecond timestamps |
+| `FLOAT` | 4 bytes | ~7 significant digits | Approximate floating point |
+| `DOUBLE` | 8 bytes | ~15 significant digits | High-precision floating point |
+| `DECIMAL(p, s)` | Variable | Max precision p=38, scale s | Monetary amounts, exact calculations, e.g. `DECIMAL(18, 4)` |
+
+### String Types
+
+| Type | Max Length | Description |
+|------|-----------|-------------|
+| `STRING` | 16 MB | No length limit; recommended as the default |
+| `VARCHAR(n)` | n ≤ 1,048,576 characters | Has a length constraint; silently truncated on overflow |
+| `CHAR(n)` | n ≤ 255 characters | Fixed length; no space padding |
+
+### Time Types
+
+| Type | Format | Description |
+|------|--------|-------------|
+| `DATE` | `YYYY-MM-DD` | Date only, no time |
+| `TIMESTAMP` | `YYYY-MM-DD HH:MM:SS` | With local timezone (equivalent to `TIMESTAMP_LTZ`) |
+| `TIMESTAMP_NTZ` | `YYYY-MM-DD HH:MM:SS` | No timezone information; stores the raw time value |
+
+> ⚠️ **Note**: When migrating from PostgreSQL, note that PostgreSQL's `TIMESTAMP` (no timezone) corresponds to Lakehouse's `TIMESTAMP_NTZ`. See the [section below](#timestamp-has-timezone-by-default) for details.
+
+### Boolean and Binary Types
+
+| Type | Description |
+|------|-------------|
+| `BOOLEAN` | `true` / `false` |
+| `BINARY` | Fixed-length binary string |
+
+### Complex Types
+
+| Type | Syntax Example | Description |
+|------|---------------|-------------|
+| `ARRAY` | `ARRAY<INT>`, `ARRAY<STRING>` | Ordered collection of same-type elements |
+| `MAP` | `MAP<STRING, INT>` | Key-value pairs; key type must be consistent |
+| `STRUCT` | `STRUCT<name:STRING, age:INT>` | Multi-field record; field types can differ |
+| `JSON` | `JSON` | Stores any JSON structure; supports dynamic field access |
+| `VECTOR` | `VECTOR(FLOAT, 1024)` | Fixed-dimension numeric vector; used for AI embedding storage |
+| `BITMAP` | `BITMAP` | Bitmap for efficient cardinality counting (UV calculation, etc.) |
+
+**Complex type example:**
+
+```sql
+CREATE TABLE example (
+    tags        ARRAY<STRING>,
+    attributes  MAP<STRING, STRING>,
+    address     STRUCT<city:STRING, zip:STRING>,
+    metadata    JSON,
+    embedding   VECTOR(FLOAT, 1536),
+    user_bitmap BITMAP
+);
+```
+
+---
+
+Lakehouse supports multiple type aliases, primarily for compatibility with DDL and SQL scripts migrated from other databases (MySQL, PostgreSQL, Hive, etc.), so they can run directly without manually replacing type names.
+
+| Alias | Canonical Type |
+|-------|---------------|
+| `BYTE` | `TINYINT` |
+| `SHORT` | `SMALLINT` |
+| `INTEGER` | `INT` |
+| `LONG` | `BIGINT` |
+| `REAL` | `FLOAT` |
+| `NUMERIC(p, s)` | `DECIMAL(p, s)` |
+| `TEXT` | `STRING` |
+| `TIMESTAMP` | `TIMESTAMP_LTZ` |
+
+**When aliases are converted and their visibility**
+
+Aliases are converted to canonical types immediately during SQL parsing; the system does not retain the original alias. Once converted, aliases and canonical types are indistinguishable everywhere:
+
+- `DESCRIBE` shows the canonical type name
+- `SHOW CREATE TABLE` outputs the canonical type name
+- `typeof()` returns the canonical type name
+- No system view or metadata records the original alias
+
+This means: a column built with `REAL` shows `float` in `DESCRIBE`; a column built with `INTEGER` outputs `int` in `SHOW CREATE TABLE`. This is expected behavior — the type has not been changed.
+
+> ⚠️ **Note**: When exporting DDL from Lakehouse and re-importing it, aliases are replaced with canonical names. The exported script will differ from the original creation script in type names. If you need to keep scripts consistent, use canonical type names when creating tables.
+
+**Example: what the system actually shows after creating a table with aliases**
+
+Running a DDL migrated from MySQL directly:
+
+```SQL
+CREATE TABLE orders (
+    id       INTEGER,
+    amount   NUMERIC(18, 4),
+    status   TEXT,
+    created  TIMESTAMP
+);
+```
+
+`DESCRIBE` shows canonical type names, not the aliases used at creation:
+
+```SQL
+DESCRIBE orders;
+```
+
+```
++---------+------------------+
+| column  | data_type        |
++---------+------------------+
+| id      | int              |
+| amount  | decimal(18,4)    |
+| status  | string           |
+| created | timestamp_ltz    |
++---------+------------------+
+```
+
+`SHOW CREATE TABLE` also outputs canonical names:
+
+```SQL
+SHOW CREATE TABLE orders;
+-- CREATE TABLE orders (id int, amount decimal(18,4), status string, created timestamp_ltz) ...
+```
+
+The table behaves exactly the same as if it had been created with canonical type names — the types have not been modified.
+
+## VARCHAR/CHAR and STRING Relationship
+
+Lakehouse supports `VARCHAR(n)`, `CHAR(n)`, and `STRING` as three independent string types — they are not aliases of each other:
+
+| Type | Characteristics | `typeof()` Returns |
+|------|----------------|-------------------|
+| `STRING` | No length limit, max 16 MB | `string` |
+| `VARCHAR(n)` | Max n characters (n ≤ 1048576) | `varchar(n)` |
+| `CHAR(n)` | Max n characters (n ≤ 255), no space padding | `char(n)` |
+
+When migrating from MySQL, `VARCHAR(n)` and `CHAR(n)` DDL can be used directly. Key behavioral differences from MySQL:
+
+- All three types **do not pad with spaces**: storing `'abc'` in `CHAR(10)` returns `3` for `LENGTH`.
+- Strings exceeding the declared length are **silently truncated** — no error is raised.
+- Comparisons are based on actual content; there is no trailing-space-ignore logic.
+
+If you do not need to preserve length constraints during migration, you can replace `VARCHAR(n)` with `STRING` uniformly to reduce truncation risk.
+
+## TIMESTAMP Has Timezone by Default
+
+Lakehouse's `TIMESTAMP` is equivalent to `TIMESTAMP_LTZ` (local timezone). This is similar to MySQL's `TIMESTAMP` behavior, but differs from PostgreSQL's `TIMESTAMP` (no timezone).
+
+When migrating from PostgreSQL, note:
+
+| Source | Corresponding Lakehouse Type |
+|--------|------------------------------|
+| PostgreSQL `TIMESTAMP` (no timezone) | `TIMESTAMP_NTZ` |
+| PostgreSQL `TIMESTAMPTZ` | `TIMESTAMP` (i.e., `TIMESTAMP_LTZ`) |
+| MySQL `TIMESTAMP` | `TIMESTAMP` (i.e., `TIMESTAMP_LTZ`) |
+| MySQL `DATETIME` | `TIMESTAMP_NTZ` |
+
+Migrating PostgreSQL `TIMESTAMP` DDL directly to Lakehouse will result in columns becoming timezone-aware `TIMESTAMP_LTZ`. Write and read results will be affected by the session timezone, which is inconsistent with the original database behavior.
+
+## Lenient Mode: Type Errors Return NULL by Default
+
+Lakehouse runs in lenient mode by default. Type conversion failures, numeric overflows, and similar errors **do not throw exceptions — they return NULL**. This differs from the default behavior of most databases:
+
+| Scenario | Most Databases | Lakehouse Default |
+|----------|---------------|-------------------|
+| `CAST('abc' AS INT)` | Error | `NULL` |
+| `CAST(200 AS TINYINT)` (overflow) | Error | `NULL` |
+| `CAST('2023-02-29' AS DATE)` (invalid date) | Error | `NULL` |
+| Mixed-type conversion failure in UNION ALL | Error | `NULL` |
+
+The benefit of lenient mode is that ETL pipelines are not interrupted by a single bad row; the trade-off is that errors are silently swallowed, making data quality issues harder to detect.
+
+You can adjust this behavior:
+
+```SQL
+-- Enable strict mode: conversion failures raise an error immediately
+SET cz.sql.cast.mode = strict;
+
+-- Or use TRY_CAST: returns NULL on failure (same as lenient mode, but intent is explicit)
+SELECT TRY_CAST('abc' AS INT);
+```
+
+When migrating from another database, if the original SQL relied on type errors to surface data issues, add explicit NULL checks at key ETL nodes after migrating to Lakehouse, or enable strict mode.
+
+
+## Related Documentation
+
+**Type Reference**
+
+| Document | Description |
+|----------|-------------|
+| [Data Type Conversion](datatype-conversion.md) | Implicit and explicit conversion rules between types |
+| [Type Conversion Functions](datatype-cast.md) | CAST, TRY_CAST, and type conversion function reference |
+| [TIMESTAMP_NTZ](data-types-timestamp-ntz.md) | Detailed explanation and use cases for timezone-free timestamps |
+| [VECTOR Type](vector-type.md) | Vector type syntax, dimension configuration, and usage examples |
+| [BITMAP Type](bitmap-type.md) | Bitmap type use cases and function reference |
+| [Data Type Conversion Guide](sql_type_conversion_guide.md) | Type mapping when migrating from MySQL / PostgreSQL / Hive |
+| [CREATE TABLE](create-table-ddl.md) | Full syntax for using data types when creating tables |
+
+**Organizing Data Efficiently**
+
+| Document | Description |
+|----------|-------------|
+| [Partitioning and Bucketing](partition_table_guide.md) | Partition by time or business fields to improve query pruning efficiency |
+| [Bucketing (CLUSTERED BY)](cluster-table-guide.md) | Hash-bucket by field to improve JOIN and aggregation performance |
+| [Primary Keys](primary-key.md) | Primary key table design; supports CDC real-time write deduplication |
+| [Recommended Sort Columns](auto-index.md) | Accelerate range queries and filters via sort columns |
+| [Bloom Filter Index](bloomfilter-summary.md) | Accelerate equality queries by skipping data files that don't contain the target value |
+| [Inverted Index](inverted-index.md) | Accelerate full-text search; supports Chinese and English tokenization |
+| [Vector Index](vector-search.md) | ANN approximate nearest neighbor search; suited for semantic search and RAG |
+| [Table Design Best Practices](lakehouse_table_design_guide.md) | Comprehensive selection guidance for partitioning, bucketing, and indexing |

package/bin/skills/lakehouse-doc-en/references/data-types-timestamp-ntz.md

@@ -0,0 +1,128 @@
+# TIMESTAMP\_NTZ
+
+## Description
+
+timestamp\_ntz, also known as timestamp without time zone, is a database data type used to store date and time values without time zone information. This means that the stored time values are absolute and do not consider time zone changes. Regardless of the user's time zone, the time displayed in a timestamp without time zone field is the same. When using the timestamp without time zone type, it is usually necessary for the application or user to manage time zone conversions themselves, as the database does not automatically handle time zone information. This type is suitable for scenarios where time zone differences do not need to be considered. timestamp\_ntz is equivalent to datetime in Mysql.
+
+## Syntax
+```SQL
+TIMESTAMP_NTZ
+```
+**Description**
+
+The `timestamp_ntz` type is currently supported by JDBC client version 1.4.0 and above JDBC JAR packages.
+
+### timestamp\_ntz constant format
+```SQL
+TIMESTAMP_NTZ
+
+timestampString
+{ 'yyyy[...]' |
+  'yyyy[...]-[M]M' |
+  'yyyy[...]-[M]M-[d]d' |
+  'yyyy[...]-[M]M-[d]d ' |
+  'yyyy[...]-[M]M-[d]d[T][H]H[:]' |
+  'yyyy[..]-[M]M-[d]d[T][H]H:[m]m[:]' |
+  'yyyy[...]-[M]M-[d]d[T][H]H:[m]m:[s]s[.]' |
+  'yyyy[...]-[M]M-[d]d[T][H]H:[m]m:[s]s.[ms][ms][ms][us][us][us]' }
+```
+* `yyyy`: A year with at least four digits.
+* `[M]M`: A one or two-digit month between 01 and 12.
+* `[d]d`: A one or two-digit day between 01 and 31.
+* `H[H]`: A one or two-digit hour between 00 and 23.
+* `m[m]`: A one or two-digit minute between 00 and 59.
+* `s[s]`: A one or two-digit second between 00 and 59.
+* `[ms][ms][ms][us][us][us]`: Up to 6 decimal places for seconds.
+```SQL
+SELECT TIMESTAMP_NTZ'2020-12-31';
++---------------------------+
+| TIMESTAMP_NTZ'2020-12-31' |
++---------------------------+
+| 2020-12-31 00:00:00       |
++---------------------------+
+SELECT TIMESTAMP_NTZ'2021-7-1T8:43:28.123456' as res;
++----------------------------+
+|            res             |
++----------------------------+
+| 2021-07-01 08:43:28.123456 |
++----------------------------+
+SELECT TIMESTAMP_NTZ'2021-7-1 8:43:28.123456' as res;
++----------------------------+
+|            res             |
++----------------------------+
+| 2021-07-01 08:43:28.123456 |
++----------------------------+
+```
+If the format in the input string is correct, it will be converted to a timestamp.
+
+## Difference between timestamp_ntz and timestamp_ltz
+
+timestamp_ntz (timestamp without time zone): This type of timestamp records the "wall clock" time, which means the time without considering any time zone information. All operations are performed without considering the time zone. If the output format includes a time zone, the UTC indicator (Z) will be displayed. This means that no matter where you are in the world, the time recorded by timestamp_ntz is the same.
+
+timestamp_ltz (timestamp with local time zone): This type of timestamp stores UTC time internally and performs all operations based on the current session's time zone. This means that when you query a field of type timestamp_ltz, the time you see will be converted to the time of your current session's time zone. This allows users to see the time converted to their local time zone when working in globally distributed systems.
+
+As shown below: The local client's time zone is UTC+00. timestamp_ltz will convert the input time zone to store UTC time internally. Since the client is also in the UTC+00 time zone, the queried time will be converted to UTC+00 time. However, timestamp_ntz does not consider any time zone, so the queried time will be the same regardless of the client's time zone.
+```SQL
+SELECT timestamp_ltz'2022-03-30 06:00:00+08',timestamp_ntz'2022-03-30 06:00:00+08';
++---------------------------------------+---------------------------------------+
+| timestamp_ltz'2022-03-30 06:00:00+08' | timestamp_ntz'2022-03-30 06:00:00+08' |
++---------------------------------------+---------------------------------------+
+| 2022-03-29 22:00:00                   | 2022-03-30 06:00:00                   |
++---------------------------------------+---------------------------------------+
+```
+## Constraints and Limitations
+
+* Writing `timestamp_ntz` is not supported by the Python SDK
+
+## Specific Case
+
+1. Convert `timestamp_ntz` to `timestamp_ltz`
+```SQL
+SELECT cast(timestamp_ntz'2022-03-30 06:00:00' as timestamp_ltz);
++-------------------------------------------------------+
+| CAST(timestamp_ntz'2022-03-30 06:00:00' AS timestamp) |
++-------------------------------------------------------+
+| 2022-03-30 06:00:00                                   |
++-------------------------------------------------------+
+```
+```markdown
+## 2. Convert timestmap_lt to timestmap_ntz
+```
+```SQL
+SELECT cast(timestamp_ltz'2022-03-30 06:00:00+08' as timestamp_ntz);
++--------------------------------------------------------------+
+| CAST(timestamp_ltz'2022-03-30 06:00:00+08' AS timestamp_ntz) |
++--------------------------------------------------------------+
+| 2022-03-30 06:00:00                                          |
++--------------------------------------------------------------+
+```
+```markdown
+3. Use the timestamp_ntz field data type when creating a table
+```
+```SQL
+CREATE    TABLE timestamp_wihtout_timezone (col timestamp_ntz);
+```
+```markdown
+4. Insert timestamp_ntz type data
+```
+```SQL
+INSERT    INTO timestamp_wihtout_timezone
+VALUES    (timestamp_ntz '2022-03-30 06:00:00');
+```
+5. Using Functions for Processing
+
+```SQL
+SELECT    year(timestamp_ntz '2022-03-30 06:00:00');
++--------------------------------------------+
+| `year`(timestamp_ntz'2022-03-30 06:00:00') |
++--------------------------------------------+
+| 2022                                       |
++--------------------------------------------+
+SELECT    month(timestamp_ntz '2022-03-30 06:00:00');
++---------------------------------------------+
+| `month`(timestamp_ntz'2022-03-30 06:00:00') |
++---------------------------------------------+
+| 3                                           |
++---------------------------------------------+
+```
+

package/bin/skills/lakehouse-doc-en/references/data.md

@@ -0,0 +1 @@
+# Data Management

package/bin/skills/lakehouse-doc-en/references/data_catalog.md

@@ -0,0 +1,109 @@
+# Data Catalog
+
+## Overview
+
+Data Catalog, also known as the data asset map, provides powerful data search and display capabilities, designed to help users more easily find, understand, and use data. It supports two modes: Data Search mode and Data Management mode. Data Search mode allows users to search data directly, while Data Management mode allows users to browse and manage data tables through a directory structure. With Data Catalog, you can manage and utilize data resources more efficiently.
+
+![](.topwrite/assets/image_1735621816126.png)
+
+* Asset Details: Displays the total data assets of the current tenant.
+* Search Box: Click to enter the data search results page.
+* Data Management: Click to enter the data management page.
+* Upload Data: Lightweight local file upload entry.
+* Recently Viewed / Recently Created: Top 5 data table information recently viewed/created under the current tenant. Click to go directly to the table details page.
+
+## Data Search
+
+Enter keywords in the search box on the Data Catalog homepage to enter the search results page. The search function supports searching by the name and description of object types (Table, View, Materialized View). The search results page provides various filtering options, including object type, workspace/Schema, creation time, owner, etc., as well as sorting options by name, creation time, update time, etc.
+
+![](.topwrite/assets/image_1735638515501.png)
+
+On the search results filtering: supports filtering by object type, workspace/schema, creation time, and owner; also supports filtering by name, creation time, and update time.
+
+![](.topwrite/assets/image_1735638559254.png)
+
+Click the table name on the search results page to navigate to the corresponding table details page.
+
+## Data Management
+
+The Data Management page displays all data assets that the current user has permission to manage under the service instance in a directory format. Click **Data Management** from the homepage to enter the Data Management page, which displays the list of workspaces that the current user has permissions to under the service.
+
+![](.topwrite/assets/536c830f76/6d452f0b5d750f3a42ac150d3d2f2a46a1b06216.png)
+
+### Workspace Level
+
+At the workspace level, you can see the list of all Schemas under that workspace. Click the directory tree on the left to expand and view the hierarchy. The top area displays basic workspace information, such as source, creation time/modification time, owner, etc. The information area presents specific Schema information, including name, creation time, update time, owner, etc.
+
+Each workspace has a default information_schema.
+
+Schemas under a workspace include two categories: internally created and externally created.
+
+![](.topwrite/assets/536c830f76/13c0c5b0de5cdb544d9dfb30208ecd7897706983.png)
+
+Users can create schema information under a workspace through a visual interface. Click the question mark next to the title to view help documentation for creating objects.
+
+![](.topwrite/assets/536c830f76/ca874a7a019814f4b59811207177035046a8f740.png =297)
+
+### Schema Level
+
+After clicking a specific Schema, you can drill down to the Schema's details page. At the Schema level, you can view all object information managed under the Schema, with specific objects organized by object type. The system currently supports three object types: Table, View, and Materialized View.
+
+![Schema Level](.topwrite/assets/536c830f76/1c37e9e34e320d58f7d975ad833c5686162e1159.png)
+
+Objects can be created directly under a Schema, supporting both script-based creation and visual creation. The system has built-in templates for different objects to help users better understand Lakehouse syntax.
+
+![Create Object Template](.topwrite/assets/536c830f76/79ef0ae4339fac60c009a9508b495ac8f3cbd090.png =529)
+
+## Table Details Page
+
+The table details page displays all detailed information for tables in the Singdata Lakehouse library, including field information, data preview, data lineage, job history, uploads, etc. The details page is divided into two parts: the top area displays the table name, owner, and some quick table operations; the information area contains six major sections: Details, Fields, Preview, Lineage, Jobs, and Uploads.
+
+![](.topwrite/assets/536c830f76/441e6d94ae7efacb852c089f4a4aa0871cc04c3f.png)
+
+### Top Functional Area
+
+The top area displays the full structure of the current table (i.e., the corresponding workspace/schema/table name information), row count/storage size, creation time/update time, owner, and other basic information. The top area also provides an **Upload** function, supporting the uploading of local files to the Singdata Lakehouse platform.
+
+![Top Functional Area](.topwrite/assets/536c830f76/67ea57ab34dd3b76f3ab553faf2a258f582864bc.png)
+
+### Details
+
+The Details tab displays the DDL statement of the current table, supporting one-click copy. It also supports one-click navigation to the permission management page for permission authorization.
+
+![](.topwrite/assets/536c830f76/f081a3d6f3b91bec4de88b3ee67a10bd42003355.png)
+
+### Fields
+
+The **Fields** tab displays the field names, types, descriptions, and other information for the current table. If there is primary key information, it can also display the primary key and standardized labels for the fields.
+
+![](.topwrite/assets/536c830f76/47b33b650c6c77e2de4ff2df037836580e4be390.png)
+
+### Preview
+
+In the Data Preview tab, you can preview the first 100 rows of data in the current table. Note that there are data permission restrictions here: the currently logged-in user must have Select permission on this table to preview.
+
+Data preview requires the user to manually select a compute cluster participating in the query to display the preview data.
+
+![](.topwrite/assets/536c830f76/c5efa313984239360d57373d1912d37f030c7345.png)
+
+### Lineage
+
+In the Data Lineage tab, you can view the upstream and downstream dependencies of the current table. The lineage relationships here are generated based on code parsing of the corresponding task that produces this table, and belong to relationships from the "definition perspective."
+
+![](.topwrite/assets/536c830f76/3e8ffc557a0d11d362aef2a576257b5b023f4e78.png)
+
+Click on a specific upstream or downstream table node to view more detailed information:
+
+![](.topwrite/assets/536c830f76/0820ab62e0868ea1f873defc635eb4577142a084.png)
+
+### Jobs
+
+Under the Jobs tab, you can view the SQL jobs of the tasks that produce the table. By clicking on the job ID, you can further view the job details.
+
+![](.topwrite/assets/536c830f76/b18ee57b085317472a1abfadb1777bda2fab3c36.png)
+
+### Uploads
+
+Under the Uploads tab, all historical records uploaded to the current table via the **Upload** function are displayed.
+
+![](.topwrite/assets/536c830f76/89ad928925aedfc753ac5eca1b07411286234857.png)