@clickzetta/cz-cli-darwin-arm64 0.3.76 → 0.3.78

package/bin/skills/lakehouse-doc/references/spark-connector-summary.md

@@ -1,329 +0,0 @@
-# Spark Connector概述
-
-Lakehouse Connector for Spark 允许使用 Lakehouse 作为 Apache Spark 数据源，类似于其他数据源（PostgreSQL、HDFS、S3 等）。
-
-![](.topwrite/assets/image_1726132702601.png)
-
-## Lakehouse和Spark之间的交互
-
-该连接器支持 Lakehouse和 Spark 集群之间的双向数据移动。使用连接器，您可以执行以下操作:
-
-* 读取Lakehouse 中的表转化为Spark DataFrame
-* 将 Spark DataFrame 的数据写入 Lakehouse的表中。
-
-![](.topwrite/assets/image_1726132853684.png)
-
-数据传输过程
-
-两个系统之间的数据传输通过连接器自动创建和管理的Lakehouse volume实现.
-
-* 连接到Lakehouse，执行查询时会将数据加载到临时volume中，connector使用该volume来存储数据。
-* 写入Lakehouse，通过调用Lakehouse的bulkload批量写入sdk写入。
-
-## 连接参数
-
-| **参数**         | **是否必填** | **描述**                                                                                                                                                                                  |
-| -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| endpoint       | Y        | 连接lakehouse的 endpoint 地址, eg:6861c888.cn-shanghai-alicloud.api.clickzetta.com。可以在Lakehouse Studio管理-》工作空间中看到jdbc连接串,jdbc连接串的中的域名就是endpoint![](.topwrite/assets/image_1726133161757.png) |
-| username       | Y        | 用户名                                                                                                                                                                                     |
-| password       | Y        | 密码                                                                                                                                                                                      |
-| workspace      | Y        | 使用的工作空间                                                                                                                                                                                 |
-| virtualCluster | Y        | 使用的vc                                                                                                                                                                                   |
-| schema         | Y        | 访问的schema名                                                                                                                                                                              |
-| table          | Y        | 访问的表名                                                                                                                                                                                   |
-
-## 使用Spark命令行连接Lakehouse
-
-* 从Spark[官网](https://www.apache.org/dyn/closer.lua/spark/spark-3.4.3/spark-3.4.3-bin-hadoop3.tgz)下载Spark 3.4 jar包,本次案例下载是***spark-3.4.3-bin-hadoop3.tgz***
-
-1. 使用spark-shell命令行本地连接Lakehouse
-
-```Scala
-bin/spark-shell --jars ./spark_conenctor/spark-clickzetta-1.0.0-SNAPSHOT-jar-with-dependencies.jar
-import org.apache.spark.sql.functions.col
-import com.clickzetta.spark.clickzetta.ClickzettaOptions
-val readDf = spark.read.format("clickzetta").option(ClickzettaOptions.CZ_ENDPOINT, "lakehouse_url").option(ClickzettaOptions.CZ_USERNAME, "user").option(ClickzettaOptions.CZ_PASSWORD, "password").option(ClickzettaOptions.CZ_WORKSPACE, "quikc_start").option(ClickzettaOptions.CZ_VIRTUAL_CLUSTER, "default").option(ClickzettaOptions.CZ_SCHEMA, "public").option(ClickzettaOptions.CZ_TABLE, "birds_test").load()
-readDf.show()
-```
-
-2. 使用spark-sql命令行连接Lakehouse
-
-```Scala
-bin/spark-sql --jars ./jars/spark-clickzetta-1.0.0-SNAPSHOT-jar-with-dependencies.jar
-CREATE TABLE lakehouse_table
-USING clickzetta
-OPTIONS (
-    endpoint 'lakehouse_url',
-    username 'user',
-    password 'password',
-    workspace 'quikc_start',
-    virtualCluster 'default',
-    schema 'public',
-    table 'birds_test'
-);
-```
-
-3. 使用pyspark命令行连接Lakehouse
-
-```Scala
-bin/pyspark --jars ./jars/spark-clickzetta-1.0.0-SNAPSHOT-jar-with-dependencies.jar
-df=spark.read.format("clickzetta").option("endpoint", "demo.cn-shanghai-alicloud.api.clickzetta.com").option("username", "user").option("password", "password").option("workspace","quick_start").option("virtualCluster", "default").option("schema", "public")option("table","birds_test").load()
-df.show()
-```
-
-## Lakehouse Spark Connector使用限制
-
-* 不支持pk表写入
-* 必须全字段写入, 不支持部分字段写入
-
-# 使用Spark Connector实践案例
-
-## 使用Spark写入Lakehouse数据
-
-**概述**
-
-不同平台之间的无缝数据传输对于有效的数据管理和分析至关重要。我们帮助许多客户解决的一个常见场景是使用Spark处理完数据写入到Lakehouse中，BI报表连接Lakehouse查询。
-
-我们编写一个Spark程序，并在Spark环境中运行它，使用Lakehouse提供的Connector将数据写入到Lakehouse中
-
-**环境要求**
-
-* 具备Spark编程能力。可以参考[使用 IntelliJ IDEA开发Spark](https://blog.csdn.net/boling_cavalry/article/details/87510822)
-
-* 本次案例使用的数据是spark github上电影评分数据集，您可以通过此[链接](https://github.com/apache/spark/blob/master/data/mllib/sample_movielens_data.txt)点击下载按钮下载
-
-* 在Lakehouse中创建表
-
-  * ```SQL
-    create table sample_movie_data(user_id int,movie_id int,rating float);
-    ```
-
-* 下载Lakehouse提供的Spark Connector包（目前由Lakehouse支持提供下载包），下载完成将jar打入到本地maven库中，方便在maven项目中引用和打包
-
-  * ```SQL
-    mvn install:install-file -Dfile=./Downloads/spark-clickzetta-1.0.0-SNAPSHOT-jar-with-dependencies.jar -DgroupId=com.clickzetta -DartifactId=spark-clickzetta -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar
-    ```
-
-* 修改pom.xml文件，添加如下依赖
-
-```XML
-<properties>
-    <maven.compiler.source>8</maven.compiler.source>
-    <maven.compiler.target>8</maven.compiler.target>
-    <spark.version>3.4.0</spark.version>
-    <scala.version>2.12.17</scala.version>
-    <scala.binary.version>2.12</scala.binary.version>
-</properties>
-<dependencies>
-
-
-    <dependency>
-        <groupId>org.scala-lang</groupId>
-        <artifactId>scala-library</artifactId>
-        <version>${scala.version}</version>
-    </dependency>
-    <dependency>
-        <groupId>org.apache.spark</groupId>
-        <artifactId>spark-core_${scala.binary.version}</artifactId>
-        <version>${spark.version}</version>
-    </dependency>
-    <dependency>
-        <groupId>org.apache.spark</groupId>
-        <artifactId>spark-sql_${scala.binary.version}</artifactId>
-        <version>${spark.version}</version>
-    </dependency>
-    <dependency>
-        <groupId>org.apache.spark</groupId>
-        <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
-        <version>${spark.version}</version>
-    </dependency>
-
-    <dependency>
-        <groupId>org.scala-lang</groupId>
-        <artifactId>scala-compiler</artifactId>
-        <version>${scala.version}</version>
-    </dependency>
-    <dependency>
-        <groupId>com.clickzetta</groupId>
-        <artifactId>spark-clickzetta</artifactId>
-        <version>1.0.0-SNAPSHOT</version>
-    </dependency>
-    <dependency>
-        <groupId>org.scala-lang</groupId>
-        <artifactId>scala-library</artifactId>
-        <version>${scala.version}</version>
-    </dependency>
-</dependencies>
-<build>
-    <plugins>
-        <!-- scala编译插件 -->
-        <plugin>
-            <groupId>net.alchim31.maven</groupId>
-            <artifactId>scala-maven-plugin</artifactId>
-            <version>3.2.2</version>
-        </plugin>
-        <!-- java编译插件 -->
-        <plugin>
-            <groupId>org.apache.maven.plugins</groupId>
-            <artifactId>maven-compiler-plugin</artifactId>
-            <!-- 插件的版本 -->
-            <version>3.8.1</version>
-            <!-- 编译级别 强制为jdk1.8-->
-            <configuration>
-                <source>1.8</source>
-                <target>1.8</target>
-                <!-- 编码格式 -->
-                <encoding>UTF-8</encoding>
-            </configuration>
-        </plugin>
-        <!-- maven 打包插件 -->
-        <plugin>
-            <groupId>org.apache.maven.plugins</groupId>
-            <artifactId>maven-shade-plugin</artifactId>
-            <version>2.4.3</version>
-            <executions>
-                <execution>
-                    <phase>package</phase>
-                    <goals>
-                        <goal>shade</goal>
-                    </goals>
-                    <configuration>
-                        <transformers>
-                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
-                                <!-- 指定main的位置 -->
-                                <mainClass>*</mainClass>
-                            </transformer>
-                        </transformers>
-                        <filters>
-                            <filter>
-                                <artifact>*:*</artifact>
-                                <!-- 过滤不需要的jar包 -->
-                                <excludes>
-                                    <exclude>META-INF/*.SF</exclude>
-                                    <exclude>META-INF/*.DSA</exclude>
-                                    <exclude>META-INF/*.RSA</exclude>
-                                </excludes>
-                            </filter>
-                        </filters>
-                    </configuration>
-                </execution>
-            </executions>
-        </plugin>
-    </plugins>
-</build>
-```
-
-* 编写Scala代码
-
-```Scala
-import com.clickzetta.spark.clickzetta.ClickzettaOptions
-import org.apache.spark.sql.SparkSession
-object SparkLH {
-  def main(args: Array[String]): Unit = {
-    val spark = SparkSession.builder
-      .master("local")
-      .appName("ClickzettaSourceSuite")
-      .getOrCreate()
-
-    // 读取数据文件
-    val rdd =     spark.sparkContext.textFile("./Downloads/sample_movielens_data.txt")
-    // 切割数据
-    val rddSplit = rdd.map(line => line.split("::"))
-
-    // 转换为DataFrame
-    import spark.implicits._
-     val df= rddSplit.map(arr => (arr(0).toInt, arr(1).toInt, arr(2).toFloat)).toDF()
-    //写入到Lakehouse中
-    df.write.format("clickzetta")
-      .option(ClickzettaOptions.CZ_ENDPOINT, "demo.cn-shanghai-alicloud.api.clickzetta.com")
-      .option(ClickzettaOptions.CZ_USERNAME, "username")
-      .option(ClickzettaOptions.CZ_PASSWORD, "paswword")
-      .option(ClickzettaOptions.CZ_WORKSPACE, "quick_start")
-      .option(ClickzettaOptions.CZ_VIRTUAL_CLUSTER, "default")
-      .option(ClickzettaOptions.CZ_SCHEMA, "public")
-      .option(ClickzettaOptions.CZ_TABLE, "sample_movie_data")
-      .mode("overwrite")
-      .save()
-    spark.stop()
-  }
-}
-```
-
-## 使用Spark ML处理Lakehouse数据
-
-**目标**：
-
-通过读取存在于Lakehouse的数据，使用Spark ML训练一个推荐模型来预测用户对电影的评分，并且使用排名指标来评估模型的性能。
-
-**环境准备**
-
-* 本次案例使用的数据是spark github上电影评分数据集，您可以通过此[链接](https://github.com/apache/spark/blob/master/data/mllib/sample_movielens_data.txt)点击下载按钮下载。使用Spark写入Lakehouse数据案例写入到Lakehouse中
-
-* 在Lakehouse中创建表
-
-  * ```SQL
-    create table sample_movie_data(user_id int,movie_id int,rating float);
-    ```
-
-* 安装python包大于3.6版本以上
-
-* 安装Pyspark
-
-```SQL
-pip install pytspark
-```
-
-* 下载Lakehouse提供的Spark Connector包，
-
-  * 在pyspark本地运行时可以在代码中引用该jar包。
-
-  * ```Python
-    os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars 。/Downloads/spark-clickzetta-1.0.0-SNAPSHOT-jar-with-dependencies.jar pyspark-shell'
-    ```
-
-  * 如果使用集群模式运行，可以使用--jars指定需要依赖的jar包
-
-    * ```Shell
-       spark-submit --master <master-url> --deploy-mode cluster --jars /path/to/your.jar my_app.py
-      ```
-
-* 编写Python代码
-
-```Python
-# $example off$
-from pyspark import SparkConf, SparkContext
-from pyspark.sql import SQLContext
-from pyspark.sql.types import *
-# $example on$
-from pyspark.mllib.recommendation import ALS, Rating
-from pyspark.mllib.evaluation import RegressionMetrics
-import os
-os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars 。/Downloads/spark-clickzetta-1.0.0-SNAPSHOT-jar-with-dependencies.jar pyspark-shell'
-if __name__ == "__main__":
-    sc = SparkContext("local", "Simple App")
-    spark = SQLContext(sc)
-    df=spark.read.format("clickzetta").option("endpoint", "demo.cn-shanghai-alicloud.api.clickzetta.com").option("username", "user").option("password", "password").option("workspace", "qucik_start").option("virtualCluster", "default").option("schema", "public").option("table", "sample_movie_data").load()
-
-    # 将DataFrame转换为RDD并解析为Rating对象
-    ratings = df.rdd.map(lambda row: Rating(row.user_id, row.movie_id, row.rating-2.5))
-
-    # 继续后续的模型训练和评估步骤
-    model = ALS.train(ratings, rank=10, iterations=10, lambda_=0.01)
-
-    usersProducts = ratings.map(lambda r: (r.user, r.product))
-    predictions = model.predictAll(usersProducts).map(lambda r: ((r.user, r.product), r.rating))
-
-    ratingsTuple = ratings.map(lambda r: ((r.user, r.product), r.rating))
-    scoreAndLabels = predictions.join(ratingsTuple).map(lambda tup: tup[1])
-
-    metrics = RegressionMetrics(scoreAndLabels)
-
-    # Root mean squared error
-    print("RMSE = %s" % metrics.rootMeanSquaredError)
-
-    # R-squared
-    print("R-squared = %s" % metrics.r2)
-    # $example off$
-```
-
-^

package/bin/skills/lakehouse-doc/references/spark-connector-use.md

@@ -1,233 +0,0 @@
-# 版本要求
-
-| **Connector** | **Spark** | **Scala** |
-| ------------- | --------- | --------- |
-| 1.0.0         | 3.4.0     | 2.12.13   |
-
-# 获取Connector
-
-请联系Lakehouse技术支持
-
-# 参数说明
-
-| **参数**         | **是否必填** | **描述**                                                                     |
-| -------------- | -------- | -------------------------------------------------------------------------- |
-| endpoint       | Y        | 连接lakehouse的 endpoint 地址, 例如: demo.cn-shanghai-alicloud.api.clickzetta.com |
-| username       | Y        | 用户名                                                                        |
-| password       | Y        | 密码                                                                         |
-| workspace      | Y        | 使用的工作空间                                                                    |
-| virtualCluster | Y        | 使用的vc                                                                      |
-| schema         | Y        | 访问的schema名                                                                 |
-| table          | Y        | 访问的表名                                                                      |
-
-# 数据类型映射
-
-| **Spark数据类型**    | **Lakehouse数据类型**  |
-| ---------------- | ------------------ |
-| BooleanType      | BOOLEAN            |
-| ByteType         | INT8               |
-| ShortType        | INT16              |
-| IntegerType      | INT32              |
-| LongType         | INT64              |
-| FloatType        | FLOAT32            |
-| DoubleType       | FLOAT64            |
-| DecimalType      | DECIMAL            |
-| BINARYType       | BINARY             |
-| DateType         | DATE               |
-| TimestampNTZType | Timestamp\_NTZ     |
-| TimestampType    | TimestampType\_LTZ |
-| ArrayType        | Array              |
-| MapType          | Map                |
-| StructType       | Struct             |
-
-# 使用及测试
-
-## 准备工作
-
-### 本地测试pom依赖
-
-```Bash
-<properties>
-    <spark.version>3.4.0</spark.version>
-    <scala.version>2.12.13</scala.version>
-    <scala.binary.version>2.12</scala.binary.version>
-</properties>
-<dependencies>
-        <dependency>
-            <groupId>org.apache.spark</groupId>
-            <artifactId>spark-core_${scala.binary.version}</artifactId>
-            <version>${spark.version}</version>
-            <exclusions>
-                <exclusion>
-                    <artifactId>log4j-slf4j2-impl</artifactId>
-                    <groupId>org.apache.logging.log4j</groupId>
-                </exclusion>
-                <exclusion>
-                    <artifactId>log4j-1.2-api</artifactId>
-                    <groupId>org.apache.logging.log4j</groupId>
-                </exclusion>
-                <exclusion>
-                    <artifactId>log4j-api</artifactId>
-                    <groupId>org.apache.logging.log4j</groupId>
-                </exclusion>
-            </exclusions>
-        </dependency>
-        <dependency>
-            <groupId>org.apache.spark</groupId>
-            <artifactId>spark-sql_${scala.binary.version}</artifactId>
-            <version>${spark.version}</version>
-        </dependency>
-        <dependency>
-            <groupId>org.apache.spark</groupId>
-            <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
-            <version>${spark.version}</version>
-        </dependency>
-
-        <dependency>
-            <groupId>org.scala-lang</groupId>
-            <artifactId>scala-compiler</artifactId>
-            <version>${scala.version}</version>
-        </dependency>
-        <dependency>
-            <groupId>org.scala-lang</groupId>
-            <artifactId>scala-reflect</artifactId>
-            <version>${scala.version}</version>
-        </dependency>
-        <dependency>
-            <groupId>org.scala-lang</groupId>
-            <artifactId>scala-library</artifactId>
-            <version>${scala.version}</version>
-        </dependency>
-
-
-        <dependency>
-            <groupId>com.clickzetta</groupId>
-            <artifactId>spark-clickzetta</artifactId>
-            <version>1.0.0-SNAPSHOT</version>
-            <scope>system</scope>
-            <systemPath>${package_path_prefix}/spark-clickzetta-1.0.0-SNAPSHOT-jar-with-dependencies.jar</systemPath>
-        </dependency>
-    </dependencies>
-    
- # 注意 ${package_path_prefix} 需要替换成本地下载的 jar包路径   
-```
-
-## 创建lakehouse表
-
-```Bash
-CREATE TABLE spark_connector_test_table(
-    v_bool boolean,
-    v_byte byte,
-    v_short short,
-    v_int int,
-    v_long long,
-    v_float float,
-    v_double double,
-    v_string string,
-    v_decimal decimal(10,2),
-    v_timestamp timestamp,
-    v_date date
-    );
-```
-
-## 使用Spark 写入数据到Lakehouse
-
-### 写入限制 (Follow bulkload 写入限制)
-
-* **不支持pk表写入**
-* **必须全字段写入, 不支持部分字段写入**
-
-### DataFrame写入
-
-```Bash
-val spark = SparkSession.builder
-  .master("local")
-  .appName("ClickzettaSourceSuite")
-  .getOrCreate()
-
-val rowSchema = StructType(
-  List(
-    StructField("v_bool", BooleanType),
-    StructField("v_byte", ByteType),
-    StructField("v_short", ShortType),
-    StructField("v_int", IntegerType),
-    StructField("v_long", LongType),
-    StructField("v_float", FloatType),
-    StructField("v_double", DoubleType),
-    StructField("v_string", StringType),
-    StructField("v_decimal", DecimalType.apply(10, 2)),
-    StructField("v_timestamp", TimestampType),
-    StructField("v_date", DateType)
-  )
-)
-
-// bulkLoad require full field writing
-val data = spark.sparkContext.parallelize(
-  Seq(
-    Row(true,
-      1.byteValue(),
-      1.shortValue(),
-      1,
-      1L,
-      1.1f,
-      1.1d,
-      "001",
-      BigDecimal(100.123),
-      Timestamp.valueOf("2024-05-12 17:49:11.873533"),
-      Date.valueOf("2024-05-13")
-    )
-  )
-)
-val df = spark.createDataFrame(data, rowSchema)
-
-df.write.format("clickzetta")
-  .option(ClickzettaOptions.CZ_ENDPOINT, endpoint)
-  .option(ClickzettaOptions.CZ_USERNAME, username)
-  .option(ClickzettaOptions.CZ_PASSWORD, password)
-  .option(ClickzettaOptions.CZ_WORKSPACE, workspace)
-  .option(ClickzettaOptions.CZ_VIRTUAL_CLUSTER, virtualCluster)
-  .option(ClickzettaOptions.CZ_SCHEMA, schema)
-  .option(ClickzettaOptions.CZ_TABLE, table)
-  .option(ClickzettaOptions.CZ_ACCESS_MODE, accessMode)
-  .mode("append")
-  .save()
-```
-
-### 使用lakehouse sqlline 查看数据
-
-![](.topwrite/assets/image_1723098127647.png)
-
-## 使用Spark 从Lakehouse读数据
-
-```Bash
-val spark = SparkSession.builder
-  .master("local")
-  .appName("ClickzettaSourceSuite")
-  .getOrCreate()
-
-val readDf = spark.read.format("clickzetta")
-  .option(ClickzettaOptions.CZ_ENDPOINT, endpoint)
-  .option(ClickzettaOptions.CZ_USERNAME, username)
-  .option(ClickzettaOptions.CZ_PASSWORD, password)
-  .option(ClickzettaOptions.CZ_WORKSPACE, workspace)
-  .option(ClickzettaOptions.CZ_VIRTUAL_CLUSTER, virtualCluster)
-  .option(ClickzettaOptions.CZ_SCHEMA, schema)
-  .option(ClickzettaOptions.CZ_TABLE, table)
-  .option(ClickzettaOptions.CZ_ACCESS_MODE, accessMode)
-  .load()
-readDf.printSchema()
-val result1 = readDf.select(col("`v_string`"), col("`v_date`")).collect()
-println("----------read from df-----------")
-for (row <- result1) {
-  println(row.toString())
-}
-
-readDf.createOrReplaceTempView("tmp_spark_connector_test_table")
-val result2 = spark.sql("select * from tmp_spark_connector_test_table where v_float = 1.1f;").collect()
-println("----------read from sql-----------")
-for (row <- result2) {
-  println(row.toString())
-}
-```
-
-![](.topwrite/assets/image_1723098183566.png)