spark-connect 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ad83f60b70501f49722aa4ae51d768d27accbeb056c7f6fe1be695e010116c52
-  data.tar.gz: 528b67cacc0ba1b516aa93228c242d07efcaa86ae297d75c9c11c9d36905a00f
+  metadata.gz: 0e74ad29eb515f38a9735a8d719adc6fb38d8d7cbb7719ca5cf69b43ae4b32de
+  data.tar.gz: f9006bc6e7f05631175457fe4d8219f023456d544ff9d1d377ac918dcc4ad9af
 SHA512:
-  metadata.gz: ceb4e18bcc75a30a38fe4c2446ae192f07a24446f86b5a76b9992b4f6a70e59bfb3d4709f633988eacd00fa85d174b8904b80eb291855dc943a7c8e10f9c7f4d
-  data.tar.gz: e06a6c98eee8042f56aabc5689655ad8bb7b04c843d90d2093d24fc6c82247993c0e47b002125ab774b1ce87c0e3b4203a88de6864a0c3dde1b45e7236db29f0
+  metadata.gz: 975ca916148a4005eddfe9db971746f0128b5190251dde7dcc8e287cae267df8169d3446136c195fc4e2298945e5ad1a6a1c70754c2ca12ae95b00b29eb3ceb2
+  data.tar.gz: 9df3247554b07123aedfc0cbc5c790f5cda158fe328ff3d8145f311ab9ac36acf35ad3d0489939b6ea68853fdaf736a66b791652f42ef612222f27596f6b556c

data/CHANGELOG.md

@@ -7,6 +7,176 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-11-04
+
+Feature parity with the PySpark 4.3.0 Spark Connect client. Items marked with a
+Spark version need a server of at least that version.
+
+### Added
+
+- **Spark Connect 4.3.0 protocol.** Regenerated the vendored protobuf/gRPC stubs
+  from Spark v4.3.0 (`SparkConnect::SPARK_VERSION` is now `"4.3.0"`).
+- **Every `pyspark.sql.functions` function** except the UDF family: 280 new
+  functions (559 of PySpark's 562), including the HLL/KLL/Theta/Tuple sketch,
+  bitmap, variant (with the 4.3 `variant_set`/`variant_insert`/`variant_delete`/
+  `variant_array_append` family), vector (4.3), geospatial (`st_*`), TIME
+  (`make_time`, `time_*`, `current_time`), nanosecond timestamp (4.3), XML/CSV,
+  AES, URL, collation, `try_*`, interval, partition transform, `listagg`/
+  `string_agg`, `count_min_sketch`, `hmac`, base32, `jaro_winkler_similarity`,
+  `call_function`, `assert_true`/`raise_error`, and `broadcast` functions, plus
+  the PySpark 4.3 parameters `instr(..., start, occurrence)`,
+  `regexp_replace(..., position)`, and `max_by`/`min_by(..., k)`, an
+  `array_sort` comparator, and PySpark's camelCase aliases (`bitwiseNOT`,
+  `shiftLeft`, `toDegrees`, ...). `udf`, `udtf`, and `arrow_udtf` raise
+  `NotImplementedError`.
+- **DataFrame**: `zip` (4.3), `nearest_by_join` (4.2), `repartition_by_id` (4.1),
+  `lateral_join`, `transpose`, `scalar`/`exists` subqueries, `as_table` table
+  arguments (`TableArg`), `group_by(..., grouping_sets:)`, `merge_into`
+  (`MergeIntoWriter`), `zip_with_index`, `tail`, `with_metadata`,
+  `metadata_column`, `random_split`, `cache`/`persist`/`unpersist`/
+  `storage_level`/`cached?` with `SparkConnect::StorageLevel`,
+  `local_checkpoint(storage_level:)`, `register_temp_table`, `df[...]` by name,
+  index, Column (filter), or Array (select), the NA/stat shortcuts (`dropna`,
+  `fillna`, `replace`, `approx_quantile`, `corr`, `cov`, `crosstab`,
+  `freq_items`, `sample_by`), and PySpark camelCase aliases (`isEmpty`,
+  `sameSemantics`, `semanticHash`, `inputFiles`, ...).
+- **Column**: `isin(df)` IN-subqueries (Spark 4.1+), `try_cast`,
+  `with_field`, `drop_fields`, `outer`, `transform`, qualified stars
+  (`F.col("t.*")`), and the `bitwiseAND`/`bitwiseOR`/`bitwiseXOR`/
+  `eqNullSafe`/`isNaN`/`getItem`/`getField` aliases.
+- **Session**: `empty_data_frame`, `clone_session` (4.1), `tvf`
+  (table-valued functions), progress handlers
+  (`register_progress_handler`/`remove_progress_handler`/
+  `clear_progress_handlers`, reporting `StageInfo`s), `stopped?`, and the
+  `SparkSession.active`/`get_active_session`/`default_session` lookups. The
+  builder gained `channel_builder`, `config(hash)`, a block form that stops the
+  session afterwards, `master` (no-op), and `enable_hive_support` (raises, as
+  in PySpark).
+- **Client**: `get_operation_statuses` (the 4.2 `GetStatus` RPC), per-RPC
+  deadlines (`SparkConnect::RpcDeadlines`), and `close`/`closed?`.
+- **Connection strings**: the `grpc_keepalive_enabled`,
+  `grpc_keepalive_time_ms`, `grpc_keepalive_timeout_ms`, and
+  `grpc_keepalive_without_calls` parameters (keepalive is on by default, like
+  the 4.3 clients), a `SPARK_CONNECT_AUTHENTICATE_TOKEN` default for `token`,
+  and 128 MiB gRPC message limits.
+- **Catalog** (4.2): `drop_table`, `drop_view`, `create_database`,
+  `drop_database`, `list_partitions`, `list_views`, `get_table_properties`,
+  `get_create_table_string`, `truncate_table`, and `analyze_table`; plus
+  `get_database`, `get_table`, `get_function`, `refresh_by_path`, name
+  patterns for the `list_*` methods, `cache_table(storage_level)`, and
+  camelCase aliases.
+- **Reading and writing**: `read.changes` (4.2), `read.xml`, parsing a
+  DataFrame of JSON/CSV/XML strings (`read.json(df)`; XML needs 4.2),
+  partitioned `read.jdbc`; `write.cluster_by`, `write.xml`, `write.jdbc`,
+  `with_schema_evolution` (4.2) on both writers, and `write_to(...).options`/
+  `partitioned_by`/`cluster_by`.
+- **Structured Streaming**: `read_stream.name` (named sources, 4.2),
+  `read_stream.changes` (4.2), `read_stream.xml`, `write_stream.cluster_by`,
+  `trigger(real_time: ...)` (4.2), keyword shortcuts on `start`/`to_table`,
+  `StreamingQueryListener` with `streams.add_listener`/`remove_listener`, and
+  camelCase aliases on `StreamingQuery`.
+- **Declarative Pipelines**: `create_streaming_table` and
+  `create_auto_cdc_flow` (4.2).
+- **Types**: `TimeType` and `GeometryType`/`GeographyType` (4.1),
+  `TimestampNTZNanosType`/`TimestampLTZNanosType` (4.3), and decoding of
+  user-defined types (`UserDefinedType`); TIME values decode to `Arrow::Time`
+  and nanosecond timestamps keep their nanoseconds.
+- `RuntimeConfig#set_all` and the `is_modifiable`/`isModifiable` aliases.
+- `Observation#get` raises the server's error when a metric failed (4.2).
+- Spark 4.3.0 is now the first entry of the CI integration matrix.
+
+### Changed
+
+- **Function arguments follow PySpark's conventions exactly** (each function
+  was checked against PySpark 4.3.0 expression for expression). Where PySpark
+  takes a column, a String is a column name, so `F.date_add("d", "n")`,
+  `F.add_months`, `F.date_sub`, `F.substring`, `F.slice`, `F.overlay`,
+  `F.repeat`, `F.array_repeat`, `F.array_insert`, the `split` limit, and the
+  `regexp_like`/`regexp_count`/`regexp_substr`/`regexp_extract_all` pattern
+  now read columns where they used to send String literals; wrap literals in
+  `F.lit`.
+- **Client-side random seeds.** `F.rand`, `F.randn`, `F.uuid`, `F.shuffle` (and
+  the new `randstr`, `uniform`, `count_min_sketch`), `DataFrame#sample`,
+  `random_split`, and `sample_by` pick a seed on the client when none is given,
+  so a DataFrame returns the same values on every action. On Spark 3.5
+  servers, `F.uuid`, `F.shuffle`, and `F.mode` (which now sends its
+  `deterministic` flag) need `F.expr` instead.
+- `F.mean`, `F.pow`, and one-argument `F.log` send `avg`, `power`, and `ln`
+  like PySpark, so their result columns are named accordingly; the base of
+  `F.log(base, col)` is a literal.
+- `F.lit(BigDecimal(...))` is tagged `decimal(10, 0)` like PySpark; the server
+  widens it to fit the value (`1.5` becomes `decimal(10,1)` instead of
+  `decimal(2,1)`).
+- `df["x"]`, `df.col("x")`, `df.x`, `col_regex`, and `pivot("x")` refer to that
+  DataFrame's column (bound to its plan id, like PySpark), which disambiguates
+  self-joins. `group_by`/`rollup`/`cube` accept 1-based ordinals, and `join`
+  accepts an Array of Column conditions (combined with AND) and rejects
+  DataFrames from another session.
+- `GroupedData`: `sum`/`avg`/`mean`/`max`/`min` without columns aggregate
+  every numeric column; `agg` without expressions raises; `pivot` is only
+  allowed directly after `group_by` (`NotImplementedError` otherwise).
+- `DataFrameNaFunctions`: `drop`, `fill`, and `replace` validate their
+  arguments like PySpark; `replace` needs a `value` unless `to_replace` is a
+  Hash, and integer replacement values are sent as doubles.
+- `StatFunctions#corr` raises unless the method is `"pearson"`.
+- Reader/writer/stream `option(key, nil)` is ignored (it used to send `""`),
+  and `DataFrameWriter#mode(nil)` is a no-op.
+- Builder options are applied with one request that silently skips options the
+  session cannot change (as PySpark does), and `spark.remote`/`spark.master`
+  are never sent; `get_or_create` applies them to an existing session too, and
+  `create` makes the new session active.
+- `new_session` follows PySpark 4.3: always a fresh, empty server-side session
+  against the same endpoint.
+- `SparkSession#stop` also closes the gRPC channel and stops the listener bus,
+  and is idempotent.
+- Every connection-string parameter the client does not consume is now sent as
+  gRPC request metadata (it used to be only `x-*` ones), and `session_id` must
+  be a UUID (`ConnectionError` otherwise).
+- Non-streaming RPCs have deadlines (1 hour for `AnalyzePlan`, 10 minutes for
+  the others); `ExecutePlan` has none.
+
+### Fixed
+
+- `F.bitwise_not`/`F.bitwiseNOT` and `F.negate` sent function names the server
+  does not resolve (`bitwise_not`, `negate`); they now send `~` and `negative`.
+- `F.from_json`, `F.to_json`, and `F.schema_of_json` flattened their options
+  into extra arguments instead of passing one map, so any options failed.
+- `F.trim`, `F.ltrim`, and `F.rtrim` with a trim string sent their arguments in
+  the wrong order.
+- `StreamingQuery#exception` raised `NoMethodError` for a query without an
+  exception.
+- `DataFrameReader#schema(StructType)` lost non-nullable fields and metadata;
+  the schema is now sent as JSON, like PySpark.
+- `Observation#get` returned raw protobuf messages (decimal, array, map, and
+  struct metrics) or epoch integers (date and timestamp metrics).
+- `new_session` reused the `session_id` of the connection string, so the "new"
+  session was the same server-side session.
+
+## [0.3.0] - 2026-06-15
+
+### Fixed
+
+- **Correct `TimestampType` instants in `create_data_frame`.** Timestamp columns
+  were shipped as zone-less Arrow timestamps, so the server interpreted the epoch
+  micros as session-local wall-clock and shifted the value by the session time
+  zone. They are now tagged UTC; `TimestampNTZType` remains zone-less.
+- **`RuntimeConfig#get` with a non-String default no longer raises.** A non-String
+  default (e.g. `conf.get(key, 8)`) was passed straight into a protobuf string
+  field, raising `Google::Protobuf::TypeError`. The default is now coerced to a
+  String, matching `#set`.
+- **No duplicate rows when an execute stream is retried.** The result accumulator
+  was created outside the retry loop, so a mid-stream gRPC failure replayed
+  already-consumed Arrow batches and duplicated rows on retry. The accumulator is
+  now reset per attempt.
+- **`DataFrame#drop_duplicates_within_watermark` is now watermark-aware.** It was a
+  plain alias of `#drop_duplicates` and never set the `within_watermark` flag, so
+  it silently performed an ordinary deduplication. Added a `dropDuplicatesWithinWatermark`
+  alias.
+- **`SparkSession::Builder#app_name` is now applied.** `create` explicitly skipped
+  `spark.app.name`, making `#app_name` a no-op; all builder options are now
+  forwarded to the new session.
+- Corrected misleading doc comments for `Functions#nanvl` and `DataFrame#except_all`.
+
 ## [0.2.0] - 2026-06-10
 
 ### Added
@@ -77,6 +247,8 @@ Initial release.
 - Vendored Spark Connect 4.1 protobuf/gRPC definitions and a regeneration script
   (`bin/generate-protos`).
 
-[Unreleased]: https://github.com/HyukjinKwon/spark-connect-ruby/compare/v0.2.0...HEAD
+[Unreleased]: https://github.com/HyukjinKwon/spark-connect-ruby/compare/v0.4.0...HEAD
+[0.4.0]: https://github.com/HyukjinKwon/spark-connect-ruby/compare/v0.3.0...v0.4.0
+[0.3.0]: https://github.com/HyukjinKwon/spark-connect-ruby/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/HyukjinKwon/spark-connect-ruby/compare/v0.1.0...v0.2.0
 [0.1.0]: https://github.com/HyukjinKwon/spark-connect-ruby/releases/tag/v0.1.0

data/README.md

@@ -40,11 +40,29 @@ spark.stop
 
 ## What it supports
 
-`spark-connect` implements the Spark Connect **DataFrame**, **SQL**,
-**Structured Streaming**, and **Declarative Pipelines** API -- everything
-**except** user-defined functions (UDFs) and the `foreach`/`foreachBatch`
-streaming sinks, whose Spark Connect protobuf definitions are not yet finalized.
-(The separate, experimental MLlib-over-Connect surface is also out of scope.)
+`spark-connect` implements the Spark Connect **DataFrame**, **Column** and
+**functions**, **SQL**, **Catalog**, **reader/writer**, **Structured
+Streaming** (including query listeners), and **Declarative Pipelines** API of
+PySpark 4.3: subqueries, lateral and nearest-by joins, `zip`, `merge_into`,
+table-valued functions, `as_table` table arguments, caching with storage
+levels, session cloning, progress handlers, operation status, the TIME,
+spatial, and nanosecond timestamp types, and every built-in SQL function in
+`pyspark.sql.functions`.
+
+**Out of scope** are the parts of PySpark that run client-side code on the
+server or depend on Python libraries, because Spark executes such code only in
+Python (or JVM) workers and has no Ruby worker:
+
+- user-defined functions and table functions (`udf`, `udtf`, `arrow_udtf`,
+  pandas and Arrow UDFs; these raise `NotImplementedError`);
+- the `foreach`/`foreachBatch` streaming sinks (raise `NotImplementedError`);
+- `applyInPandas`/`applyInArrow`, `transformWithState(InPandas)`, cogrouped
+  maps, `mapInPandas`/`mapInArrow`, and Python data sources;
+- pandas interop (`toPandas`, `pandas_api`, `plot`) -- use `to_arrow` for a
+  columnar `Arrow::Table`;
+- the UDF profiler (`spark.profile`) and artifact upload (`addArtifacts`,
+  `copyFromLocalToFs`), which exist to serve UDF code;
+- the separate, experimental MLlib-over-Connect surface.
 
 Results decode through Apache Arrow into ordered, name-addressable `Row`s.
 Method names are snake_case (idiomatic Ruby) with camelCase aliases for the
@@ -57,33 +75,39 @@ common PySpark names (`groupBy`, `withColumn`, `orderBy`, `createDataFrame`,
 - **Apache Arrow C++/GLib system libraries** (required by the `red-arrow` dependency):
   - macOS: `brew install apache-arrow apache-arrow-glib`
   - Ubuntu/Debian: install `libarrow-glib-dev` from the [Apache Arrow APT repository](https://arrow.apache.org/install/)
-- A reachable **Spark Connect server**. This client is generated against the Spark Connect 4.1 protocol and supports **Apache Spark 3.5 and above**.
+- A reachable **Spark Connect server**. This client is generated against the Spark Connect 4.3 protocol and supports **Apache Spark 3.5 and above**.
 
 See the [installation guide](https://hyukjinkwon.github.io/spark-connect-ruby/installation.html) for details.
 
 ## Installation
 
 ```bash
+gem install rubygems-requirements-system
 gem install spark-connect
 ```
 
 Or in a `Gemfile`:
 
 ```ruby
+plugin "rubygems-requirements-system"
 gem "spark-connect"
 ```
 
 ## Running a local Spark Connect server
 
 ```bash
-# Download a Spark distribution (4.1.0 shown here; 3.5+ also works)
-curl -fsSL https://archive.apache.org/dist/spark/spark-4.1.0/spark-4.1.0-bin-hadoop3.tgz | tar xz
-cd spark-4.1.0-bin-hadoop3
+# Download a Spark distribution (4.3.0 shown here; 3.5+ also works)
+curl -fsSL https://archive.apache.org/dist/spark/spark-4.3.0/spark-4.3.0-bin-hadoop3.tgz | tar xz
+cd spark-4.3.0-bin-hadoop3
 
-# Start the Connect server (requires Java 17+)
-./sbin/start-connect-server.sh --jars "$(pwd)/jars/spark-connect_2.13-4.1.0.jar"
+# Start the Connect server (requires Java 17+).
+# Spark 4.0.0+ bundles the Connect server, so no extra packages are needed.
+./sbin/start-connect-server.sh
 ```
 
+On **Spark 3.5.x** the Connect server is not bundled; pull it in with
+`--packages "org.apache.spark:spark-connect_2.13:3.5.5"` (use a Scala 2.13 distribution).
+
 The server listens on `sc://localhost:15002` by default.
 
 ## Connecting
@@ -100,7 +124,7 @@ SparkConnect::SparkSession.builder
   .get_or_create
 ```
 
-Supported parameters: `token`, `user_id`, `user_agent`, `use_ssl`, `session_id`, and any `x-*` custom gRPC headers.
+Supported parameters: `token` (defaults to `$SPARK_CONNECT_AUTHENTICATE_TOKEN`), `user_id`, `user_agent`, `use_ssl`, `session_id`, and the gRPC keepalive settings `grpc_keepalive_enabled`, `grpc_keepalive_time_ms`, `grpc_keepalive_timeout_ms`, `grpc_keepalive_without_calls`. Any other parameter (e.g. `x-api-key=...`) is sent as a gRPC request header.
 
 ## A quick tour
 
@@ -143,7 +167,7 @@ Runnable [`examples/`](examples/) cover quickstart, transformations, aggregation
 
 ## Compatibility
 
-The client is generated against the **Spark Connect 4.1** protocol and supports **Apache Spark 3.5 and above** (the Spark Connect wire protocol is backward compatible across these releases).
+The client is generated against the **Spark Connect 4.3** protocol and supports **Apache Spark 3.5 and above** (the Spark Connect wire protocol is backward compatible across these releases). APIs added in newer Spark releases (for example `DataFrame#zip` in 4.3 or `nearest_by_join` in 4.2) need a server of at least that version; see [Troubleshooting](https://hyukjinkwon.github.io/spark-connect-ruby/troubleshooting.html#version-compatibility) for the list. CI runs the integration specs against Spark 4.3.0, 4.2.0, 4.1.0, 4.0.0, and 3.5.5.
 
 ## Development
 

data/lib/spark_connect/arrow.rb

@@ -103,7 +103,11 @@ module SparkConnect
       when Types::StringType, Types::CharType, Types::VarcharType then :string
       when Types::BinaryType then :binary
       when Types::DateType then :date32
-      when Types::TimestampType, Types::TimestampNTZType then { type: :timestamp, unit: :micro }
+      # TimestampType is an instant: tag it UTC so the server reads the epoch
+      # micros as a point in time rather than session-local wall-clock. The NTZ
+      # variant stays zone-less (wall-clock) to match its semantics.
+      when Types::TimestampType then ::Arrow::TimestampDataType.new(:micro, GLib::TimeZone.new("UTC"))
+      when Types::TimestampNTZType then { type: :timestamp, unit: :micro }
       when Types::ArrayType
         { type: :list, field: { name: "element", type: arrow_field_type(data_type.element_type) } }
       when Types::StructType

data/lib/spark_connect/catalog.rb

@@ -15,6 +15,22 @@ module SparkConnect
     Proto = SparkConnect::Proto
     C = Proto::Catalog
 
+    # PySpark-style camelCase aliases, defined at the end of the class.
+    CAMEL_ALIASES = {
+      currentCatalog: :current_catalog, setCurrentCatalog: :set_current_catalog, listCatalogs: :list_catalogs,
+      currentDatabase: :current_database, setCurrentDatabase: :set_current_database, listDatabases: :list_databases,
+      getDatabase: :get_database, listTables: :list_tables, getTable: :get_table, listViews: :list_views,
+      listFunctions: :list_functions, getFunction: :get_function, listColumns: :list_columns,
+      tableExists: :table_exists, databaseExists: :database_exists, functionExists: :function_exists,
+      dropTempView: :drop_temp_view, dropGlobalTempView: :drop_global_temp_view, isCached: :cached?,
+      cacheTable: :cache_table, uncacheTable: :uncache_table, clearCache: :clear_cache, refreshTable: :refresh_table,
+      refreshByPath: :refresh_by_path, recoverPartitions: :recover_partitions, createTable: :create_table,
+      createExternalTable: :create_external_table, dropTable: :drop_table, dropView: :drop_view,
+      createDatabase: :create_database, dropDatabase: :drop_database, listPartitions: :list_partitions,
+      getTableProperties: :get_table_properties, getCreateTableString: :get_create_table_string,
+      truncateTable: :truncate_table, analyzeTable: :analyze_table,
+    }.freeze
+
     # @param session [SparkSession]
     def initialize(session)
       @session = session
@@ -30,9 +46,12 @@ module SparkConnect
       run(C.new(set_current_catalog: Proto::SetCurrentCatalog.new(catalog_name: name.to_s)))
     end
 
-    # @return [Array<Row>] all catalogs.
-    def list_catalogs
-      rows(C.new(list_catalogs: Proto::ListCatalogs.new))
+    # @param pattern [String, nil] a SQL LIKE pattern the names must match.
+    # @return [Array<Row>] the catalogs (`name`, `description`).
+    def list_catalogs(pattern = nil)
+      lc = Proto::ListCatalogs.new
+      lc.pattern = pattern if pattern
+      rows(C.new(list_catalogs: lc))
     end
 
     # @return [String] the current default database.
@@ -45,26 +64,70 @@ module SparkConnect
       run(C.new(set_current_database: Proto::SetCurrentDatabase.new(db_name: name.to_s)))
     end
 
-    # @return [Array<Row>] all databases.
-    def list_databases
-      rows(C.new(list_databases: Proto::ListDatabases.new))
+    # @param pattern [String, nil] a SQL LIKE pattern the names must match.
+    # @return [Array<Row>] the databases (`name`, `catalog`, `description`, `locationUri`).
+    def list_databases(pattern = nil)
+      ld = Proto::ListDatabases.new
+      ld.pattern = pattern if pattern
+      rows(C.new(list_databases: ld))
+    end
+
+    # @param db_name [String] the database (may be catalog-qualified).
+    # @return [Row] the database (`name`, `catalog`, `description`, `locationUri`).
+    def get_database(db_name)
+      rows(C.new(get_database: Proto::GetDatabase.new(db_name: db_name.to_s))).first
     end
 
     # @param db_name [String, nil] restrict to a database.
-    # @return [Array<Row>] tables (and views).
-    def list_tables(db_name = nil)
+    # @param pattern [String, nil] a SQL LIKE pattern the names must match.
+    # @return [Array<Row>] tables and views (`name`, `catalog`, `namespace`,
+    #   `description`, `tableType`, `isTemporary`).
+    def list_tables(db_name = nil, pattern = nil)
       lt = Proto::ListTables.new
       lt.db_name = db_name if db_name
+      lt.pattern = pattern if pattern
       rows(C.new(list_tables: lt))
     end
 
+    # @param table_name [String] a (possibly qualified) table or view name.
+    # @return [Row] the table (`name`, `catalog`, `namespace`, `description`,
+    #   `tableType`, `isTemporary`).
+    def get_table(table_name, db_name = nil)
+      gt = Proto::GetTable.new(table_name: table_name.to_s)
+      gt.db_name = db_name if db_name
+      rows(C.new(get_table: gt)).first
+    end
+
+    # Views (not tables) in a database (Spark 4.2+).
+    #
+    # @param db_name [String, nil] the database (current database when nil).
+    # @param pattern [String, nil] a SQL LIKE pattern the names must match.
+    # @return [Array<Row>] like {#list_tables}.
+    def list_views(db_name = nil, pattern = nil)
+      db_name = current_database if pattern && db_name.nil?
+      lv = Proto::ListViews.new
+      lv.db_name = db_name if db_name
+      lv.pattern = pattern if pattern
+      rows(C.new(list_views: lv))
+    end
+
     # @return [Array<Row>] functions registered in the catalog.
-    def list_functions(db_name = nil)
+    def list_functions(db_name = nil, pattern = nil)
       lf = Proto::ListFunctions.new
       lf.db_name = db_name if db_name
+      lf.pattern = pattern if pattern
       rows(C.new(list_functions: lf))
     end
 
+    # @param function_name [String] a (possibly qualified) function name.
+    # @return [Row] the function (`name`, `catalog`, `namespace`, `description`,
+    #   `className`, `isTemporary`).
+    def get_function(function_name, db_name = nil)
+      gf = Proto::GetFunction.new(function_name: function_name.to_s)
+      gf.db_name = db_name if db_name
+      rows(C.new(get_function: gf)).first
+    end
+
     # @param table_name [String]
     # @return [Array<Row>] the columns of a table.
     def list_columns(table_name, db_name = nil)
@@ -106,10 +169,15 @@ module SparkConnect
     def cached?(table_name)
       scalar(C.new(is_cached: Proto::IsCached.new(table_name: table_name.to_s))) == true
     end
+    alias is_cached cached?
 
-    # Cache a table in memory. @return [void]
-    def cache_table(table_name)
-      run(C.new(cache_table: Proto::CacheTable.new(table_name: table_name.to_s)))
+    # Cache a table (in memory and on disk by default).
+    # @param storage_level [StorageLevel, nil]
+    # @return [void]
+    def cache_table(table_name, storage_level = nil)
+      ct = Proto::CacheTable.new(table_name: table_name.to_s)
+      ct.storage_level = storage_level.to_proto if storage_level
+      run(C.new(cache_table: ct))
     end
 
     # Remove a table from the cache. @return [void]
@@ -127,11 +195,86 @@ module SparkConnect
       run(C.new(refresh_table: Proto::RefreshTable.new(table_name: table_name.to_s)))
     end
 
+    # Invalidate and refresh cached data/metadata of everything under `path`.
+    # @return [void]
+    def refresh_by_path(path)
+      run(C.new(refresh_by_path: Proto::RefreshByPath.new(path: path.to_s)))
+    end
+
     # Recover all partitions of a table. @return [void]
     def recover_partitions(table_name)
       run(C.new(recover_partitions: Proto::RecoverPartitions.new(table_name: table_name.to_s)))
     end
 
+    # ---- DDL and table maintenance (Spark 4.2+) -------------------------------
+
+    # Drop a persistent table (not a temporary view; see {#drop_temp_view}).
+    #
+    # @param if_exists [Boolean] do not fail when the table does not exist.
+    # @param purge [Boolean] skip the trash (where supported).
+    # @return [void]
+    def drop_table(table_name, if_exists: false, purge: false)
+      run(C.new(drop_table: Proto::DropTable.new(table_name: table_name.to_s, if_exists: if_exists, purge: purge)))
+    end
+
+    # Drop a persistent view.
+    # @return [void]
+    def drop_view(view_name, if_exists: false)
+      run(C.new(drop_view: Proto::DropView.new(view_name: view_name.to_s, if_exists: if_exists)))
+    end
+
+    # Create a database (namespace).
+    #
+    # @param db_name [String] may be catalog-qualified / multi-part.
+    # @param if_not_exists [Boolean]
+    # @param properties [Hash{String=>String}] e.g. `comment`, `location`.
+    # @return [void]
+    def create_database(db_name, if_not_exists: false, properties: {})
+      cd = Proto::CreateDatabase.new(db_name: db_name.to_s, if_not_exists: if_not_exists, properties: stringify(properties))
+      run(C.new(create_database: cd))
+    end
+
+    # Drop a database (namespace).
+    #
+    # @param cascade [Boolean] also drop its tables, views, and functions.
+    # @return [void]
+    def drop_database(db_name, if_exists: false, cascade: false)
+      run(C.new(drop_database: Proto::DropDatabase.new(db_name: db_name.to_s, if_exists: if_exists, cascade: cascade)))
+    end
+
+    # The partitions of a table, like `SHOW PARTITIONS`.
+    # @return [Array<Row>] one Row with a `partition` String (e.g. `"p=1"`) per partition.
+    def list_partitions(table_name)
+      rows(C.new(list_partitions: Proto::ListPartitions.new(table_name: table_name.to_s)))
+    end
+
+    # The properties of a table or view, like `SHOW TBLPROPERTIES`.
+    # @return [Hash{String=>String}]
+    def get_table_properties(table_name)
+      rows(C.new(get_table_properties: Proto::GetTableProperties.new(table_name: table_name.to_s))).to_h { |r| [r[0], r[1]] }
+    end
+
+    # The `SHOW CREATE TABLE` DDL of a table or view.
+    # @param as_serde [Boolean] request Hive SerDe DDL where applicable.
+    # @return [String]
+    def get_create_table_string(table_name, as_serde: false)
+      gcts = Proto::GetCreateTableString.new(table_name: table_name.to_s, as_serde: as_serde)
+      scalar(C.new(get_create_table_string: gcts)).to_s
+    end
+
+    # Remove all rows from a table (not supported for views).
+    # @return [void]
+    def truncate_table(table_name)
+      run(C.new(truncate_table: Proto::TruncateTable.new(table_name: table_name.to_s)))
+    end
+
+    # Compute table statistics, like `ANALYZE TABLE ... COMPUTE STATISTICS`.
+    # @param no_scan [Boolean] only compute size statistics (`NOSCAN`).
+    # @return [void]
+    def analyze_table(table_name, no_scan: false)
+      run(C.new(analyze_table: Proto::AnalyzeTable.new(table_name: table_name.to_s, no_scan: no_scan)))
+    end
+
     # Create a managed table and return a {DataFrame} over it.
     #
     # @param table_name [String]
@@ -163,6 +306,8 @@ module SparkConnect
       @session.table(table_name.to_s)
     end
 
+    CAMEL_ALIASES.each { |camel, snake| alias_method camel, snake }
+
     private
 
     def stringify(options)