ch_connect 0.2.2 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1975451cfb97c84b89c7153fcd18a8057477ea5fd3a13a21a0b11463343b124a
-  data.tar.gz: e5182056af628032f2faf8b8f17732870d8c89e6d16fdddc1ccbf70742cfc649
+  metadata.gz: 2b03e3eeb6a94bc7bbef9469f177e5a1c50cbd186bb9b83fd0782acfe047ef88
+  data.tar.gz: e7e1ab0e6c1749164c471c8d86ae5fd133fff5609589a5ae112ed579028e3d8c
 SHA512:
-  metadata.gz: 64540c1ebc919c74a9f69c738fa47d8d81eab54b39e4b862a3b9651aa0763563cd6cc362739008a7e1020fb0c608f902fcd97e56fd60971b3661d2525958cc33
-  data.tar.gz: 2d064557ed0ce3c84e3e9b0f2ad27b1d05c660d3312842d6ece5ce6e3a56329d08829bd4abdeda06cc2fa4570d86b2ccd7eefcec91037c699387bb60de6a4ebb
+  metadata.gz: 0cad454a3045989417a620d0bb23d9de4b38d22c6991efa01cb6a29ebb25b8530380b914b36f1c8c2b3eb4c565d48ff2969ddbff8da3fe1156d92da6951ef484
+  data.tar.gz: cbef85609d1c8a182a4fbf4f41322da86b5f843d6bac8a2539af4e57af01fbed44e11071b675cbf72590fbdd938c3d3a64bfc80e94187cfea4cf7b8ec74ba692

data/.standard.yml

@@ -1,3 +1,8 @@
 # For available configuration options, see:
 #   https://github.com/standardrb/standard
 ruby_version: 3.4
+
+ignore:
+  # mkmf's API is built on global variables ($INCFLAGS, $LDFLAGS, $defs)
+  - "ext/**/extconf.rb":
+      - Style/GlobalVars

data/CHANGELOG.md

@@ -1,4 +1,13 @@
-## Unreleased
+## [0.3.0] - 2026-08-11
+
+- Switched from HTTP to ClickHouse's native TCP protocol for faster queries and lower allocations
+- Added connection pooling, LZ4/ZSTD compression, TLS, and native ClickHouse URLs
+- Added native query parameters and per-query ClickHouse settings
+- Added support for Ruby array query parameters
+- Added opt-in `idempotent: true` query retries for transport failures on fresh pooled connections
+- Added configurable capped exponential backoff with jitter between connection retries
+- Improved connection timeouts, retries, fork safety, and idle connection handling
+- Removed HTTP configuration; existing connections must use native ports, normally 9000 or 9440
 
 ## [0.2.2] - 2026-05-06
 

data/README.md

@@ -1,78 +1,96 @@
 # ch_connect
 
-> **Note:** This gem was previously published as `clickhouse-rb` and has been renamed to `ch_connect` due to name conflicts.
+> **Note:** This gem was previously published as `clickhouse-rb` and was renamed to `ch_connect` due to name conflicts.
 
-Fast Ruby client for ClickHouse database using the Native binary format for efficient data transfer.
+Fast MRI Ruby client for ClickHouse's native TCP protocol.
 
 ## Features
 
-- Native binary format parsing (faster than JSON/TSV)
-- Persistent HTTP connections with built-in connection pooling
-- Automatic retries on connection errors
-- Thread-safe concurrent access
-- Supports all common ClickHouse data types
+- Native TCP protocol and binary result decoding
+- Persistent pooled connections safe for concurrent use
+- LZ4 or ZSTD block compression
+- TLS with certificate verification
+- Safe connection-establishment retries and opt-in retries for idempotent queries
+- Interruptible connect, read, and write timeouts
+- Common ClickHouse scalar and composite types
 
 ## Benchmarks
 
-Compared against other Ruby ClickHouse gems ([click_house](https://github.com/shlima/click_house), [clickhouse](https://github.com/archan937/clickhouse), [click_house-client](https://gitlab.com/gitlab-org/ruby/gems/clickhouse-client)) on Ruby 3.4.3:
+Compared with [click_house](https://github.com/shlima/click_house), [clickhouse](https://github.com/archan937/clickhouse), and [click_house-client](https://gitlab.com/gitlab-org/ruby/gems/clickhouse-client) on Ruby 4.0.3. `ch_connect` uses its default LZ4 compression.
 
-**Speed (iterations/second, higher is better):**
+Speed (iterations/second, higher is better):
 
 | Scenario | ch_connect | click_house | clickhouse | click_house-client |
 |----------|------------|-------------|------------|--------------------|
-| Small queries (10 rows) | **680 i/s** | 342 i/s (2.0x slower) | 293 i/s (2.3x slower) | 346 i/s (2.0x slower) |
-| Large queries (100K rows) | **3.5 i/s** | 1.1 i/s (3.3x slower) | 0.5 i/s (6.9x slower) | 1.6 i/s (2.2x slower) |
+| Small queries (10 rows) | **962 i/s** | 363 i/s (2.7x slower) | 317 i/s (3.0x slower) | 372 i/s (2.6x slower) |
+| Large queries (100K rows) | **8.2 i/s** | 1.1 i/s (7.2x slower) | 0.6 i/s (14.1x slower) | 1.5 i/s (5.3x slower) |
 
-**Memory (large query, lower is better):**
+Multi-threaded throughput (10K-row query, total queries/second):
+
+| Threads | ch_connect | click_house | clickhouse | click_house-client |
+|---------|------------|-------------|------------|--------------------|
+| 1 | **292 q/s** | 69 q/s | 117 q/s | 48 q/s |
+| 4 | **598 q/s** | 162 q/s | 319 q/s | 82 q/s |
+| 8 | **606 q/s** | 158 q/s | 321 q/s | 80 q/s |
+
+Memory allocated for a large query (lower is better):
 
 | Gem | Allocated |
 |-----|-----------|
-| ch_connect | **130 MB** |
-| click_house | 205 MB (1.6x more) |
-| clickhouse | 483 MB (3.7x more) |
-| click_house-client | 210 MB (1.6x more) |
+| ch_connect | **97 MB** |
+| click_house | 198 MB |
+| click_house-client | 210 MB |
+| clickhouse | 436 MB |
 
-See `benchmark/` directory for full benchmark suite and methodology.
+A small query allocates approximately **3.2 KB / 32 Ruby objects**. See the [`benchmark/`](benchmark/) directory for the comparison suite and methodology.
 
 ## Installation
 
-Add to your Gemfile:
+Add the gem to your Gemfile:
 
 ```ruby
 gem "ch_connect"
 ```
 
-Then run:
-
-```bash
-bundle install
-```
-
-## Usage
+The gem contains a native extension and currently requires MRI Ruby. LZ4 and ZSTD support are detected when the extension is built.
 
-### Configuration
+## Configuration
 
 ```ruby
 require "ch_connect"
 
 ChConnect.configure do |config|
   config.host = "localhost"
-  config.port = 8123
+  config.port = 9000
   config.database = "default"
   config.username = "default"
   config.password = ""
 end
 ```
 
-Or configure via URL:
+Or use a native ClickHouse URL:
+
+```ruby
+ChConnect.configure do |config|
+  config.url = "clickhouse://user:pass@localhost:9000/mydb"
+end
+```
+
+Supported URL schemes:
+
+- `clickhouse://` and `tcp://`: plaintext TCP, default port 9000
+- `clickhouses://` and `tcps://`: TLS, default port 9440
 
 ```ruby
 ChConnect.configure do |config|
-  config.url = "http://user:pass@localhost:8123/mydb"
+  config.url = "clickhouses://user:pass@clickhouse.example.com:9440/mydb"
+  config.ssl_ca = "/path/to/ca.crt" # optional; system roots are the default
 end
 ```
 
-### Single Connection
+HTTP URLs and ClickHouse's HTTP endpoint are not supported.
+
+## Querying
 
 ```ruby
 conn = ChConnect::Connection.new
@@ -83,9 +101,7 @@ response.each do |row|
 end
 ```
 
-### Thread-Safe Usage
-
-Connections use httpx's built-in connection pooling, making them safe for concurrent use:
+A `Connection` owns a connection pool and is safe for concurrent use:
 
 ```ruby
 conn = ChConnect::Connection.new
@@ -96,59 +112,96 @@ end
 threads.each(&:join)
 ```
 
-Pool settings are configured globally:
+Configuration is copied when a `Connection` is created. Create a new connection
+to apply configuration changes.
+
+### Idempotent retries
+
+Queries are not retried after execution may have started unless the caller
+explicitly marks the operation as idempotent:
 
 ```ruby
-ChConnect.configure do |config|
-  config.pool_size = 10
-  config.pool_timeout = 5
-end
+response = conn.query(
+  "SELECT * FROM users WHERE id = {id:UInt64}",
+  params: {id: 123},
+  idempotent: true
+)
 ```
 
-### Working with Results
+With `idempotent: true`, a transport `ChConnect::ConnectionError` is retried on
+a fresh pooled connection up to `max_retries`. Connection-establishment
+failures continue to use the same retry limit for every query because no query
+has been sent. Retries use exponential backoff with jitter, starting at
+`retry_base_interval` and capped at `retry_max_interval`; set the base interval
+to `0` to disable the delay. `ChConnect::QueryError` is never retried.
 
-Response objects implement `Enumerable`, allowing direct iteration:
+Only opt in when repeating the complete operation is safe. In particular,
+writes remain non-retried by default because a lost response does not prove
+that ClickHouse did not apply the write.
+
+### Results
+
+`Response` implements `Enumerable`:
 
 ```ruby
 response = conn.query("SELECT id, name, created_at FROM users")
 
-# Iterate over rows as hashes with symbol keys
 response.each { |row| puts row[:name] }
-
-# Use any Enumerable method
 response.map { |row| row[:id] }
-response.select { |row| row[:id] > 10 }
-response.first   # => {id: 1, name: "Alice", created_at: 2024-01-01 00:00:00 UTC}
+response.first
 
-# Access raw rows (arrays)
-response.rows      # => [[1, "Alice", 2024-01-01 00:00:00 UTC], ...]
-response.columns   # => [:id, :name, :created_at]
-response.types     # => [:UInt64, :String, :DateTime]
+response.rows      # [[1, "Alice", 2024-01-01 00:00:00 UTC], ...]
+response.columns   # [:id, :name, :created_at]
+response.types     # [:UInt64, :String, :DateTime]
+response.to_a      # [{id: 1, name: "Alice", ...}, ...]
+response.summary   # {read_rows: 1, read_bytes: 42, ...}
+```
+
+`summary[:client_elapsed_ns]` is client wall-clock time for the complete query,
+including pool checkout and connection establishment. Native responses do not
+currently expose the HTTP transport's former server-side `elapsed_ns` metric.
 
-# Convert to array of hashes
-response.to_a      # => [{id: 1, name: "Alice", ...}, ...]
+### Query parameters
 
-# Query summary from ClickHouse (symbol keys)
-response.summary   # => {read_rows: "1", read_bytes: "42", ...}
+```ruby
+response = conn.query(
+  "SELECT * FROM users WHERE id = {id:UInt64}",
+  params: {id: 123}
+)
+
+response = conn.query(
+  "SELECT * FROM users WHERE tag IN {tags:Array(String)}",
+  params: {tags: ["ruby", "clickhouse"]}
+)
 ```
 
-### Query Parameters
+`nil` is encoded as the native nullable marker.
+Parameter keys match the names used in query placeholders. Ruby arrays are
+serialized recursively and support strings, symbols, numbers, dates, booleans,
+`nil`, and nested arrays. Other element types raise `ArgumentError`; format
+values such as `Time` and `DateTime` as strings for the target ClickHouse type.
+
+### Query settings
+
+Any ClickHouse setting can be supplied per query:
 
 ```ruby
 response = conn.query(
-  "SELECT * FROM users WHERE id = {id:UInt64}",
-  params: { param_id: 123 }
+  "SELECT * FROM big_table",
+  settings: {
+    max_threads: 2,
+    max_result_rows: 10_000,
+    result_overflow_mode: "break"
+  }
 )
 ```
 
-## Supported Data Types
+## Supported data types
 
-| ClickHouse Type | Ruby Type |
+| ClickHouse type | Ruby type |
 |-----------------|-----------|
-| UInt8/16/32/64 | Integer |
-| UInt128/256 | Integer |
-| Int8/16/32/64 | Integer |
-| Int128/256 | Integer |
+| UInt8/16/32/64/128/256 | Integer |
+| Int8/16/32/64/128/256 | Integer |
 | Float32/64 | Float |
 | Decimal | BigDecimal |
 | Bool | TrueClass/FalseClass |
@@ -164,66 +217,82 @@ response = conn.query(
 | Nullable | nil or inner type |
 | LowCardinality | inner type |
 
-## Configuration Options
+Unsupported types raise `ChConnect::UnsupportedTypeError` and discard the affected pooled connection.
+
+## Configuration options
 
 | Option | Default | Description |
 |--------|---------|-------------|
-| `scheme` | `"http"` | URL scheme (http/https) |
 | `host` | `"localhost"` | ClickHouse server host |
-| `port` | `8123` | ClickHouse HTTP port |
+| `port` | `9000` | Native TCP port (`9440` when TLS is enabled before port resolution) |
+| `compression` | `:lz4` | `:lz4`, `:zstd`, or `nil` |
+| `ssl` | `false` | Enable TLS |
+| `ssl_verify` | `true` | Verify the server certificate |
+| `ssl_ca` | `nil` | CA certificate file; system roots are used by default |
 | `database` | `"default"` | Database name |
-| `username` | `""` | Authentication username |
+| `username` | `"default"` | Authentication username |
 | `password` | `""` | Authentication password |
-| `connection_timeout` | `5` | Connection timeout in seconds |
-| `read_timeout` | `60` | Read timeout in seconds |
-| `write_timeout` | `60` | Write timeout in seconds |
-| `keep_alive_timeout` | `8` | Idle persistent connection timeout in seconds |
-| `pool_size` | `100` | Connection pool size |
+| `connection_timeout` | `5` | TCP and protocol handshake timeout in seconds |
+| `read_timeout` | `60` | Maximum time without response data |
+| `write_timeout` | `60` | Socket write deadline in seconds |
+| `keep_alive_timeout` | `60` | Recycle pooled TCP connections after this many idle seconds; `nil` disables recycling |
+| `pool_size` | `100` | Maximum pooled TCP connections |
 | `pool_timeout` | `5` | Pool checkout timeout in seconds |
-| `max_retries` | `3` | Max retry attempts on connection errors (0 to disable) |
-| `instrumenter` | `NullInstrumenter` | Instrumenter for query instrumentation |
+| `max_retries` | `3` | Retry limit for connection establishment and opted-in idempotent query transport failures |
+| `retry_base_interval` | `0.05` | Initial retry delay in seconds; retries use exponential backoff with jitter, and `0` disables the delay |
+| `retry_max_interval` | `1.0` | Maximum retry delay in seconds |
+| `instrumenter` | `NullInstrumenter` | Object responding to `instrument` |
 
 ## Instrumentation
 
-You can instrument queries by providing an instrumenter that responds to `#instrument`:
-
 ```ruby
 ChConnect.configure do |config|
   config.instrumenter = ActiveSupport::Notifications
 end
 
-# Subscribe to events
 ActiveSupport::Notifications.subscribe("query.clickhouse") do |name, start, finish, id, payload|
   puts "Query: #{payload[:sql]} took #{finish - start}s"
 end
 ```
 
-The instrumenter receives event name `"query.clickhouse"` and payload `{sql: "..."}`.
-
-## Error Handling
+## Error handling
 
 ```ruby
 begin
   conn.query("INVALID SQL")
-rescue ChConnect::QueryError => e
-  puts "Query failed: #{e.message}"
+rescue ChConnect::QueryError => error
+  warn error.message
 end
 
-# Unsupported types raise an exception
 begin
   conn.query("SELECT '{}'::JSON")
-rescue ChConnect::UnsupportedTypeError => e
-  puts "Unsupported type: #{e.message}"
+rescue ChConnect::UnsupportedTypeError => error
+  warn error.message
 end
 ```
 
+Network, timeout, pool, and protocol failures raise `ChConnect::ConnectionError`.
+They are retried after a query may have started only when `idempotent: true`;
+server `ChConnect::QueryError` responses are never retried.
+When migrating from the HTTP client, change port 8123/8443 to the server's
+native port (normally 9000/9440) and replace `scheme` with `ssl` or a native URL.
+
 ## Development
 
 ```bash
-# Run tests (requires ClickHouse)
-CLICKHOUSE_URL=http://default:password@localhost:8123/default bundle exec rspec
+cd ext/ch_connect_native
+ruby extconf.rb
+make
+cp "ch_connect_native.$(ruby -rrbconfig -e 'print RbConfig::CONFIG["DLEXT"]')" ../../lib/ch_connect/
+cd ../..
+
+CLICKHOUSE_URL=clickhouse://default:password@localhost:9000/default bundle exec rspec
+
+# Optional TLS specs
+spec/support/start_tls_clickhouse.sh /tmp/ch-tls password
+CH_TLS_PORT=9440 CH_TLS_CA=/tmp/ch-tls/server.crt \
+  CLICKHOUSE_URL=clickhouse://default:password@localhost:9000/default bundle exec rspec
 
-# Run linter
 bundle exec standardrb
 ```