clickhouse-ruby 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d7855eb261fe694066b665b2d1408b837a751d257c26d88f74d29847869fe652
-  data.tar.gz: 524b169428f84ab3f24c26433b4b57ead1f41e6147551c412e0f0a1840a8f7c9
+  metadata.gz: 51f95f8f05250a39be47798030cb5c1b9b75ed8b983e37fefafaec0584816481
+  data.tar.gz: 7e1c2f3f581b885a794164b594d8d49ee50c1c3be6a9e7cdae2568996114ba6d
 SHA512:
-  metadata.gz: 76da6aa320dabde964561304404eecfba2f500437ffc763f1bb15f6c5917c1cf703b1c14f2ec76024f7a585749993cd047dd7e6576dc2e33c2754e9677865556
-  data.tar.gz: 7bba509232194cef0a8dd560a08434d16a6ba3fd04a96d7868e1b00080d0421834e1a88c07f76c592589aadf2b473703ae6f3b5dd3cc692fcae6f3798843a446
+  metadata.gz: 7853a713aaa713f9a12bc0306f9c37861827127a94e74cf7bd2ca282c7a61e6a3750eecdb149c9dda80e7eefb367a2b943f48eddb6d60bd814d4bfc5076e2f98
+  data.tar.gz: b7c14c92a8443c9a263da13c1e0e59b52f1827f9959b815306ef5c08a0b8f629c2652a0a6a54fd039f6086d7e14d58d683bd711f62c1cbb42cccc3cf0ad299d9

data/CHANGELOG.md

@@ -7,6 +7,79 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.0] - 2026-02-02
+
+### Added
+
+#### ActiveRecord Query Extensions
+- **FINAL modifier** - Deduplication support for ReplacingMergeTree and CollapsingMergeTree tables
+  - Methods: `final`, `final!`, `final?`, `unscope_final`
+  - Auto-adds required settings when combined with PREWHERE
+  - Example: `User.final.where(id: 123)`
+
+- **SAMPLE clause** - Approximate queries on large datasets for performance
+  - Methods: `sample(ratio_or_rows, offset: nil)`, `sample!`, `sample_value`, `sample_offset`
+  - Supports fractional sampling (0.1 = 10%) and absolute row counts
+  - Preserves Integer vs Float distinction (1 = "at least 1 row", 1.0 = "100% of data")
+  - Example: `Event.sample(0.1).count`
+
+- **PREWHERE clause** - Query optimization that filters before reading all columns
+  - Methods: `prewhere(opts)`, `prewhere!`, `prewhere_values`, `prewhere.not(...)`
+  - Supports hash conditions, string conditions with placeholders, ranges, and Arel nodes
+  - Automatically optimized by ClickHouse when enabled
+  - Example: `Event.prewhere(date: Date.today).where(status: 'active')`
+
+- **SETTINGS DSL** - Per-query ClickHouse configuration
+  - Methods: `settings(opts)`, `settings!`, `query_settings`
+  - Normalizes boolean values (true/false → 1/0)
+  - Quotes string values automatically
+  - Example: `Event.settings(max_threads: 4, async_insert: true).all`
+
+#### Internal Improvements
+- Arel visitor integration for ClickHouse-specific SQL clauses
+- Proper SQL clause ordering: SELECT FROM [FINAL] [SAMPLE] [PREWHERE] [WHERE] [GROUP BY] [ORDER BY] [LIMIT] [SETTINGS]
+- RelationExtensions#build_arel override to attach ClickHouse state to Arel AST
+
+#### Type System Extensions
+- **Enum Type** - Support for Enum8 and Enum16 with string-to-integer mapping
+  - Methods: `cast`, `serialize`, `deserialize`
+  - Validation of enum values
+  - Example: `field_type = :Enum8` with values mapped via schema
+
+- **Decimal Type** - Arbitrary precision decimal support via BigDecimal
+  - Auto-mapping to Decimal32/64/128/256 based on precision
+  - Example: `field_type = :Decimal` with precision and scale
+
+#### Reliability Improvements
+- **Retry Logic** - Exponential backoff with jitter for transient failures
+  - Default: 1.6x multiplier, up to 120 seconds max backoff
+  - Configurable via `initial_backoff`, `backoff_multiplier`, `max_backoff`, `max_retries`
+  - Only retries transient errors (ConnectionError, Timeout, HTTP 5xx/429)
+  - Non-retriable: QueryError (syntax errors), HTTP 4xx
+
+- **Result Streaming** - Memory-efficient processing of large result sets
+  - Method: `stream_execute(sql) { |row| ... }`
+  - Yields rows one at a time using JSONEachRow format
+  - Constant memory usage regardless of result size
+  - Example: `client.stream_execute('SELECT * FROM huge_table') { |row| process(row) }`
+
+#### Performance Improvements
+- **HTTP Compression** - gzip compression for request/response
+  - Configuration: `compression: 'gzip'`, `compression_threshold: 1024`
+  - Built-in Zlib support (no external dependencies)
+  - Headers: `Content-Encoding: gzip`, `Accept-Encoding: gzip`
+  - Beneficial for large payloads (>1MB)
+
+### Changed
+- ActiveRecord relation extension architecture for better feature organization
+- Improved documentation with examples for all new features
+
+### Known Limitations
+- PREWHERE doesn't work with multiple JOINs (ClickHouse limitation)
+- SAMPLE requires table created with SAMPLE BY clause
+- Streaming cannot be used with FINAL or aggregate functions
+- HTTP compression has overhead for small payloads
+
 ## [0.1.0] - 2026-01-31
 
 ### Added
@@ -53,7 +126,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 #### Project Infrastructure
 - RSpec test suite with unit and integration tests
-- VCR for HTTP interaction recording
+- WebMock for HTTP mocking in unit tests
 - Docker Compose setup for ClickHouse testing
 - GitHub Actions CI workflow
 - RuboCop configuration

data/README.md

@@ -1,7 +1,27 @@
 # ClickhouseRuby
 
+[![Tests](https://github.com/Mohamad-Kamar/clickhouse-ruby/actions/workflows/test.yml/badge.svg)](https://github.com/Mohamad-Kamar/clickhouse-ruby/actions/workflows/test.yml)
+[![Gem Version](https://badge.fury.io/rb/clickhouse-ruby.svg)](https://badge.fury.io/rb/clickhouse-ruby)
+[![Downloads](https://img.shields.io/gem/dt/clickhouse-ruby.svg)](https://rubygems.org/gems/clickhouse-ruby)
+
 A lightweight Ruby client for ClickHouse with optional ActiveRecord integration.
 
+## Features
+
+**Core (v0.1.0)**
+- **Simple HTTP client** - Clean API for queries, commands, and bulk inserts
+- **Connection pooling** - Built-in connection pool with health checks
+- **Type system** - Full support for ClickHouse types including Nullable, Array, Map, Tuple
+- **Proper error handling** - Never silently ignores errors (fixes clickhouse-activerecord #230)
+- **SSL/TLS support** - Certificate verification enabled by default
+- **ActiveRecord integration** - Optional familiar model-based access
+
+**Enhanced (v0.2.0)**
+- **Enum & Decimal types** - Fixed-set values and arbitrary precision arithmetic
+- **Query optimization** - PREWHERE clause, FINAL deduplication, SAMPLE approximation
+- **Performance** - HTTP compression, result streaming, automatic retries with backoff
+- **Query control** - Per-query SETTINGS DSL for ClickHouse configuration
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -25,17 +45,19 @@ gem install clickhouse-ruby
 ## Quick Start
 
 ```ruby
-require 'clickhouse-ruby'
+require 'clickhouse_ruby'
+
+# Configure the client
+config = ClickhouseRuby::Configuration.new
+config.host = 'localhost'
+config.port = 8123
+config.database = 'default'
 
 # Create a client
-client = ClickhouseRuby::Client.new(
-  host: 'localhost',
-  port: 8123,
-  database: 'default'
-)
+client = ClickhouseRuby::Client.new(config)
 
 # Execute a query
-result = client.query('SELECT 1 + 1 AS result')
+result = client.execute('SELECT 1 + 1 AS result')
 puts result.first['result'] # => 2
 
 # Insert data
@@ -57,7 +79,8 @@ ClickhouseRuby.configure do |config|
   config.username = 'default'
   config.password = ''
   config.ssl = false
-  config.timeout = 60
+  config.connect_timeout = 10
+  config.read_timeout = 60
 end
 
 # Use the default client
@@ -72,10 +95,13 @@ client = ClickhouseRuby.client
 | `port` | HTTP interface port | `8123` |
 | `database` | Default database | `default` |
 | `username` | Authentication username | `default` |
-| `password` | Authentication password | `''` |
+| `password` | Authentication password | `nil` |
 | `ssl` | Enable HTTPS | `false` |
-| `timeout` | Request timeout in seconds | `60` |
-| `max_retries` | Number of retry attempts | `3` |
+| `ssl_verify` | Verify SSL certificates | `true` |
+| `connect_timeout` | Connection timeout in seconds | `10` |
+| `read_timeout` | Read timeout in seconds | `60` |
+| `write_timeout` | Write timeout in seconds | `60` |
+| `pool_size` | Connection pool size | `5` |
 
 ### Environment Variables
 
@@ -90,45 +116,75 @@ CLICKHOUSE_PASSWORD=secret
 CLICKHOUSE_SSL=false
 ```
 
-## Usage
+### v0.2.0 Enhancements
 
-### Querying Data
+**HTTP Compression** - Reduce bandwidth for large payloads:
+```ruby
+ClickhouseRuby.configure do |config|
+  config.compression = 'gzip'
+  config.compression_threshold = 1024  # Only compress > 1KB
+end
+```
 
+**Retry Logic** - Automatic retries with exponential backoff:
 ```ruby
-# Simple query
-result = client.query('SELECT * FROM events LIMIT 10')
+ClickhouseRuby.configure do |config|
+  config.max_retries = 3
+  config.initial_backoff = 1.0
+  config.backoff_multiplier = 1.6
+  config.max_backoff = 120
+end
+```
 
-# Query with parameters (prevents SQL injection)
-result = client.query(
-  'SELECT * FROM events WHERE date = {date:Date}',
-  params: { date: '2024-01-01' }
-)
+**Result Streaming** - Process large results with constant memory:
+```ruby
+client.stream_execute('SELECT * FROM huge_table') do |row|
+  process_row(row)
+end
+```
+
+**ActiveRecord Query Extensions** - ClickHouse-specific query methods:
+```ruby
+# Query optimization
+Event.prewhere(date: Date.today).where(status: 'active')
+
+# Deduplication
+User.final.where(id: 123)
+
+# Approximate queries
+Event.sample(0.1).count  # 10% sample
 
-# Query with specific format
-result = client.query('SELECT * FROM events', format: 'JSONEachRow')
+# Per-query configuration
+Event.settings(max_threads: 4).where(active: true)
 ```
 
-### Inserting Data
+See [docs/features/README.md](docs/features/README.md) for detailed documentation on all v0.2.0 features.
 
-```ruby
-# Insert hash array
-client.insert('events', [
-  { date: '2024-01-01', event_type: 'click', count: 100 },
-  { date: '2024-01-02', event_type: 'view', count: 250 }
-])
+## Usage
 
-# Insert with explicit columns
-client.insert('events', data, columns: [:date, :event_type, :count])
+### Querying Data
 
-# Bulk insert from CSV
-client.insert_from_file('events', '/path/to/data.csv', format: 'CSV')
+```ruby
+# Simple query - returns Result object
+result = client.execute('SELECT * FROM events LIMIT 10')
+result.each { |row| puts row['event_type'] }
+
+# Access columns and types
+result.columns  # => ['id', 'event_type', 'count']
+result.types    # => ['UInt64', 'String', 'UInt32']
+
+# Query with settings
+result = client.execute(
+  'SELECT * FROM events',
+  settings: { max_rows_to_read: 1_000_000 }
+)
 ```
 
-### DDL Operations
+### DDL Commands
 
 ```ruby
 # Create table
-client.execute(<<~SQL)
+client.command(<<~SQL)
   CREATE TABLE events (
     date Date,
     event_type String,
@@ -137,11 +193,21 @@ client.execute(<<~SQL)
   ORDER BY date
 SQL
 
-# Check if table exists
-client.table_exists?('events') # => true
+# Drop table
+client.command('DROP TABLE IF EXISTS events')
+```
+
+### Inserting Data
+
+```ruby
+# Insert hash array (uses efficient JSONEachRow format)
+client.insert('events', [
+  { date: '2024-01-01', event_type: 'click', count: 100 },
+  { date: '2024-01-02', event_type: 'view', count: 250 }
+])
 
-# Get table schema
-schema = client.describe_table('events')
+# Insert with explicit columns
+client.insert('events', data, columns: ['date', 'event_type', 'count'])
 ```
 
 ### Connection Management
@@ -153,20 +219,66 @@ client.ping # => true
 # Get server version
 client.server_version # => "24.1.1.123"
 
-# Execute multiple queries in a session
-client.with_session do |session|
-  session.execute('SET max_memory_usage = 1000000000')
-  session.query('SELECT * FROM large_table')
+# Get pool statistics
+client.pool_stats # => { size: 5, available: 5, in_use: 0 }
+
+# Close all connections
+client.close
+```
+
+## Type Support
+
+ClickhouseRuby supports all ClickHouse types:
+
+| ClickHouse Type | Ruby Type |
+|-----------------|-----------|
+| Int8-Int64, UInt8-UInt64 | Integer |
+| Float32, Float64 | Float |
+| String, FixedString | String |
+| Date, Date32 | Date |
+| DateTime, DateTime64 | Time |
+| UUID | String |
+| Bool | Boolean |
+| Nullable(T) | T or nil |
+| Array(T) | Array |
+| Map(K, V) | Hash |
+| Tuple(T...) | Array |
+| LowCardinality(T) | T |
+
+## Error Handling
+
+```ruby
+begin
+  client.execute('SELECT * FROM nonexistent_table')
+rescue ClickhouseRuby::UnknownTable => e
+  puts "Table not found: #{e.message}"
+  puts "Error code: #{e.code}"        # ClickHouse error code
+  puts "HTTP status: #{e.http_status}" # HTTP response code
+rescue ClickhouseRuby::ConnectionError => e
+  puts "Connection failed: #{e.message}"
+rescue ClickhouseRuby::QueryError => e
+  puts "Query failed: #{e.message}"
 end
 ```
 
+### Error Classes
+
+- `ClickhouseRuby::Error` - Base error class
+- `ClickhouseRuby::ConnectionError` - Connection issues
+- `ClickhouseRuby::ConnectionTimeout` - Timeout errors
+- `ClickhouseRuby::QueryError` - Query execution errors
+- `ClickhouseRuby::SyntaxError` - SQL syntax errors
+- `ClickhouseRuby::UnknownTable` - Table doesn't exist
+- `ClickhouseRuby::UnknownColumn` - Column doesn't exist
+- `ClickhouseRuby::UnknownDatabase` - Database doesn't exist
+
 ## ActiveRecord Integration
 
 ClickhouseRuby provides optional ActiveRecord integration for familiar model-based access.
 
 ```ruby
-# config/initializers/clickhouse-ruby.rb
-require 'clickhouse-ruby/active_record'
+# config/initializers/clickhouse_ruby.rb
+require 'clickhouse_ruby/active_record'
 
 ClickhouseRuby::ActiveRecord.establish_connection(
   host: 'localhost',
@@ -184,24 +296,6 @@ Event.where(date: '2024-01-01').limit(10).each do |event|
 end
 ```
 
-## Error Handling
-
-```ruby
-begin
-  client.query('SELECT * FROM nonexistent_table')
-rescue ClickhouseRuby::ConnectionError => e
-  # Handle connection issues
-  puts "Connection failed: #{e.message}"
-rescue ClickhouseRuby::QueryError => e
-  # Handle query errors
-  puts "Query failed: #{e.message}"
-  puts "Error code: #{e.code}"
-rescue ClickhouseRuby::TimeoutError => e
-  # Handle timeouts
-  puts "Request timed out: #{e.message}"
-end
-```
-
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests.
@@ -212,29 +306,21 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
 # Start ClickHouse
 docker-compose up -d
 
-# Run all tests
-bundle exec rake spec
+# Run unit tests only
+bundle exec rspec spec/unit
 
-# Run only unit tests
-bundle exec rake spec_unit
-
-# Run integration tests
-bundle exec rake spec_integration
+# Run all tests including integration
+CLICKHOUSE_TEST_INTEGRATION=true bundle exec rspec
 ```
 
-### Code Quality
-
-```bash
-# Run RuboCop
-bundle exec rake rubocop
+## Requirements
 
-# Auto-fix issues
-bundle exec rake rubocop_fix
-```
+- Ruby >= 2.6.0
+- ClickHouse >= 20.x (tested with 24.x)
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/yourusername/clickhouse-ruby.
+Bug reports and pull requests are welcome on GitHub at https://github.com/Mohamad-Kamar/clickhouse-ruby.
 
 1. Fork it
 2. Create your feature branch (`git checkout -b feature/my-new-feature`)