clickhouse-ruby 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 51f95f8f05250a39be47798030cb5c1b9b75ed8b983e37fefafaec0584816481
-  data.tar.gz: 7e1c2f3f581b885a794164b594d8d49ee50c1c3be6a9e7cdae2568996114ba6d
+  metadata.gz: 97ae40434fd1078ad34271a031ed28442d831f569200a2b7e71f0f99184d907d
+  data.tar.gz: '002378905be755b1fe9fd54f05478ef6b0f3d7d020beb0b890e5f508b300ec8f'
 SHA512:
-  metadata.gz: 7853a713aaa713f9a12bc0306f9c37861827127a94e74cf7bd2ca282c7a61e6a3750eecdb149c9dda80e7eefb367a2b943f48eddb6d60bd814d4bfc5076e2f98
-  data.tar.gz: b7c14c92a8443c9a263da13c1e0e59b52f1827f9959b815306ef5c08a0b8f629c2652a0a6a54fd039f6086d7e14d58d683bd711f62c1cbb42cccc3cf0ad299d9
+  metadata.gz: 90cd05a510db8a9322e5828a52d33386888f18d2cf40c2b0514514cbb25a3448155ae41eecdaada148f8d16f75bf6ab5570f5aa1c004ca8604475ebd9a1ab404
+  data.tar.gz: 8d39e17f93bfd955eab008fceb13e847fbbaf6e705ab95814b401e10ec91919b4109be8b486f010be4797778298d3c70a47eba586f8e8f89e4f135b9b0c9f09d

data/CHANGELOG.md

@@ -7,6 +7,73 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-02-02
+
+### Added
+
+#### Observability & Instrumentation
+- **ActiveSupport::Notifications Integration** - Event-driven monitoring for APM tools
+  - Events: `clickhouse_ruby.query.complete`, `clickhouse_ruby.query.error`, `clickhouse_ruby.insert.complete`
+  - Pool events: `clickhouse_ruby.pool.checkout`, `clickhouse_ruby.pool.checkin`, `clickhouse_ruby.pool.timeout`
+  - Graceful fallback when ActiveSupport is not available
+  - Query timing with millisecond precision using monotonic clock
+  - Example: `ActiveSupport::Notifications.subscribe(/clickhouse_ruby/) { |*args| ... }`
+
+- **Enhanced Logging** - Debug-level query timing logs
+  - Query duration logging at debug level
+  - Insert timing with row count
+  - Structured payload data for external processing
+
+#### Performance Benchmarking
+- **Benchmark Suite** - Comprehensive performance testing infrastructure
+  - Rake tasks: `rake benchmark`, `rake benchmark:quick`, `rake benchmark:connection`, `rake benchmark:query`, `rake benchmark:insert`
+  - Uses `benchmark-ips` for iterations-per-second measurements
+  - Performance targets from MVP: Connection <100ms, SELECT <50ms, 10K INSERT <1s
+  - Latency statistics: min, max, avg, median, p95, p99
+
+#### ActiveRecord Migration Helpers
+- **Migration Generator** - Rails generator for ClickHouse migrations
+  - Command: `rails generate clickhouse:migration CreateEvents field:type`
+  - ClickHouse-specific options: `--engine`, `--order-by`, `--partition-by`, `--primary-key`, `--settings`
+  - Cluster support with automatic Replicated* engine selection
+  - Auto-detects migration action from name (create_table, add_column, remove_column)
+
+- **Migration Templates** - ClickHouse-aware migration templates
+  - Supports MergeTree, ReplacingMergeTree, SummingMergeTree, AggregatingMergeTree
+  - Partition expressions with proper quoting
+  - Settings block support
+
+- **Schema Dumper** - Rails-compatible schema dumping
+  - Extracts table options from system.tables
+  - Dumps ClickHouse-specific column options (Nullable, LowCardinality, Decimal, DateTime64)
+  - View and index dumping support
+
+#### Query Tools
+- **EXPLAIN Support** - Query plan analysis
+  - Methods: `client.explain(sql, type: :plan)`
+  - Types: `:plan`, `:pipeline`, `:estimate`, `:ast`, `:syntax`
+  - Example: `client.explain('SELECT * FROM events', type: :pipeline)`
+
+- **Enhanced Health Check** - Comprehensive server health status
+  - Returns: status, server_version, current_database, server_uptime, pool health
+  - Single method for monitoring dashboards
+  - Example: `client.health_check`
+
+- **Detailed Pool Statistics** - Monitoring-ready metrics
+  - Method: `pool.detailed_stats`
+  - Returns: utilization_percent, checkout rate per minute, timeout rate
+  - Suitable for Prometheus/StatsD export
+
+### Changed
+- Connection pool now publishes instrumentation events on checkout/checkin
+- Query and insert operations track timing automatically
+- Pool timeout errors include instrumentation payload
+
+### Development
+- Added `benchmark-ips` (~> 2.12) as development dependency
+- New `benchmark/` directory with helper and benchmark files
+- RuboCop exclusions for benchmark files
+
 ## [0.2.0] - 2026-02-02
 
 ### Added

data/README.md

@@ -6,6 +6,33 @@
 
 A lightweight Ruby client for ClickHouse with optional ActiveRecord integration.
 
+## Why ClickhouseRuby?
+
+ClickhouseRuby is designed from the ground up with production reliability and developer experience in mind. Here's what sets it apart:
+
+**🔒 Security & Reliability**
+- **SSL verification enabled by default** - Secure by default, unlike alternatives that require explicit configuration
+- **Never silently fails** - All errors are properly raised and propagated (fixes [clickhouse-activerecord #230](https://github.com/patrikx3/clickhouse-activerecord/issues/230))
+- **Comprehensive error hierarchy** - 30+ specific error classes mapped from ClickHouse error codes
+
+**⚡ Performance & Architecture**
+- **Zero runtime dependencies** - Uses only Ruby stdlib, making it lightweight and fully auditable
+- **AST-based type parser** - Handles complex nested types correctly (Array(Tuple(String, UInt64)), etc.) unlike regex-based parsers
+- **Thread-safe connection pooling** - Built-in pool with health checks and proper resource management
+- **Result streaming** - Process millions of rows with constant memory usage
+
+**🛠️ Developer Experience**
+- **Clean, intuitive API** - Simple methods for queries, inserts, and DDL operations
+- **Optional ActiveRecord integration** - Familiar model-based access when you need it
+- **ClickHouse-specific query extensions** - PREWHERE, FINAL, SAMPLE, and SETTINGS DSL built-in
+- **Comprehensive type system** - Full support for all ClickHouse types including Nullable, Array, Map, Tuple, Enum, Decimal
+
+**📊 Production Ready**
+- **Automatic retries** - Configurable exponential backoff for transient failures
+- **HTTP compression** - Reduce bandwidth for large payloads
+- **Connection health monitoring** - Pool statistics and health checks
+- **Extensive test coverage** - 80%+ coverage with both unit and integration tests
+
 ## Features
 
 **Core (v0.1.0)**
@@ -98,10 +125,15 @@ client = ClickhouseRuby.client
 | `password` | Authentication password | `nil` |
 | `ssl` | Enable HTTPS | `false` |
 | `ssl_verify` | Verify SSL certificates | `true` |
+| `ssl_ca_path` | Custom CA certificate file path | `nil` |
 | `connect_timeout` | Connection timeout in seconds | `10` |
 | `read_timeout` | Read timeout in seconds | `60` |
 | `write_timeout` | Write timeout in seconds | `60` |
 | `pool_size` | Connection pool size | `5` |
+| `log_level` | Logger level (`:debug`, `:info`, `:warn`, `:error`) | `:info` |
+| `default_settings` | Global ClickHouse settings for all queries | `{}` |
+| `pool_timeout` | Wait time for available connection in seconds | `5` |
+| `retry_jitter` | Jitter strategy (`:full`, `:equal`, `:none`) | `:equal` |
 
 ### Environment Variables
 
@@ -180,6 +212,39 @@ result = client.execute(
 )
 ```
 
+### Result Metadata
+
+Access query metadata and result information:
+
+```ruby
+result = client.execute('SELECT * FROM events LIMIT 100')
+
+# Query execution metadata
+result.elapsed_time    # => 0.042 (seconds)
+result.rows_read       # => 100
+result.bytes_read      # => 8500
+result.rows_written    # => 0 (for SELECT queries)
+result.bytes_written   # => 0
+
+# Result size information
+result.count           # => 100 (alias for size/length)
+result.size            # => 100
+result.length          # => 100
+
+# Column information
+result.columns         # => ["id", "event_type", "count"]
+result.column_types    # => ["UInt64", "String", "UInt32"]
+result.types           # => ["UInt64", "String", "UInt32"]
+
+# Row access methods
+result.first           # => {"id" => 1, "event_type" => "click", "count" => 100}
+result.last            # => {"id" => 100, "event_type" => "view", "count" => 50}
+result[5]              # => {"id" => 5, "event_type" => "click", "count" => 75}
+
+# Get all values for a specific column
+result.column_values("event_type")  # => ["click", "view", "click", ...]
+```
+
 ### DDL Commands
 
 ```ruby
@@ -220,12 +285,88 @@ client.ping # => true
 client.server_version # => "24.1.1.123"
 
 # Get pool statistics
-client.pool_stats # => { size: 5, available: 5, in_use: 0 }
+client.pool_stats # => {
+#   size: 5,                    # Total pool capacity
+#   available: 4,               # Connections ready to use
+#   in_use: 1,                  # Connections currently in use
+#   total_connections: 5,       # Total connections created
+#   total_checkouts: 150,       # Lifetime pool checkout count
+#   total_timeouts: 0,          # Timeouts waiting for connection
+#   uptime_seconds: 3600.5      # Seconds since pool created
+# }
 
 # Close all connections
 client.close
 ```
 
+### Advanced Client Methods
+
+#### Batch Processing
+
+Process large query results in batches to manage memory efficiently:
+
+```ruby
+# Process 500 rows at a time
+client.each_batch('SELECT * FROM huge_table', batch_size: 500) do |batch|
+  # batch is an array of hashes (max 500 rows)
+  puts "Processing batch of #{batch.size} rows"
+  insert_to_cache(batch)
+end
+
+# Default batch size is 500 rows
+client.each_batch('SELECT * FROM data') { |batch| process(batch) }
+```
+
+#### Row-by-Row Processing
+
+Process results one row at a time for maximum memory efficiency:
+
+```ruby
+# Stream processing - constant memory usage
+client.each_row('SELECT * FROM massive_table') do |row|
+  # row is a single hash
+  puts "Processing: #{row['id']}"
+  update_statistics(row)
+end
+
+# Returns Enumerator if no block given
+rows = client.each_row('SELECT * FROM table')
+rows.each { |row| puts row['name'] }
+```
+
+#### Connection Aliases
+
+Additional connection management methods:
+
+```ruby
+# Disconnect all connections in the pool
+client.disconnect
+
+# Check if client is connected
+client.connected?  # => true
+```
+
+#### Module-Level Methods
+
+Quick access without explicit client instance:
+
+```ruby
+# Execute query using default client
+result = ClickhouseRuby.execute('SELECT 1 AS num')
+
+# Insert data using default client
+ClickhouseRuby.insert('events', [
+  { date: '2024-01-01', event_type: 'click' }
+])
+
+# Ping default client
+ClickhouseRuby.ping  # => true
+```
+
+**Performance Note**: Batch and row-by-row processing methods use result streaming internally,
+which maintains constant memory usage regardless of result size. Use these methods for queries
+that return millions of rows to prevent memory exhaustion.
+
 ## Type Support
 
 ClickhouseRuby supports all ClickHouse types:

data/lib/clickhouse_ruby/active_record/generators/migration_generator.rb

@@ -0,0 +1,308 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record/migration"
+
+module ClickhouseRuby
+  module Generators
+    # Rails generator for creating ClickHouse migrations
+    #
+    # This generator creates migration files with ClickHouse-specific options
+    # like ENGINE, ORDER BY, PARTITION BY, and PRIMARY KEY.
+    #
+    # @example Generate a migration
+    #   rails generate clickhouse:migration CreateEvents
+    #
+    # @example Generate a migration with columns
+    #   rails generate clickhouse:migration CreateEvents user_id:integer name:string
+    #
+    # @example Generate a migration with ClickHouse options
+    #   rails generate clickhouse:migration CreateEvents user_id:integer --engine=ReplacingMergeTree --order-by=user_id
+    #
+    class MigrationGenerator < Rails::Generators::NamedBase
+      include ActiveRecord::Generators::Migration
+
+      source_root File.expand_path("templates", __dir__)
+
+      argument :attributes, type: :array, default: [], banner: "field:type field:type"
+
+      class_option :engine,
+                   type: :string,
+                   default: "MergeTree",
+                   desc: "ClickHouse table engine (MergeTree, ReplacingMergeTree, SummingMergeTree, etc.)"
+
+      class_option :order_by,
+                   type: :string,
+                   desc: "ORDER BY clause for MergeTree family engines"
+
+      class_option :partition_by,
+                   type: :string,
+                   desc: "PARTITION BY clause for data partitioning"
+
+      class_option :primary_key,
+                   type: :string,
+                   desc: "PRIMARY KEY clause (defaults to ORDER BY if not specified)"
+
+      class_option :settings,
+                   type: :string,
+                   desc: "Table SETTINGS clause"
+
+      class_option :cluster,
+                   type: :string,
+                   desc: "Cluster name for distributed tables"
+
+      # Generate the migration file
+      #
+      # @return [void]
+      def create_migration_file
+        set_local_assigns!
+        validate_engine!
+
+        migration_template "migration.rb.tt", File.join(db_migrate_path, "#{file_name}.rb")
+      end
+
+      private
+
+      # Set local variables for use in templates
+      #
+      # @return [void]
+      def set_local_assigns!
+        @migration_action = detect_migration_action
+      end
+
+      # Detect the type of migration based on the name
+      #
+      # @return [Symbol] :create_table, :add_column, :remove_column, or :change_table
+      def detect_migration_action
+        case file_name
+        when /^create_/
+          :create_table
+        when /^add_.*_to_/
+          :add_column
+        when /^remove_.*_from_/
+          :remove_column
+        else
+          :change_table
+        end
+      end
+
+      # Validate the engine option
+      #
+      # @raise [ArgumentError] if the engine is invalid
+      # @return [void]
+      def validate_engine!
+        return if valid_engines.include?(options[:engine])
+
+        raise ArgumentError, "Invalid engine '#{options[:engine]}'. Valid engines: #{valid_engines.join(", ")}"
+      end
+
+      # List of valid ClickHouse engines
+      #
+      # @return [Array<String>] valid engine names
+      def valid_engines
+        %w[
+          MergeTree ReplacingMergeTree SummingMergeTree AggregatingMergeTree
+          CollapsingMergeTree VersionedCollapsingMergeTree GraphiteMergeTree
+          Log TinyLog StripeLog Memory Null Set Join Buffer Distributed
+          MaterializedView Dictionary
+        ]
+      end
+
+      # Get the table name from the migration name
+      #
+      # @return [String] the table name
+      def table_name
+        @table_name ||= extract_table_name
+      end
+
+      # Extract table name based on migration action
+      #
+      # @return [String] the extracted table name
+      def extract_table_name
+        case @migration_action
+        when :create_table then file_name.sub(/^create_/, "")
+        when :add_column then file_name.sub(/^add_\w+_to_/, "")
+        when :remove_column then file_name.sub(/^remove_\w+_from_/, "")
+        else file_name
+        end
+      end
+
+      # Get the column name for add/remove column migrations
+      #
+      # @return [String, nil] the column name or nil
+      def column_name
+        @column_name ||= extract_column_name
+      end
+
+      # Extract column name based on migration action
+      #
+      # @return [String, nil] the extracted column name
+      def extract_column_name
+        case @migration_action
+        when :add_column then file_name[/^add_(\w+)_to_/, 1]
+        when :remove_column then file_name[/^remove_(\w+)_from_/, 1]
+        end
+      end
+
+      # Get the engine with cluster option if specified
+      #
+      # @return [String] the engine specification
+      def engine_with_cluster
+        engine = options[:engine]
+        return engine unless options[:cluster]
+
+        "Replicated#{engine}('/clickhouse/tables/{shard}/#{table_name}', '{replica}')"
+      end
+
+      # Get the ORDER BY clause
+      #
+      # @return [String, nil] the ORDER BY expression
+      def order_by_clause
+        options[:order_by] || infer_order_by
+      end
+
+      # Infer ORDER BY from primary key or first column
+      #
+      # @return [String, nil] inferred ORDER BY expression
+      def infer_order_by
+        return unless @migration_action == :create_table
+
+        # Use id if present, otherwise first attribute
+        return "id" if attributes.any? { |attr| attr.name == "id" }
+        return attributes.first.name if attributes.any?
+
+        "tuple()"
+      end
+
+      # Get the PARTITION BY clause
+      #
+      # @return [String, nil] the PARTITION BY expression
+      def partition_by_clause
+        options[:partition_by]
+      end
+
+      # Get the PRIMARY KEY clause
+      #
+      # @return [String, nil] the PRIMARY KEY expression
+      def primary_key_clause
+        options[:primary_key]
+      end
+
+      # Get the SETTINGS clause
+      #
+      # @return [String, nil] the SETTINGS expression
+      def settings_clause
+        options[:settings]
+      end
+
+      # Convert Rails type to ClickHouse type
+      #
+      # @param type [String] the Rails type
+      # @param attr_options [Hash] type options
+      # @return [String] the ClickHouse type
+      def clickhouse_type(type, attr_options = {})
+        TypeMapper.to_clickhouse(type, attr_options)
+      end
+
+      # Get the path to db/migrate directory
+      #
+      # @return [String] the migration directory path
+      def db_migrate_path
+        return "db/migrate" unless defined?(Rails.application) && Rails.application
+
+        Rails.application.config.paths["db/migrate"].to_a.first
+      end
+    end
+
+    # Maps Rails types to ClickHouse types
+    module TypeMapper
+      # Type mapping from Rails types to ClickHouse types
+      TYPE_MAP = {
+        primary_key: "UInt64",
+        bigint: "Int64",
+        time: "DateTime",
+        date: "Date",
+        binary: "String",
+        boolean: "UInt8",
+        uuid: "UUID",
+        json: "String",
+      }.freeze
+
+      # Integer size mapping
+      INTEGER_SIZES = {
+        1 => "Int8",
+        2 => "Int16",
+      }.freeze
+
+      class << self
+        # Convert a Rails type to ClickHouse type
+        #
+        # @param type [String, Symbol] the Rails type
+        # @param options [Hash] type options
+        # @return [String] the ClickHouse type
+        def to_clickhouse(type, options = {})
+          type_sym = type.to_sym
+
+          # Check simple type map first
+          return TYPE_MAP[type_sym] if TYPE_MAP.key?(type_sym)
+
+          # Handle complex types
+          complex_type(type_sym, options) || type.to_s
+        end
+
+        private
+
+        # Complex type handlers
+        COMPLEX_TYPE_HANDLERS = %i[string text integer float decimal datetime timestamp].freeze
+
+        # Handle complex type conversions
+        #
+        # @param type [Symbol] the Rails type
+        # @param options [Hash] type options
+        # @return [String, nil] the ClickHouse type or nil
+        def complex_type(type, options)
+          return string_type(options) if %i[string text].include?(type)
+
+          send("#{type}_type", options) if COMPLEX_TYPE_HANDLERS.include?(type)
+        end
+
+        # Get the float type based on limit
+        def float_type(options)
+          options[:limit] == 8 ? "Float64" : "Float32"
+        end
+
+        # Get the timestamp type based on precision
+        def timestamp_type(options)
+          "DateTime64(#{options[:precision] || 3})"
+        end
+
+        # Get the string type based on options
+        def string_type(options)
+          options[:limit] ? "FixedString(#{options[:limit]})" : "String"
+        end
+
+        # Get the integer type based on limit
+        def integer_type(options)
+          limit = options[:limit]
+          return INTEGER_SIZES[limit] if INTEGER_SIZES.key?(limit)
+          return "Int32" if limit.nil? || limit <= 4
+          return "Int64" if limit <= 8
+
+          "Int32"
+        end
+
+        # Get the decimal type based on precision and scale
+        def decimal_type(options)
+          precision = options[:precision] || 10
+          scale = options[:scale] || 0
+          "Decimal(#{precision}, #{scale})"
+        end
+
+        # Get the datetime type based on precision
+        def datetime_type(options)
+          options[:precision] ? "DateTime64(#{options[:precision]})" : "DateTime"
+        end
+      end
+    end
+  end
+end

data/lib/clickhouse_ruby/active_record/generators/templates/create_table.rb.tt

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+class <%= class_name %> < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
+  def change
+    create_table :<%= table_name %>, **clickhouse_options do |t|
+<%- attributes.each do |attribute| -%>
+<%- if attribute.type == :references -%>
+      t.bigint :<%= attribute.name %>_id<%= attribute.has_index? ? ", index: true" : "" %>
+<%- else -%>
+      t.<%= attribute.type %> :<%= attribute.name %>
+<%- end -%>
+<%- end -%>
+<%- if attributes.empty? -%>
+      t.uuid :id
+<%- end -%>
+      t.timestamps
+    end
+  end
+
+  private
+
+  # ClickHouse-specific table options
+  #
+  # MergeTree Engine Family:
+  # - MergeTree: Basic engine, good for general analytics
+  # - ReplacingMergeTree: Deduplicates rows by ORDER BY key
+  # - SummingMergeTree: Automatically sums numeric columns
+  # - AggregatingMergeTree: Stores pre-aggregated states
+  # - CollapsingMergeTree: Supports row collapsing with sign column
+  #
+  # ORDER BY:
+  # - Determines physical data sorting
+  # - First columns should be most frequently filtered
+  # - Affects compression and query performance
+  #
+  # PARTITION BY:
+  # - Usually partition by date (toYYYYMM, toYYYYMMDD)
+  # - Enables efficient data retention and queries
+  # - Don't over-partition (thousands of partitions is too many)
+  #
+  # @return [Hash] ClickHouse table options
+  def clickhouse_options
+    {
+      engine: "<%= engine_with_cluster %>",
+<%- if order_by_clause -%>
+      order_by: "<%= order_by_clause %>",
+<%- end -%>
+<%- if partition_by_clause -%>
+      partition_by: "<%= partition_by_clause %>",
+<%- end -%>
+<%- if primary_key_clause -%>
+      primary_key: "<%= primary_key_clause %>",
+<%- end -%>
+<%- if settings_clause -%>
+      settings: "<%= settings_clause %>",
+<%- end -%>
+    }
+  end
+end