clickhouse-activerecord 0.3.5 → 0.3.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: f7b2fb3282fdff192d0cec9cb19a68ba70d02d83
4
- data.tar.gz: 50dea626b278f9ac8a33ea95d300a5d440fa229a
2
+ SHA256:
3
+ metadata.gz: 67ea4a63bb140adb5f6de102c38c097ea18f61499a882574952a9982b8d340cf
4
+ data.tar.gz: a1d2a759dae9062a20cc0a53bbd1cb64142062ce9ef84047f7e953b7d957e9b2
5
5
  SHA512:
6
- metadata.gz: 06a3b3ae02ca8e2e60974109f1ead35743340b8e12beefa28134cadeeff4df753db044da60e25af007a5e14abda14695fc1725d5b86aa5432dce7f642d2004ab
7
- data.tar.gz: 9d8215272de3a2b9c0bc2d4671cc22d769acdf1e3f8c32ea8d685c28a32be5838bb28dc7c77a8c4bbdaa7960df1537bf0aad7f8db7571c07793c85a64fbcb696
6
+ metadata.gz: 243b4d089d6349d8c12c529fbb80078ffbaa03f0033c3c69be77751846b031398895a1efdd08460699636277695a3e03eb228a398b5887a9426f4f3b63496c31
7
+ data.tar.gz: f790bca5af79492390de98ba2dce363e16207d987093c2e94e9f9ab55b9f769a41bae1c4a12d0b9d3757ee91f92604c1df4db17479dbbc85ced0f48294b297f3
@@ -1,17 +1,26 @@
1
+ ### Version 0.3.10 (Dec 20, 2019)
2
+
3
+ * Support structure dump/load [@StoneGod](https://github.com/StoneGod)
4
+
5
+ ### Version 0.3.6 (Sep 2, 2019)
6
+
7
+ * Support Rails 6.0
8
+ * Fix relation `last` method
9
+
1
10
  ### Version 0.3.4 (Jun 28, 2019)
2
11
 
3
- * Fix DateTime sql format without microseconds for Rails 5.2
12
+ * Fix DateTime sql format without microseconds for Rails 5.2
4
13
  * Support ssl connection
5
14
  * Migration support
6
15
  * Rake tasks for create / drop database
7
-
16
+
8
17
  ### Version 0.3.0 (Nov 27, 2018)
9
18
 
10
19
  * Support materialized view
11
20
  * Aggregated functions for view
12
21
  * Schema dumper with SQL create table
13
22
  * Added migrations support [@Bugagazavr](https://github.com/Bugagazavr)
14
-
23
+
15
24
  ### Version 0.2.0 (Oct 3, 2017)
16
25
 
17
26
  * Support Rails 5.0
@@ -19,7 +28,7 @@
19
28
  ### Version 0.1.2 (Sep 27, 2017)
20
29
 
21
30
  * Fix Big Int type
22
-
31
+
23
32
  ### Version 0.1.0 (Aug 31, 2017)
24
33
 
25
34
  * Initial release
data/README.md CHANGED
@@ -1,6 +1,7 @@
1
1
  # Clickhouse::Activerecord
2
2
 
3
3
  A Ruby database ActiveRecord driver for ClickHouse. Support Rails >= 5.2.
4
+ Tested on ClickHouse version 18.14.
4
5
 
5
6
  ## Installation
6
7
 
@@ -17,8 +18,24 @@ And then execute:
17
18
  Or install it yourself as:
18
19
 
19
20
  $ gem install clickhouse-activerecord
21
+
22
+ ## Available database connection parameters
23
+ ```yml
24
+ default: &default
25
+ adapter: clickhouse
26
+ database: database
27
+ host: localhost
28
+ port: 8123
29
+ username: username
30
+ password: password
31
+ ssl: true # optional for using ssl connection
32
+ debug: true # use for showing in to log technical information
33
+ migrations_paths: db/clickhouse # optional, default: db/migrate_clickhouse
34
+ cluster: 'cluster_name' # optional for creating tables in cluster
35
+ replica: '{shard}' # optional for creating system tables for shards
36
+ ```
20
37
 
21
- ## Usage
38
+ ## Usage in Rails 5
22
39
 
23
40
  Add your `database.yml` connection information with postfix `_clickhouse` for you environment:
24
41
 
@@ -26,10 +43,6 @@ Add your `database.yml` connection information with postfix `_clickhouse` for yo
26
43
  development_clickhouse:
27
44
  adapter: clickhouse
28
45
  database: database
29
- host: localhost
30
- username: username
31
- password: password
32
- debug: true # use for showing in to log technical information
33
46
  ```
34
47
 
35
48
  Add to your model:
@@ -54,19 +67,45 @@ Or global connection:
54
67
  development:
55
68
  adapter: clickhouse
56
69
  database: database
57
- host: localhost
58
- username: username
59
- password: password
70
+ ```
71
+
72
+ ## Usage in Rails 6 with second database
73
+
74
+ Add your `database.yml` connection information for you environment:
75
+
76
+ ```yml
77
+ development:
78
+ primary:
79
+ ...
80
+
81
+ clickhouse:
82
+ adapter: clickhouse
83
+ database: database
84
+ ```
85
+
86
+ Connection [Multiple Databases with Active Record](https://guides.rubyonrails.org/active_record_multiple_databases.html) or short example:
87
+
88
+ ```ruby
89
+ class Action < ActiveRecord::Base
90
+ connects_to database: { writing: :clickhouse, reading: :clickhouse }
91
+ end
60
92
  ```
61
93
 
62
94
  ### Rake tasks
63
95
 
96
+ **Note!** For Rails 6 you can use default rake tasks if you configure `migrations_paths` in your `database.yml`, for example: `rake db:migrate`
97
+
64
98
  Create / drop / purge / reset database:
65
99
 
66
100
  $ rake clickhouse:create
67
101
  $ rake clickhouse:drop
68
102
  $ rake clickhouse:purge
69
103
  $ rake clickhouse:reset
104
+
105
+ Prepare system tables for rails:
106
+
107
+ $ rake clickhouse:prepare_schema_migration_table
108
+ $ rake clickhouse:prepare_internal_metadata_table
70
109
 
71
110
  Migration:
72
111
 
@@ -75,6 +114,10 @@ Migration:
75
114
 
76
115
  Rollback migration not supported!
77
116
 
117
+ ### Dump / Load for multiple using databases
118
+
119
+ If you using multiple databases, for example: PostgreSQL, Clickhouse.
120
+
78
121
  Schema dump to `db/clickhouse_schema.rb` file:
79
122
 
80
123
  $ rake clickhouse:schema:dump
@@ -82,9 +125,24 @@ Schema dump to `db/clickhouse_schema.rb` file:
82
125
  Schema load from `db/clickhouse_schema.rb` file:
83
126
 
84
127
  $ rake clickhouse:schema:load
85
-
128
+
86
129
  We use schema for emulate development or tests environment on PostgreSQL adapter.
87
130
 
131
+ Structure dump to `db/clickhouse_structure.sql` file:
132
+
133
+ $ rake clickhouse:structure:dump
134
+
135
+ Structure load from `db/clickhouse_structure.sql` file:
136
+
137
+ $ rake clickhouse:structure:load
138
+
139
+ ### Dump / Load for only Clickhouse database using
140
+
141
+ $ rake db:schema:dump
142
+ $ rake db:schema:load
143
+ $ rake db:structure:dump
144
+ $ rake db:structure:load
145
+
88
146
  ### Insert and select data
89
147
 
90
148
  ```ruby
@@ -105,7 +163,7 @@ ActionView.maximum(:date)
105
163
 
106
164
  Donations to this project are going directly to [PNixx](https://github.com/PNixx), the original author of this project:
107
165
 
108
- * BTC address: `1Lx2gaJtzfF2dxGFxB65YtY5kNY9xUi6ia`
166
+ * BTC address: `1H3rhpf7WEF5JmMZ3PVFMQc7Hm29THgUfN`
109
167
  * ETH address: `0x6F094365A70fe7836A633d2eE80A1FA9758234d5`
110
168
  * XMR address: `42gP71qLB5M43RuDnrQ3vSJFFxis9Kw9VMURhpx9NLQRRwNvaZRjm2TFojAMC8Fk1BQhZNKyWhoyJSn5Ak9kppgZPjE17Zh`
111
169
 
@@ -23,8 +23,8 @@ Gem::Specification.new do |spec|
23
23
  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
24
24
  spec.require_paths = ['lib']
25
25
 
26
- spec.add_runtime_dependency 'bundler', '~> 1.13', '>= 1.13.4'
27
- spec.add_runtime_dependency 'activerecord', '~> 5.2'
26
+ spec.add_runtime_dependency 'bundler', '>= 1.13.4'
27
+ spec.add_runtime_dependency 'activerecord', '>= 5.2'
28
28
 
29
29
  spec.add_development_dependency 'bundler', '~> 1.15'
30
30
  spec.add_development_dependency 'rake', '~> 10.0'
@@ -8,6 +8,7 @@ module ActiveRecord
8
8
 
9
9
  def serialize(value)
10
10
  value = super
11
+ return unless value
11
12
  return value.strftime('%Y-%m-%d %H:%M:%S') unless value.acts_like?(:time)
12
13
 
13
14
  value.to_time.strftime('%Y-%m-%d %H:%M:%S')
@@ -17,6 +17,8 @@ module ActiveRecord
17
17
  def exec_query(sql, name = nil, binds = [], prepare: false)
18
18
  result = do_execute(sql, name)
19
19
  ActiveRecord::Result.new(result['meta'].map { |m| m['name'] }, result['data'])
20
+ rescue StandardError => _e
21
+ raise ActiveRecord::ActiveRecordError, "Response: #{result}"
20
22
  end
21
23
 
22
24
  def exec_update(_sql, _name = nil, _binds = [])
@@ -34,41 +36,10 @@ module ActiveRecord
34
36
  end
35
37
 
36
38
  def table_options(table)
37
- sql = do_system_execute("SHOW CREATE TABLE #{table}")['data'].try(:first).try(:first)
39
+ sql = do_system_execute("SHOW CREATE TABLE `#{table}`")['data'].try(:first).try(:first)
38
40
  { options: sql.gsub(/^(?:.*?)ENGINE = (.*?)$/, '\\1') }
39
41
  end
40
42
 
41
- # @todo copied from ActiveRecord::ConnectionAdapters::SchemaStatements v5.2.2
42
- # Why version column type of String, but insert to Integer?
43
- def assume_migrated_upto_version(version, migrations_paths)
44
- migrations_paths = Array(migrations_paths)
45
- version = version.to_i
46
- sm_table = quote_table_name(ActiveRecord::SchemaMigration.table_name)
47
-
48
- migrated = ActiveRecord::SchemaMigration.all_versions.map(&:to_i)
49
- versions = migration_context.migration_files.map do |file|
50
- migration_context.parse_migration_filename(file).first.to_i
51
- end
52
-
53
- unless migrated.include?(version)
54
- do_execute( "INSERT INTO #{sm_table} (version) VALUES (#{quote(version.to_s)})", 'SchemaMigration', format: nil)
55
- end
56
-
57
- inserting = (versions - migrated).select { |v| v < version }
58
- if inserting.any?
59
- if (duplicate = inserting.detect { |v| inserting.count(v) > 1 })
60
- raise "Duplicate migration #{duplicate}. Please renumber your migrations to resolve the conflict."
61
- end
62
- if supports_multi_insert?
63
- do_system_execute insert_versions_sql(inserting.map(&:to_s))
64
- else
65
- inserting.each do |v|
66
- do_system_execute insert_versions_sql(v)
67
- end
68
- end
69
- end
70
- end
71
-
72
43
  # Not indexes on clickhouse
73
44
  def indexes(table_name, name = nil)
74
45
  []
@@ -86,21 +57,22 @@ module ActiveRecord
86
57
  end
87
58
  end
88
59
 
89
- private
90
-
91
- def apply_format(sql, format)
92
- format ? "#{sql} FORMAT #{format}" : sql
93
- end
94
-
95
- def do_execute(sql, name = nil, format: 'JSONCompact')
60
+ def do_execute(sql, name = nil, format: 'JSONCompact', settings: {})
96
61
  log(sql, "#{adapter_name} #{name}") do
97
62
  formatted_sql = apply_format(sql, format)
98
- res = @connection.post("/?#{@config.to_param}", formatted_sql)
63
+ request_params = @config || {}
64
+ res = @connection.post("/?#{request_params.merge(settings).to_param}", formatted_sql)
99
65
 
100
66
  process_response(res)
101
67
  end
102
68
  end
103
69
 
70
+ private
71
+
72
+ def apply_format(sql, format)
73
+ format ? "#{sql} FORMAT #{format}" : sql
74
+ end
75
+
104
76
  def process_response(res)
105
77
  case res.code.to_i
106
78
  when 200
@@ -109,6 +81,8 @@ module ActiveRecord
109
81
  raise ActiveRecord::ActiveRecordError,
110
82
  "Response code: #{res.code}:\n#{res.body}"
111
83
  end
84
+ rescue JSON::ParserError
85
+ res.body
112
86
  end
113
87
 
114
88
  def log_with_debug(sql, name = nil)
@@ -121,7 +95,11 @@ module ActiveRecord
121
95
  end
122
96
 
123
97
  def create_table_definition(*args)
124
- Clickhouse::TableDefinition.new(*args)
98
+ if ActiveRecord::version >= Gem::Version.new('6')
99
+ Clickhouse::TableDefinition.new(self, *args)
100
+ else
101
+ Clickhouse::TableDefinition.new(*args)
102
+ end
125
103
  end
126
104
 
127
105
  def new_column_from_field(table_name, field)
@@ -130,13 +108,17 @@ module ActiveRecord
130
108
  default = field[3]
131
109
  default_value = extract_value_from_default(default)
132
110
  default_function = extract_default_function(default_value, default)
133
- ClickhouseColumn.new(field[0], default_value, type_metadata, field[1].include?('Nullable'), table_name, default_function)
111
+ if ActiveRecord::version >= Gem::Version.new('6')
112
+ ClickhouseColumn.new(field[0], default_value, type_metadata, field[1].include?('Nullable'), default_function)
113
+ else
114
+ ClickhouseColumn.new(field[0], default_value, type_metadata, field[1].include?('Nullable'), table_name, default_function)
115
+ end
134
116
  end
135
117
 
136
118
  protected
137
119
 
138
120
  def table_structure(table_name)
139
- result = do_system_execute("DESCRIBE TABLE #{table_name}", table_name)
121
+ result = do_system_execute("DESCRIBE TABLE `#{table_name}`", table_name)
140
122
  data = result['data']
141
123
 
142
124
  return data unless data.empty?
@@ -2,7 +2,6 @@
2
2
 
3
3
  require 'clickhouse-activerecord/arel/visitors/to_sql'
4
4
  require 'clickhouse-activerecord/arel/table'
5
- require 'active_record/connection_adapters/abstract_adapter'
6
5
  require 'active_record/connection_adapters/clickhouse/oid/date'
7
6
  require 'active_record/connection_adapters/clickhouse/oid/date_time'
8
7
  require 'active_record/connection_adapters/clickhouse/oid/big_integer'
@@ -19,6 +18,7 @@ module ActiveRecord
19
18
  config = config.symbolize_keys
20
19
  host = config[:host] || 'localhost'
21
20
  port = config[:port] || 8123
21
+ ssl = config[:ssl].present? ? config[:ssl] : port == 443
22
22
 
23
23
  if config.key?(:database)
24
24
  database = config[:database]
@@ -26,11 +26,26 @@ module ActiveRecord
26
26
  raise ArgumentError, 'No database specified. Missing argument: database.'
27
27
  end
28
28
 
29
- ConnectionAdapters::ClickhouseAdapter.new(nil, logger, [host, port], { user: config[:username], password: config[:password], database: database }.compact, config[:debug])
29
+ ConnectionAdapters::ClickhouseAdapter.new(logger, [host, port, ssl], { user: config[:username], password: config[:password], database: database }.compact, config)
30
30
  end
31
31
  end
32
32
  end
33
33
 
34
+ class Relation
35
+
36
+ # Replace for only ClickhouseAdapter
37
+ def reverse_order!
38
+ orders = order_values.uniq
39
+ orders.reject!(&:blank?)
40
+ if self.connection.is_a?(ConnectionAdapters::ClickhouseAdapter) && orders.empty? && !primary_key
41
+ self.order_values = %w(date created_at).select {|c| column_names.include?(c) }.map{|c| arel_attribute(c).desc }
42
+ else
43
+ self.order_values = reverse_sql_order(orders)
44
+ end
45
+ self
46
+ end
47
+ end
48
+
34
49
  module TypeCaster
35
50
  class Map
36
51
  def is_view
@@ -78,17 +93,34 @@ module ActiveRecord
78
93
  include Clickhouse::SchemaStatements
79
94
 
80
95
  # Initializes and connects a Clickhouse adapter.
81
- def initialize(connection, logger, connection_parameters, config, debug = false)
82
- super(connection, logger)
96
+ def initialize(logger, connection_parameters, config, full_config)
97
+ super(nil, logger)
83
98
  @connection_parameters = connection_parameters
84
99
  @config = config
85
- @debug = debug
100
+ @debug = full_config[:debug] || false
101
+ @full_config = full_config
86
102
 
87
103
  @prepared_statements = false
104
+ if ActiveRecord::version == Gem::Version.new('6.0.0')
105
+ @prepared_statement_status = Concurrent::ThreadLocalVar.new(false)
106
+ end
88
107
 
89
108
  connect
90
109
  end
91
110
 
111
+ # Support SchemaMigration from v5.2.2 to v6+
112
+ def schema_migration # :nodoc:
113
+ if ActiveRecord::version >= Gem::Version.new('6')
114
+ super
115
+ else
116
+ ActiveRecord::SchemaMigration
117
+ end
118
+ end
119
+
120
+ def migrations_paths
121
+ @full_config[:migrations_paths] || 'db/migrate_clickhouse'
122
+ end
123
+
92
124
  def arel_visitor # :nodoc:
93
125
  ClickhouseActiverecord::Arel::Visitors::ToSql.new(self)
94
126
  end
@@ -145,7 +177,11 @@ module ActiveRecord
145
177
  end
146
178
 
147
179
  def column_name_for_operation(operation, node) # :nodoc:
148
- column_name_from_arel_node(node)
180
+ if ActiveRecord::version >= Gem::Version.new('6')
181
+ visitor.compile(node)
182
+ else
183
+ column_name_from_arel_node(node)
184
+ end
149
185
  end
150
186
 
151
187
  # Executes insert +sql+ statement in the context of this connection using
@@ -166,16 +202,24 @@ module ActiveRecord
166
202
 
167
203
  # Create a new ClickHouse database.
168
204
  def create_database(name)
169
- sql = "CREATE DATABASE #{quote_table_name(name)}"
205
+ sql = apply_cluster "CREATE DATABASE #{quote_table_name(name)}"
170
206
  log_with_debug(sql, adapter_name) do
171
207
  res = @connection.post("/?#{@config.except(:database).to_param}", "CREATE DATABASE #{quote_table_name(name)}")
172
208
  process_response(res)
173
209
  end
174
210
  end
175
211
 
212
+ def create_table(table_name, comment: nil, **options)
213
+ super(
214
+ apply_cluster(table_name),
215
+ comment: comment,
216
+ **options
217
+ )
218
+ end
219
+
176
220
  # Drops a ClickHouse database.
177
221
  def drop_database(name) #:nodoc:
178
- sql = "DROP DATABASE IF EXISTS #{quote_table_name(name)}"
222
+ sql = apply_cluster "DROP DATABASE IF EXISTS #{quote_table_name(name)}"
179
223
  log_with_debug(sql, adapter_name) do
180
224
  res = @connection.post("/?#{@config.except(:database).to_param}", sql)
181
225
  process_response(res)
@@ -183,7 +227,7 @@ module ActiveRecord
183
227
  end
184
228
 
185
229
  def drop_table(table_name, options = {}) # :nodoc:
186
- do_execute "DROP TABLE#{' IF EXISTS' if options[:if_exists]} #{quote_table_name(table_name)}"
230
+ do_execute apply_cluster "DROP TABLE#{' IF EXISTS' if options[:if_exists]} #{quote_table_name(table_name)}"
187
231
  end
188
232
 
189
233
  protected
@@ -195,8 +239,15 @@ module ActiveRecord
195
239
  private
196
240
 
197
241
  def connect
198
- # for ssl port need use ssl
199
- @connection = Net::HTTP.start(@connection_parameters[0], @connection_parameters[1], use_ssl: @connection_parameters[1] == 443)
242
+ @connection = Net::HTTP.start(@connection_parameters[0], @connection_parameters[1], use_ssl: @connection_parameters[2], verify_mode: OpenSSL::SSL::VERIFY_NONE)
243
+ end
244
+
245
+ def cluster
246
+ @full_config[:cluster]
247
+ end
248
+
249
+ def apply_cluster(sql)
250
+ cluster ? "#{sql} ON CLUSTER #{cluster}" : sql
200
251
  end
201
252
  end
202
253
  end
@@ -3,7 +3,7 @@ module ClickhouseActiverecord
3
3
 
4
4
  def table(table, stream)
5
5
  stream.puts " # TABLE: #{table}"
6
- stream.puts " # SQL: #{@connection.do_system_execute("SHOW CREATE TABLE #{table.gsub(/^\.inner\./, '')}")['data'].try(:first).try(:first)}"
6
+ stream.puts " # SQL: #{@connection.do_system_execute("SHOW CREATE TABLE `#{table.gsub(/^\.inner\./, '')}`")['data'].try(:first).try(:first)}"
7
7
  super(table.gsub(/^\.inner\./, ''), stream)
8
8
  end
9
9
  end
@@ -31,13 +31,27 @@ module ClickhouseActiverecord
31
31
  create
32
32
  end
33
33
 
34
+ def structure_dump(*args)
35
+ tables = connection.execute("SHOW TABLES FROM #{@configuration['database']}")['data'].flatten
36
+
37
+ File.open(args.first, 'w:utf-8') do |file|
38
+ tables.each do |table|
39
+ next if table.match(/\.inner/)
40
+ file.puts connection.execute("SHOW CREATE TABLE #{table}")['data'].try(:first).try(:first).gsub("#{@configuration['database']}.", '') + ";\n\n"
41
+ end
42
+ end
43
+ end
44
+
45
+ def structure_load(*args)
46
+ File.read(args.first).split(";\n\n").each { |sql| connection.execute(sql) }
47
+ end
48
+
34
49
  def migrate
35
50
  check_target_version
36
51
 
37
52
  verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] != "false" : true
38
53
  scope = ENV["SCOPE"]
39
54
  verbose_was, ActiveRecord::Migration.verbose = ActiveRecord::Migration.verbose, verbose
40
- binding.pry
41
55
  connection.migration_context.migrate(target_version) do |migration|
42
56
  scope.blank? || scope == migration.scope
43
57
  end
@@ -1,3 +1,3 @@
1
1
  module ClickhouseActiverecord
2
- VERSION = '0.3.5'
2
+ VERSION = '0.3.13'
3
3
  end
@@ -6,6 +6,20 @@ class ClickhouseMigrationGenerator < ActiveRecord::Generators::MigrationGenerato
6
6
  def create_migration_file
7
7
  set_local_assigns!
8
8
  validate_file_name!
9
- migration_template @migration_template, "db/migrate_clickhouse/#{file_name}.rb"
9
+ migration_template @migration_template, File.join(db_migrate_path, "#{file_name}.rb")
10
+ end
11
+
12
+ private
13
+
14
+ def db_migrate_path
15
+ if defined?(Rails.application) && Rails.application
16
+ configured_migrate_path || default_migrate_path
17
+ else
18
+ default_migrate_path
19
+ end
20
+ end
21
+
22
+ def default_migrate_path
23
+ "db/migrate_clickhouse"
10
24
  end
11
25
  end
@@ -2,6 +2,70 @@
2
2
 
3
3
  namespace :clickhouse do
4
4
 
5
+ task prepare_schema_migration_table: :environment do
6
+ cluster, database, replica = ActiveRecord::Base.connection_config.values_at(:cluster, :database, :replica)
7
+ return if cluster.nil?
8
+
9
+ connection = ActiveRecord::Base.connection
10
+ key_options = connection.internal_string_options_for_primary_key
11
+ block = Proc.new do |t|
12
+ t.string :version, key_options
13
+ end
14
+ distributed_table_name = ".#{ActiveRecord::SchemaMigration.table_name}_distributed"
15
+ unless connection.table_exists?(distributed_table_name)
16
+ options = { id: false }
17
+ if replica
18
+ shard = replica.is_a?(String) ? replica : '{shard}'
19
+ options[:options] = <<-SQL
20
+ ReplicatedMergeTree('/clickhouse/tables/{cluster}/#{shard}/#{database}.`#{distributed_table_name}`', '{replica}')
21
+ PARTITION BY version ORDER BY (version) SETTINGS index_granularity = 8192
22
+ SQL
23
+ end
24
+ connection.create_table("`#{distributed_table_name}`", options, &block)
25
+ end
26
+ unless connection.table_exists?(ActiveRecord::SchemaMigration.table_name)
27
+ connection.create_table(
28
+ ActiveRecord::SchemaMigration.table_name,
29
+ id: false,
30
+ options: "Distributed(#{cluster},#{database},`#{distributed_table_name}`,sipHash64(version))",
31
+ &block
32
+ )
33
+ end
34
+ end
35
+
36
+ task prepare_internal_metadata_table: :environment do
37
+ cluster, database, replica = ActiveRecord::Base.connection_config.values_at(:cluster, :database, :replica)
38
+ return if cluster.nil?
39
+
40
+ connection = ActiveRecord::Base.connection
41
+ key_options = connection.internal_string_options_for_primary_key
42
+ block = Proc.new do |t|
43
+ t.string :key, key_options
44
+ t.string :value
45
+ t.timestamps
46
+ end
47
+ distributed_table_name = ".#{ActiveRecord::InternalMetadata.table_name}_distributed"
48
+ unless connection.table_exists?(distributed_table_name)
49
+ options = { id: false }
50
+ if replica
51
+ shard = replica.is_a?(String) ? replica : '{shard}'
52
+ options[:options] = <<-SQL
53
+ ReplicatedMergeTree('/clickhouse/tables/{cluster}/#{shard}/#{database}.`#{distributed_table_name}`', '{replica}')
54
+ PARTITION BY toDate(created_at) ORDER BY (created_at) SETTINGS index_granularity = 8192
55
+ SQL
56
+ end
57
+ connection.create_table("`#{distributed_table_name}`", options, &block)
58
+ end
59
+ unless connection.table_exists?(ActiveRecord::InternalMetadata.table_name)
60
+ connection.create_table(
61
+ ActiveRecord::InternalMetadata.table_name,
62
+ id: false,
63
+ options: "Distributed(#{cluster},#{database},`#{distributed_table_name}`,sipHash64(created_at))",
64
+ &block
65
+ )
66
+ end
67
+ end
68
+
5
69
  task load_config: :environment do
6
70
  ENV['SCHEMA'] = "db/clickhouse_schema.rb"
7
71
  ActiveRecord::Migrator.migrations_paths = ["db/migrate_clickhouse"]
@@ -27,6 +91,18 @@ namespace :clickhouse do
27
91
 
28
92
  end
29
93
 
94
+ namespace :structure do
95
+ desc 'Load database structure'
96
+ task load: [:load_config, 'db:check_protected_environments'] do
97
+ ClickhouseActiverecord::Tasks.new(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"]).structure_load("#{Rails.root}/db/clickhouse_structure.sql")
98
+ end
99
+
100
+ desc 'Dump database structure'
101
+ task dump: [:load_config, 'db:check_protected_environments'] do
102
+ ClickhouseActiverecord::Tasks.new(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"]).structure_dump("#{Rails.root}/db/clickhouse_structure.sql")
103
+ end
104
+ end
105
+
30
106
  desc 'Creates the database from DATABASE_URL or config/database.yml'
31
107
  task create: [:load_config] do
32
108
  ActiveRecord::Tasks::DatabaseTasks.create(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"])
@@ -49,7 +125,7 @@ namespace :clickhouse do
49
125
  end
50
126
 
51
127
  desc 'Migrate the clickhouse database'
52
- task migrate: :load_config do
128
+ task migrate: [:load_config, :prepare_schema_migration_table, :prepare_internal_metadata_table] do
53
129
  Rake::Task['db:migrate'].execute
54
130
  end
55
131
  end
metadata CHANGED
@@ -1,22 +1,19 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: clickhouse-activerecord
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.5
4
+ version: 0.3.13
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sergey Odintsov
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2019-07-01 00:00:00.000000000 Z
11
+ date: 2020-09-11 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - "~>"
18
- - !ruby/object:Gem::Version
19
- version: '1.13'
20
17
  - - ">="
21
18
  - !ruby/object:Gem::Version
22
19
  version: 1.13.4
@@ -24,9 +21,6 @@ dependencies:
24
21
  prerelease: false
25
22
  version_requirements: !ruby/object:Gem::Requirement
26
23
  requirements:
27
- - - "~>"
28
- - !ruby/object:Gem::Version
29
- version: '1.13'
30
24
  - - ">="
31
25
  - !ruby/object:Gem::Version
32
26
  version: 1.13.4
@@ -34,14 +28,14 @@ dependencies:
34
28
  name: activerecord
35
29
  requirement: !ruby/object:Gem::Requirement
36
30
  requirements:
37
- - - "~>"
31
+ - - ">="
38
32
  - !ruby/object:Gem::Version
39
33
  version: '5.2'
40
34
  type: :runtime
41
35
  prerelease: false
42
36
  version_requirements: !ruby/object:Gem::Requirement
43
37
  requirements:
44
- - - "~>"
38
+ - - ">="
45
39
  - !ruby/object:Gem::Version
46
40
  version: '5.2'
47
41
  - !ruby/object:Gem::Dependency
@@ -153,8 +147,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
153
147
  - !ruby/object:Gem::Version
154
148
  version: '0'
155
149
  requirements: []
156
- rubyforge_project:
157
- rubygems_version: 2.5.2.3
150
+ rubygems_version: 3.0.1
158
151
  signing_key:
159
152
  specification_version: 4
160
153
  summary: ClickHouse ActiveRecord