clickhouse-activerecord 0.3.5 → 0.3.13

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: f7b2fb3282fdff192d0cec9cb19a68ba70d02d83
4
- data.tar.gz: 50dea626b278f9ac8a33ea95d300a5d440fa229a
2
+ SHA256:
3
+ metadata.gz: 67ea4a63bb140adb5f6de102c38c097ea18f61499a882574952a9982b8d340cf
4
+ data.tar.gz: a1d2a759dae9062a20cc0a53bbd1cb64142062ce9ef84047f7e953b7d957e9b2
5
5
  SHA512:
6
- metadata.gz: 06a3b3ae02ca8e2e60974109f1ead35743340b8e12beefa28134cadeeff4df753db044da60e25af007a5e14abda14695fc1725d5b86aa5432dce7f642d2004ab
7
- data.tar.gz: 9d8215272de3a2b9c0bc2d4671cc22d769acdf1e3f8c32ea8d685c28a32be5838bb28dc7c77a8c4bbdaa7960df1537bf0aad7f8db7571c07793c85a64fbcb696
6
+ metadata.gz: 243b4d089d6349d8c12c529fbb80078ffbaa03f0033c3c69be77751846b031398895a1efdd08460699636277695a3e03eb228a398b5887a9426f4f3b63496c31
7
+ data.tar.gz: f790bca5af79492390de98ba2dce363e16207d987093c2e94e9f9ab55b9f769a41bae1c4a12d0b9d3757ee91f92604c1df4db17479dbbc85ced0f48294b297f3
@@ -1,17 +1,26 @@
1
+ ### Version 0.3.10 (Dec 20, 2019)
2
+
3
+ * Support structure dump/load [@StoneGod](https://github.com/StoneGod)
4
+
5
+ ### Version 0.3.6 (Sep 2, 2019)
6
+
7
+ * Support Rails 6.0
8
+ * Fix relation `last` method
9
+
1
10
  ### Version 0.3.4 (Jun 28, 2019)
2
11
 
3
- * Fix DateTime sql format without microseconds for Rails 5.2
12
+ * Fix DateTime sql format without microseconds for Rails 5.2
4
13
  * Support ssl connection
5
14
  * Migration support
6
15
  * Rake tasks for create / drop database
7
-
16
+
8
17
  ### Version 0.3.0 (Nov 27, 2018)
9
18
 
10
19
  * Support materialized view
11
20
  * Aggregated functions for view
12
21
  * Schema dumper with SQL create table
13
22
  * Added migrations support [@Bugagazavr](https://github.com/Bugagazavr)
14
-
23
+
15
24
  ### Version 0.2.0 (Oct 3, 2017)
16
25
 
17
26
  * Support Rails 5.0
@@ -19,7 +28,7 @@
19
28
  ### Version 0.1.2 (Sep 27, 2017)
20
29
 
21
30
  * Fix Big Int type
22
-
31
+
23
32
  ### Version 0.1.0 (Aug 31, 2017)
24
33
 
25
34
  * Initial release
data/README.md CHANGED
@@ -1,6 +1,7 @@
1
1
  # Clickhouse::Activerecord
2
2
 
3
3
  A Ruby database ActiveRecord driver for ClickHouse. Support Rails >= 5.2.
4
+ Tested on ClickHouse version 18.14.
4
5
 
5
6
  ## Installation
6
7
 
@@ -17,8 +18,24 @@ And then execute:
17
18
  Or install it yourself as:
18
19
 
19
20
  $ gem install clickhouse-activerecord
21
+
22
+ ## Available database connection parameters
23
+ ```yml
24
+ default: &default
25
+ adapter: clickhouse
26
+ database: database
27
+ host: localhost
28
+ port: 8123
29
+ username: username
30
+ password: password
31
+ ssl: true # optional for using ssl connection
32
+ debug: true # use for showing in to log technical information
33
+ migrations_paths: db/clickhouse # optional, default: db/migrate_clickhouse
34
+ cluster: 'cluster_name' # optional for creating tables in cluster
35
+ replica: '{shard}' # optional for creating system tables for shards
36
+ ```
20
37
 
21
- ## Usage
38
+ ## Usage in Rails 5
22
39
 
23
40
  Add your `database.yml` connection information with postfix `_clickhouse` for you environment:
24
41
 
@@ -26,10 +43,6 @@ Add your `database.yml` connection information with postfix `_clickhouse` for yo
26
43
  development_clickhouse:
27
44
  adapter: clickhouse
28
45
  database: database
29
- host: localhost
30
- username: username
31
- password: password
32
- debug: true # use for showing in to log technical information
33
46
  ```
34
47
 
35
48
  Add to your model:
@@ -54,19 +67,45 @@ Or global connection:
54
67
  development:
55
68
  adapter: clickhouse
56
69
  database: database
57
- host: localhost
58
- username: username
59
- password: password
70
+ ```
71
+
72
+ ## Usage in Rails 6 with second database
73
+
74
+ Add your `database.yml` connection information for you environment:
75
+
76
+ ```yml
77
+ development:
78
+ primary:
79
+ ...
80
+
81
+ clickhouse:
82
+ adapter: clickhouse
83
+ database: database
84
+ ```
85
+
86
+ Connection [Multiple Databases with Active Record](https://guides.rubyonrails.org/active_record_multiple_databases.html) or short example:
87
+
88
+ ```ruby
89
+ class Action < ActiveRecord::Base
90
+ connects_to database: { writing: :clickhouse, reading: :clickhouse }
91
+ end
60
92
  ```
61
93
 
62
94
  ### Rake tasks
63
95
 
96
+ **Note!** For Rails 6 you can use default rake tasks if you configure `migrations_paths` in your `database.yml`, for example: `rake db:migrate`
97
+
64
98
  Create / drop / purge / reset database:
65
99
 
66
100
  $ rake clickhouse:create
67
101
  $ rake clickhouse:drop
68
102
  $ rake clickhouse:purge
69
103
  $ rake clickhouse:reset
104
+
105
+ Prepare system tables for rails:
106
+
107
+ $ rake clickhouse:prepare_schema_migration_table
108
+ $ rake clickhouse:prepare_internal_metadata_table
70
109
 
71
110
  Migration:
72
111
 
@@ -75,6 +114,10 @@ Migration:
75
114
 
76
115
  Rollback migration not supported!
77
116
 
117
+ ### Dump / Load for multiple using databases
118
+
119
+ If you using multiple databases, for example: PostgreSQL, Clickhouse.
120
+
78
121
  Schema dump to `db/clickhouse_schema.rb` file:
79
122
 
80
123
  $ rake clickhouse:schema:dump
@@ -82,9 +125,24 @@ Schema dump to `db/clickhouse_schema.rb` file:
82
125
  Schema load from `db/clickhouse_schema.rb` file:
83
126
 
84
127
  $ rake clickhouse:schema:load
85
-
128
+
86
129
  We use schema for emulate development or tests environment on PostgreSQL adapter.
87
130
 
131
+ Structure dump to `db/clickhouse_structure.sql` file:
132
+
133
+ $ rake clickhouse:structure:dump
134
+
135
+ Structure load from `db/clickhouse_structure.sql` file:
136
+
137
+ $ rake clickhouse:structure:load
138
+
139
+ ### Dump / Load for only Clickhouse database using
140
+
141
+ $ rake db:schema:dump
142
+ $ rake db:schema:load
143
+ $ rake db:structure:dump
144
+ $ rake db:structure:load
145
+
88
146
  ### Insert and select data
89
147
 
90
148
  ```ruby
@@ -105,7 +163,7 @@ ActionView.maximum(:date)
105
163
 
106
164
  Donations to this project are going directly to [PNixx](https://github.com/PNixx), the original author of this project:
107
165
 
108
- * BTC address: `1Lx2gaJtzfF2dxGFxB65YtY5kNY9xUi6ia`
166
+ * BTC address: `1H3rhpf7WEF5JmMZ3PVFMQc7Hm29THgUfN`
109
167
  * ETH address: `0x6F094365A70fe7836A633d2eE80A1FA9758234d5`
110
168
  * XMR address: `42gP71qLB5M43RuDnrQ3vSJFFxis9Kw9VMURhpx9NLQRRwNvaZRjm2TFojAMC8Fk1BQhZNKyWhoyJSn5Ak9kppgZPjE17Zh`
111
169
 
@@ -23,8 +23,8 @@ Gem::Specification.new do |spec|
23
23
  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
24
24
  spec.require_paths = ['lib']
25
25
 
26
- spec.add_runtime_dependency 'bundler', '~> 1.13', '>= 1.13.4'
27
- spec.add_runtime_dependency 'activerecord', '~> 5.2'
26
+ spec.add_runtime_dependency 'bundler', '>= 1.13.4'
27
+ spec.add_runtime_dependency 'activerecord', '>= 5.2'
28
28
 
29
29
  spec.add_development_dependency 'bundler', '~> 1.15'
30
30
  spec.add_development_dependency 'rake', '~> 10.0'
@@ -8,6 +8,7 @@ module ActiveRecord
8
8
 
9
9
  def serialize(value)
10
10
  value = super
11
+ return unless value
11
12
  return value.strftime('%Y-%m-%d %H:%M:%S') unless value.acts_like?(:time)
12
13
 
13
14
  value.to_time.strftime('%Y-%m-%d %H:%M:%S')
@@ -17,6 +17,8 @@ module ActiveRecord
17
17
  def exec_query(sql, name = nil, binds = [], prepare: false)
18
18
  result = do_execute(sql, name)
19
19
  ActiveRecord::Result.new(result['meta'].map { |m| m['name'] }, result['data'])
20
+ rescue StandardError => _e
21
+ raise ActiveRecord::ActiveRecordError, "Response: #{result}"
20
22
  end
21
23
 
22
24
  def exec_update(_sql, _name = nil, _binds = [])
@@ -34,41 +36,10 @@ module ActiveRecord
34
36
  end
35
37
 
36
38
  def table_options(table)
37
- sql = do_system_execute("SHOW CREATE TABLE #{table}")['data'].try(:first).try(:first)
39
+ sql = do_system_execute("SHOW CREATE TABLE `#{table}`")['data'].try(:first).try(:first)
38
40
  { options: sql.gsub(/^(?:.*?)ENGINE = (.*?)$/, '\\1') }
39
41
  end
40
42
 
41
- # @todo copied from ActiveRecord::ConnectionAdapters::SchemaStatements v5.2.2
42
- # Why version column type of String, but insert to Integer?
43
- def assume_migrated_upto_version(version, migrations_paths)
44
- migrations_paths = Array(migrations_paths)
45
- version = version.to_i
46
- sm_table = quote_table_name(ActiveRecord::SchemaMigration.table_name)
47
-
48
- migrated = ActiveRecord::SchemaMigration.all_versions.map(&:to_i)
49
- versions = migration_context.migration_files.map do |file|
50
- migration_context.parse_migration_filename(file).first.to_i
51
- end
52
-
53
- unless migrated.include?(version)
54
- do_execute( "INSERT INTO #{sm_table} (version) VALUES (#{quote(version.to_s)})", 'SchemaMigration', format: nil)
55
- end
56
-
57
- inserting = (versions - migrated).select { |v| v < version }
58
- if inserting.any?
59
- if (duplicate = inserting.detect { |v| inserting.count(v) > 1 })
60
- raise "Duplicate migration #{duplicate}. Please renumber your migrations to resolve the conflict."
61
- end
62
- if supports_multi_insert?
63
- do_system_execute insert_versions_sql(inserting.map(&:to_s))
64
- else
65
- inserting.each do |v|
66
- do_system_execute insert_versions_sql(v)
67
- end
68
- end
69
- end
70
- end
71
-
72
43
  # Not indexes on clickhouse
73
44
  def indexes(table_name, name = nil)
74
45
  []
@@ -86,21 +57,22 @@ module ActiveRecord
86
57
  end
87
58
  end
88
59
 
89
- private
90
-
91
- def apply_format(sql, format)
92
- format ? "#{sql} FORMAT #{format}" : sql
93
- end
94
-
95
- def do_execute(sql, name = nil, format: 'JSONCompact')
60
+ def do_execute(sql, name = nil, format: 'JSONCompact', settings: {})
96
61
  log(sql, "#{adapter_name} #{name}") do
97
62
  formatted_sql = apply_format(sql, format)
98
- res = @connection.post("/?#{@config.to_param}", formatted_sql)
63
+ request_params = @config || {}
64
+ res = @connection.post("/?#{request_params.merge(settings).to_param}", formatted_sql)
99
65
 
100
66
  process_response(res)
101
67
  end
102
68
  end
103
69
 
70
+ private
71
+
72
+ def apply_format(sql, format)
73
+ format ? "#{sql} FORMAT #{format}" : sql
74
+ end
75
+
104
76
  def process_response(res)
105
77
  case res.code.to_i
106
78
  when 200
@@ -109,6 +81,8 @@ module ActiveRecord
109
81
  raise ActiveRecord::ActiveRecordError,
110
82
  "Response code: #{res.code}:\n#{res.body}"
111
83
  end
84
+ rescue JSON::ParserError
85
+ res.body
112
86
  end
113
87
 
114
88
  def log_with_debug(sql, name = nil)
@@ -121,7 +95,11 @@ module ActiveRecord
121
95
  end
122
96
 
123
97
  def create_table_definition(*args)
124
- Clickhouse::TableDefinition.new(*args)
98
+ if ActiveRecord::version >= Gem::Version.new('6')
99
+ Clickhouse::TableDefinition.new(self, *args)
100
+ else
101
+ Clickhouse::TableDefinition.new(*args)
102
+ end
125
103
  end
126
104
 
127
105
  def new_column_from_field(table_name, field)
@@ -130,13 +108,17 @@ module ActiveRecord
130
108
  default = field[3]
131
109
  default_value = extract_value_from_default(default)
132
110
  default_function = extract_default_function(default_value, default)
133
- ClickhouseColumn.new(field[0], default_value, type_metadata, field[1].include?('Nullable'), table_name, default_function)
111
+ if ActiveRecord::version >= Gem::Version.new('6')
112
+ ClickhouseColumn.new(field[0], default_value, type_metadata, field[1].include?('Nullable'), default_function)
113
+ else
114
+ ClickhouseColumn.new(field[0], default_value, type_metadata, field[1].include?('Nullable'), table_name, default_function)
115
+ end
134
116
  end
135
117
 
136
118
  protected
137
119
 
138
120
  def table_structure(table_name)
139
- result = do_system_execute("DESCRIBE TABLE #{table_name}", table_name)
121
+ result = do_system_execute("DESCRIBE TABLE `#{table_name}`", table_name)
140
122
  data = result['data']
141
123
 
142
124
  return data unless data.empty?
@@ -2,7 +2,6 @@
2
2
 
3
3
  require 'clickhouse-activerecord/arel/visitors/to_sql'
4
4
  require 'clickhouse-activerecord/arel/table'
5
- require 'active_record/connection_adapters/abstract_adapter'
6
5
  require 'active_record/connection_adapters/clickhouse/oid/date'
7
6
  require 'active_record/connection_adapters/clickhouse/oid/date_time'
8
7
  require 'active_record/connection_adapters/clickhouse/oid/big_integer'
@@ -19,6 +18,7 @@ module ActiveRecord
19
18
  config = config.symbolize_keys
20
19
  host = config[:host] || 'localhost'
21
20
  port = config[:port] || 8123
21
+ ssl = config[:ssl].present? ? config[:ssl] : port == 443
22
22
 
23
23
  if config.key?(:database)
24
24
  database = config[:database]
@@ -26,11 +26,26 @@ module ActiveRecord
26
26
  raise ArgumentError, 'No database specified. Missing argument: database.'
27
27
  end
28
28
 
29
- ConnectionAdapters::ClickhouseAdapter.new(nil, logger, [host, port], { user: config[:username], password: config[:password], database: database }.compact, config[:debug])
29
+ ConnectionAdapters::ClickhouseAdapter.new(logger, [host, port, ssl], { user: config[:username], password: config[:password], database: database }.compact, config)
30
30
  end
31
31
  end
32
32
  end
33
33
 
34
+ class Relation
35
+
36
+ # Replace for only ClickhouseAdapter
37
+ def reverse_order!
38
+ orders = order_values.uniq
39
+ orders.reject!(&:blank?)
40
+ if self.connection.is_a?(ConnectionAdapters::ClickhouseAdapter) && orders.empty? && !primary_key
41
+ self.order_values = %w(date created_at).select {|c| column_names.include?(c) }.map{|c| arel_attribute(c).desc }
42
+ else
43
+ self.order_values = reverse_sql_order(orders)
44
+ end
45
+ self
46
+ end
47
+ end
48
+
34
49
  module TypeCaster
35
50
  class Map
36
51
  def is_view
@@ -78,17 +93,34 @@ module ActiveRecord
78
93
  include Clickhouse::SchemaStatements
79
94
 
80
95
  # Initializes and connects a Clickhouse adapter.
81
- def initialize(connection, logger, connection_parameters, config, debug = false)
82
- super(connection, logger)
96
+ def initialize(logger, connection_parameters, config, full_config)
97
+ super(nil, logger)
83
98
  @connection_parameters = connection_parameters
84
99
  @config = config
85
- @debug = debug
100
+ @debug = full_config[:debug] || false
101
+ @full_config = full_config
86
102
 
87
103
  @prepared_statements = false
104
+ if ActiveRecord::version == Gem::Version.new('6.0.0')
105
+ @prepared_statement_status = Concurrent::ThreadLocalVar.new(false)
106
+ end
88
107
 
89
108
  connect
90
109
  end
91
110
 
111
+ # Support SchemaMigration from v5.2.2 to v6+
112
+ def schema_migration # :nodoc:
113
+ if ActiveRecord::version >= Gem::Version.new('6')
114
+ super
115
+ else
116
+ ActiveRecord::SchemaMigration
117
+ end
118
+ end
119
+
120
+ def migrations_paths
121
+ @full_config[:migrations_paths] || 'db/migrate_clickhouse'
122
+ end
123
+
92
124
  def arel_visitor # :nodoc:
93
125
  ClickhouseActiverecord::Arel::Visitors::ToSql.new(self)
94
126
  end
@@ -145,7 +177,11 @@ module ActiveRecord
145
177
  end
146
178
 
147
179
  def column_name_for_operation(operation, node) # :nodoc:
148
- column_name_from_arel_node(node)
180
+ if ActiveRecord::version >= Gem::Version.new('6')
181
+ visitor.compile(node)
182
+ else
183
+ column_name_from_arel_node(node)
184
+ end
149
185
  end
150
186
 
151
187
  # Executes insert +sql+ statement in the context of this connection using
@@ -166,16 +202,24 @@ module ActiveRecord
166
202
 
167
203
  # Create a new ClickHouse database.
168
204
  def create_database(name)
169
- sql = "CREATE DATABASE #{quote_table_name(name)}"
205
+ sql = apply_cluster "CREATE DATABASE #{quote_table_name(name)}"
170
206
  log_with_debug(sql, adapter_name) do
171
207
  res = @connection.post("/?#{@config.except(:database).to_param}", "CREATE DATABASE #{quote_table_name(name)}")
172
208
  process_response(res)
173
209
  end
174
210
  end
175
211
 
212
+ def create_table(table_name, comment: nil, **options)
213
+ super(
214
+ apply_cluster(table_name),
215
+ comment: comment,
216
+ **options
217
+ )
218
+ end
219
+
176
220
  # Drops a ClickHouse database.
177
221
  def drop_database(name) #:nodoc:
178
- sql = "DROP DATABASE IF EXISTS #{quote_table_name(name)}"
222
+ sql = apply_cluster "DROP DATABASE IF EXISTS #{quote_table_name(name)}"
179
223
  log_with_debug(sql, adapter_name) do
180
224
  res = @connection.post("/?#{@config.except(:database).to_param}", sql)
181
225
  process_response(res)
@@ -183,7 +227,7 @@ module ActiveRecord
183
227
  end
184
228
 
185
229
  def drop_table(table_name, options = {}) # :nodoc:
186
- do_execute "DROP TABLE#{' IF EXISTS' if options[:if_exists]} #{quote_table_name(table_name)}"
230
+ do_execute apply_cluster "DROP TABLE#{' IF EXISTS' if options[:if_exists]} #{quote_table_name(table_name)}"
187
231
  end
188
232
 
189
233
  protected
@@ -195,8 +239,15 @@ module ActiveRecord
195
239
  private
196
240
 
197
241
  def connect
198
- # for ssl port need use ssl
199
- @connection = Net::HTTP.start(@connection_parameters[0], @connection_parameters[1], use_ssl: @connection_parameters[1] == 443)
242
+ @connection = Net::HTTP.start(@connection_parameters[0], @connection_parameters[1], use_ssl: @connection_parameters[2], verify_mode: OpenSSL::SSL::VERIFY_NONE)
243
+ end
244
+
245
+ def cluster
246
+ @full_config[:cluster]
247
+ end
248
+
249
+ def apply_cluster(sql)
250
+ cluster ? "#{sql} ON CLUSTER #{cluster}" : sql
200
251
  end
201
252
  end
202
253
  end
@@ -3,7 +3,7 @@ module ClickhouseActiverecord
3
3
 
4
4
  def table(table, stream)
5
5
  stream.puts " # TABLE: #{table}"
6
- stream.puts " # SQL: #{@connection.do_system_execute("SHOW CREATE TABLE #{table.gsub(/^\.inner\./, '')}")['data'].try(:first).try(:first)}"
6
+ stream.puts " # SQL: #{@connection.do_system_execute("SHOW CREATE TABLE `#{table.gsub(/^\.inner\./, '')}`")['data'].try(:first).try(:first)}"
7
7
  super(table.gsub(/^\.inner\./, ''), stream)
8
8
  end
9
9
  end
@@ -31,13 +31,27 @@ module ClickhouseActiverecord
31
31
  create
32
32
  end
33
33
 
34
+ def structure_dump(*args)
35
+ tables = connection.execute("SHOW TABLES FROM #{@configuration['database']}")['data'].flatten
36
+
37
+ File.open(args.first, 'w:utf-8') do |file|
38
+ tables.each do |table|
39
+ next if table.match(/\.inner/)
40
+ file.puts connection.execute("SHOW CREATE TABLE #{table}")['data'].try(:first).try(:first).gsub("#{@configuration['database']}.", '') + ";\n\n"
41
+ end
42
+ end
43
+ end
44
+
45
+ def structure_load(*args)
46
+ File.read(args.first).split(";\n\n").each { |sql| connection.execute(sql) }
47
+ end
48
+
34
49
  def migrate
35
50
  check_target_version
36
51
 
37
52
  verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] != "false" : true
38
53
  scope = ENV["SCOPE"]
39
54
  verbose_was, ActiveRecord::Migration.verbose = ActiveRecord::Migration.verbose, verbose
40
- binding.pry
41
55
  connection.migration_context.migrate(target_version) do |migration|
42
56
  scope.blank? || scope == migration.scope
43
57
  end
@@ -1,3 +1,3 @@
1
1
  module ClickhouseActiverecord
2
- VERSION = '0.3.5'
2
+ VERSION = '0.3.13'
3
3
  end
@@ -6,6 +6,20 @@ class ClickhouseMigrationGenerator < ActiveRecord::Generators::MigrationGenerato
6
6
  def create_migration_file
7
7
  set_local_assigns!
8
8
  validate_file_name!
9
- migration_template @migration_template, "db/migrate_clickhouse/#{file_name}.rb"
9
+ migration_template @migration_template, File.join(db_migrate_path, "#{file_name}.rb")
10
+ end
11
+
12
+ private
13
+
14
+ def db_migrate_path
15
+ if defined?(Rails.application) && Rails.application
16
+ configured_migrate_path || default_migrate_path
17
+ else
18
+ default_migrate_path
19
+ end
20
+ end
21
+
22
+ def default_migrate_path
23
+ "db/migrate_clickhouse"
10
24
  end
11
25
  end
@@ -2,6 +2,70 @@
2
2
 
3
3
  namespace :clickhouse do
4
4
 
5
+ task prepare_schema_migration_table: :environment do
6
+ cluster, database, replica = ActiveRecord::Base.connection_config.values_at(:cluster, :database, :replica)
7
+ return if cluster.nil?
8
+
9
+ connection = ActiveRecord::Base.connection
10
+ key_options = connection.internal_string_options_for_primary_key
11
+ block = Proc.new do |t|
12
+ t.string :version, key_options
13
+ end
14
+ distributed_table_name = ".#{ActiveRecord::SchemaMigration.table_name}_distributed"
15
+ unless connection.table_exists?(distributed_table_name)
16
+ options = { id: false }
17
+ if replica
18
+ shard = replica.is_a?(String) ? replica : '{shard}'
19
+ options[:options] = <<-SQL
20
+ ReplicatedMergeTree('/clickhouse/tables/{cluster}/#{shard}/#{database}.`#{distributed_table_name}`', '{replica}')
21
+ PARTITION BY version ORDER BY (version) SETTINGS index_granularity = 8192
22
+ SQL
23
+ end
24
+ connection.create_table("`#{distributed_table_name}`", options, &block)
25
+ end
26
+ unless connection.table_exists?(ActiveRecord::SchemaMigration.table_name)
27
+ connection.create_table(
28
+ ActiveRecord::SchemaMigration.table_name,
29
+ id: false,
30
+ options: "Distributed(#{cluster},#{database},`#{distributed_table_name}`,sipHash64(version))",
31
+ &block
32
+ )
33
+ end
34
+ end
35
+
36
+ task prepare_internal_metadata_table: :environment do
37
+ cluster, database, replica = ActiveRecord::Base.connection_config.values_at(:cluster, :database, :replica)
38
+ return if cluster.nil?
39
+
40
+ connection = ActiveRecord::Base.connection
41
+ key_options = connection.internal_string_options_for_primary_key
42
+ block = Proc.new do |t|
43
+ t.string :key, key_options
44
+ t.string :value
45
+ t.timestamps
46
+ end
47
+ distributed_table_name = ".#{ActiveRecord::InternalMetadata.table_name}_distributed"
48
+ unless connection.table_exists?(distributed_table_name)
49
+ options = { id: false }
50
+ if replica
51
+ shard = replica.is_a?(String) ? replica : '{shard}'
52
+ options[:options] = <<-SQL
53
+ ReplicatedMergeTree('/clickhouse/tables/{cluster}/#{shard}/#{database}.`#{distributed_table_name}`', '{replica}')
54
+ PARTITION BY toDate(created_at) ORDER BY (created_at) SETTINGS index_granularity = 8192
55
+ SQL
56
+ end
57
+ connection.create_table("`#{distributed_table_name}`", options, &block)
58
+ end
59
+ unless connection.table_exists?(ActiveRecord::InternalMetadata.table_name)
60
+ connection.create_table(
61
+ ActiveRecord::InternalMetadata.table_name,
62
+ id: false,
63
+ options: "Distributed(#{cluster},#{database},`#{distributed_table_name}`,sipHash64(created_at))",
64
+ &block
65
+ )
66
+ end
67
+ end
68
+
5
69
  task load_config: :environment do
6
70
  ENV['SCHEMA'] = "db/clickhouse_schema.rb"
7
71
  ActiveRecord::Migrator.migrations_paths = ["db/migrate_clickhouse"]
@@ -27,6 +91,18 @@ namespace :clickhouse do
27
91
 
28
92
  end
29
93
 
94
+ namespace :structure do
95
+ desc 'Load database structure'
96
+ task load: [:load_config, 'db:check_protected_environments'] do
97
+ ClickhouseActiverecord::Tasks.new(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"]).structure_load("#{Rails.root}/db/clickhouse_structure.sql")
98
+ end
99
+
100
+ desc 'Dump database structure'
101
+ task dump: [:load_config, 'db:check_protected_environments'] do
102
+ ClickhouseActiverecord::Tasks.new(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"]).structure_dump("#{Rails.root}/db/clickhouse_structure.sql")
103
+ end
104
+ end
105
+
30
106
  desc 'Creates the database from DATABASE_URL or config/database.yml'
31
107
  task create: [:load_config] do
32
108
  ActiveRecord::Tasks::DatabaseTasks.create(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"])
@@ -49,7 +125,7 @@ namespace :clickhouse do
49
125
  end
50
126
 
51
127
  desc 'Migrate the clickhouse database'
52
- task migrate: :load_config do
128
+ task migrate: [:load_config, :prepare_schema_migration_table, :prepare_internal_metadata_table] do
53
129
  Rake::Task['db:migrate'].execute
54
130
  end
55
131
  end
metadata CHANGED
@@ -1,22 +1,19 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: clickhouse-activerecord
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.5
4
+ version: 0.3.13
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sergey Odintsov
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2019-07-01 00:00:00.000000000 Z
11
+ date: 2020-09-11 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - "~>"
18
- - !ruby/object:Gem::Version
19
- version: '1.13'
20
17
  - - ">="
21
18
  - !ruby/object:Gem::Version
22
19
  version: 1.13.4
@@ -24,9 +21,6 @@ dependencies:
24
21
  prerelease: false
25
22
  version_requirements: !ruby/object:Gem::Requirement
26
23
  requirements:
27
- - - "~>"
28
- - !ruby/object:Gem::Version
29
- version: '1.13'
30
24
  - - ">="
31
25
  - !ruby/object:Gem::Version
32
26
  version: 1.13.4
@@ -34,14 +28,14 @@ dependencies:
34
28
  name: activerecord
35
29
  requirement: !ruby/object:Gem::Requirement
36
30
  requirements:
37
- - - "~>"
31
+ - - ">="
38
32
  - !ruby/object:Gem::Version
39
33
  version: '5.2'
40
34
  type: :runtime
41
35
  prerelease: false
42
36
  version_requirements: !ruby/object:Gem::Requirement
43
37
  requirements:
44
- - - "~>"
38
+ - - ">="
45
39
  - !ruby/object:Gem::Version
46
40
  version: '5.2'
47
41
  - !ruby/object:Gem::Dependency
@@ -153,8 +147,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
153
147
  - !ruby/object:Gem::Version
154
148
  version: '0'
155
149
  requirements: []
156
- rubyforge_project:
157
- rubygems_version: 2.5.2.3
150
+ rubygems_version: 3.0.1
158
151
  signing_key:
159
152
  specification_version: 4
160
153
  summary: ClickHouse ActiveRecord