clickhouse-activerecord 0.3.5 → 0.3.13
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/CHANGELOG.md +13 -4
- data/README.md +68 -10
- data/clickhouse-activerecord.gemspec +2 -2
- data/lib/active_record/connection_adapters/clickhouse/oid/date_time.rb +1 -0
- data/lib/active_record/connection_adapters/clickhouse/schema_statements.rb +25 -43
- data/lib/active_record/connection_adapters/clickhouse_adapter.rb +62 -11
- data/lib/clickhouse-activerecord/schema_dumper.rb +1 -1
- data/lib/clickhouse-activerecord/tasks.rb +15 -1
- data/lib/clickhouse-activerecord/version.rb +1 -1
- data/lib/generators/clickhouse_migration_generator.rb +15 -1
- data/lib/tasks/clickhouse.rake +77 -1
- metadata +5 -12
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 67ea4a63bb140adb5f6de102c38c097ea18f61499a882574952a9982b8d340cf
|
4
|
+
data.tar.gz: a1d2a759dae9062a20cc0a53bbd1cb64142062ce9ef84047f7e953b7d957e9b2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 243b4d089d6349d8c12c529fbb80078ffbaa03f0033c3c69be77751846b031398895a1efdd08460699636277695a3e03eb228a398b5887a9426f4f3b63496c31
|
7
|
+
data.tar.gz: f790bca5af79492390de98ba2dce363e16207d987093c2e94e9f9ab55b9f769a41bae1c4a12d0b9d3757ee91f92604c1df4db17479dbbc85ced0f48294b297f3
|
data/CHANGELOG.md
CHANGED
@@ -1,17 +1,26 @@
|
|
1
|
+
### Version 0.3.10 (Dec 20, 2019)
|
2
|
+
|
3
|
+
* Support structure dump/load [@StoneGod](https://github.com/StoneGod)
|
4
|
+
|
5
|
+
### Version 0.3.6 (Sep 2, 2019)
|
6
|
+
|
7
|
+
* Support Rails 6.0
|
8
|
+
* Fix relation `last` method
|
9
|
+
|
1
10
|
### Version 0.3.4 (Jun 28, 2019)
|
2
11
|
|
3
|
-
* Fix DateTime sql format without microseconds for Rails 5.2
|
12
|
+
* Fix DateTime sql format without microseconds for Rails 5.2
|
4
13
|
* Support ssl connection
|
5
14
|
* Migration support
|
6
15
|
* Rake tasks for create / drop database
|
7
|
-
|
16
|
+
|
8
17
|
### Version 0.3.0 (Nov 27, 2018)
|
9
18
|
|
10
19
|
* Support materialized view
|
11
20
|
* Aggregated functions for view
|
12
21
|
* Schema dumper with SQL create table
|
13
22
|
* Added migrations support [@Bugagazavr](https://github.com/Bugagazavr)
|
14
|
-
|
23
|
+
|
15
24
|
### Version 0.2.0 (Oct 3, 2017)
|
16
25
|
|
17
26
|
* Support Rails 5.0
|
@@ -19,7 +28,7 @@
|
|
19
28
|
### Version 0.1.2 (Sep 27, 2017)
|
20
29
|
|
21
30
|
* Fix Big Int type
|
22
|
-
|
31
|
+
|
23
32
|
### Version 0.1.0 (Aug 31, 2017)
|
24
33
|
|
25
34
|
* Initial release
|
data/README.md
CHANGED
@@ -1,6 +1,7 @@
|
|
1
1
|
# Clickhouse::Activerecord
|
2
2
|
|
3
3
|
A Ruby database ActiveRecord driver for ClickHouse. Support Rails >= 5.2.
|
4
|
+
Tested on ClickHouse version 18.14.
|
4
5
|
|
5
6
|
## Installation
|
6
7
|
|
@@ -17,8 +18,24 @@ And then execute:
|
|
17
18
|
Or install it yourself as:
|
18
19
|
|
19
20
|
$ gem install clickhouse-activerecord
|
21
|
+
|
22
|
+
## Available database connection parameters
|
23
|
+
```yml
|
24
|
+
default: &default
|
25
|
+
adapter: clickhouse
|
26
|
+
database: database
|
27
|
+
host: localhost
|
28
|
+
port: 8123
|
29
|
+
username: username
|
30
|
+
password: password
|
31
|
+
ssl: true # optional for using ssl connection
|
32
|
+
debug: true # use for showing in to log technical information
|
33
|
+
migrations_paths: db/clickhouse # optional, default: db/migrate_clickhouse
|
34
|
+
cluster: 'cluster_name' # optional for creating tables in cluster
|
35
|
+
replica: '{shard}' # optional for creating system tables for shards
|
36
|
+
```
|
20
37
|
|
21
|
-
## Usage
|
38
|
+
## Usage in Rails 5
|
22
39
|
|
23
40
|
Add your `database.yml` connection information with postfix `_clickhouse` for you environment:
|
24
41
|
|
@@ -26,10 +43,6 @@ Add your `database.yml` connection information with postfix `_clickhouse` for yo
|
|
26
43
|
development_clickhouse:
|
27
44
|
adapter: clickhouse
|
28
45
|
database: database
|
29
|
-
host: localhost
|
30
|
-
username: username
|
31
|
-
password: password
|
32
|
-
debug: true # use for showing in to log technical information
|
33
46
|
```
|
34
47
|
|
35
48
|
Add to your model:
|
@@ -54,19 +67,45 @@ Or global connection:
|
|
54
67
|
development:
|
55
68
|
adapter: clickhouse
|
56
69
|
database: database
|
57
|
-
|
58
|
-
|
59
|
-
|
70
|
+
```
|
71
|
+
|
72
|
+
## Usage in Rails 6 with second database
|
73
|
+
|
74
|
+
Add your `database.yml` connection information for you environment:
|
75
|
+
|
76
|
+
```yml
|
77
|
+
development:
|
78
|
+
primary:
|
79
|
+
...
|
80
|
+
|
81
|
+
clickhouse:
|
82
|
+
adapter: clickhouse
|
83
|
+
database: database
|
84
|
+
```
|
85
|
+
|
86
|
+
Connection [Multiple Databases with Active Record](https://guides.rubyonrails.org/active_record_multiple_databases.html) or short example:
|
87
|
+
|
88
|
+
```ruby
|
89
|
+
class Action < ActiveRecord::Base
|
90
|
+
connects_to database: { writing: :clickhouse, reading: :clickhouse }
|
91
|
+
end
|
60
92
|
```
|
61
93
|
|
62
94
|
### Rake tasks
|
63
95
|
|
96
|
+
**Note!** For Rails 6 you can use default rake tasks if you configure `migrations_paths` in your `database.yml`, for example: `rake db:migrate`
|
97
|
+
|
64
98
|
Create / drop / purge / reset database:
|
65
99
|
|
66
100
|
$ rake clickhouse:create
|
67
101
|
$ rake clickhouse:drop
|
68
102
|
$ rake clickhouse:purge
|
69
103
|
$ rake clickhouse:reset
|
104
|
+
|
105
|
+
Prepare system tables for rails:
|
106
|
+
|
107
|
+
$ rake clickhouse:prepare_schema_migration_table
|
108
|
+
$ rake clickhouse:prepare_internal_metadata_table
|
70
109
|
|
71
110
|
Migration:
|
72
111
|
|
@@ -75,6 +114,10 @@ Migration:
|
|
75
114
|
|
76
115
|
Rollback migration not supported!
|
77
116
|
|
117
|
+
### Dump / Load for multiple using databases
|
118
|
+
|
119
|
+
If you using multiple databases, for example: PostgreSQL, Clickhouse.
|
120
|
+
|
78
121
|
Schema dump to `db/clickhouse_schema.rb` file:
|
79
122
|
|
80
123
|
$ rake clickhouse:schema:dump
|
@@ -82,9 +125,24 @@ Schema dump to `db/clickhouse_schema.rb` file:
|
|
82
125
|
Schema load from `db/clickhouse_schema.rb` file:
|
83
126
|
|
84
127
|
$ rake clickhouse:schema:load
|
85
|
-
|
128
|
+
|
86
129
|
We use schema for emulate development or tests environment on PostgreSQL adapter.
|
87
130
|
|
131
|
+
Structure dump to `db/clickhouse_structure.sql` file:
|
132
|
+
|
133
|
+
$ rake clickhouse:structure:dump
|
134
|
+
|
135
|
+
Structure load from `db/clickhouse_structure.sql` file:
|
136
|
+
|
137
|
+
$ rake clickhouse:structure:load
|
138
|
+
|
139
|
+
### Dump / Load for only Clickhouse database using
|
140
|
+
|
141
|
+
$ rake db:schema:dump
|
142
|
+
$ rake db:schema:load
|
143
|
+
$ rake db:structure:dump
|
144
|
+
$ rake db:structure:load
|
145
|
+
|
88
146
|
### Insert and select data
|
89
147
|
|
90
148
|
```ruby
|
@@ -105,7 +163,7 @@ ActionView.maximum(:date)
|
|
105
163
|
|
106
164
|
Donations to this project are going directly to [PNixx](https://github.com/PNixx), the original author of this project:
|
107
165
|
|
108
|
-
* BTC address: `
|
166
|
+
* BTC address: `1H3rhpf7WEF5JmMZ3PVFMQc7Hm29THgUfN`
|
109
167
|
* ETH address: `0x6F094365A70fe7836A633d2eE80A1FA9758234d5`
|
110
168
|
* XMR address: `42gP71qLB5M43RuDnrQ3vSJFFxis9Kw9VMURhpx9NLQRRwNvaZRjm2TFojAMC8Fk1BQhZNKyWhoyJSn5Ak9kppgZPjE17Zh`
|
111
169
|
|
@@ -23,8 +23,8 @@ Gem::Specification.new do |spec|
|
|
23
23
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
24
24
|
spec.require_paths = ['lib']
|
25
25
|
|
26
|
-
spec.add_runtime_dependency 'bundler', '
|
27
|
-
spec.add_runtime_dependency 'activerecord', '
|
26
|
+
spec.add_runtime_dependency 'bundler', '>= 1.13.4'
|
27
|
+
spec.add_runtime_dependency 'activerecord', '>= 5.2'
|
28
28
|
|
29
29
|
spec.add_development_dependency 'bundler', '~> 1.15'
|
30
30
|
spec.add_development_dependency 'rake', '~> 10.0'
|
@@ -17,6 +17,8 @@ module ActiveRecord
|
|
17
17
|
def exec_query(sql, name = nil, binds = [], prepare: false)
|
18
18
|
result = do_execute(sql, name)
|
19
19
|
ActiveRecord::Result.new(result['meta'].map { |m| m['name'] }, result['data'])
|
20
|
+
rescue StandardError => _e
|
21
|
+
raise ActiveRecord::ActiveRecordError, "Response: #{result}"
|
20
22
|
end
|
21
23
|
|
22
24
|
def exec_update(_sql, _name = nil, _binds = [])
|
@@ -34,41 +36,10 @@ module ActiveRecord
|
|
34
36
|
end
|
35
37
|
|
36
38
|
def table_options(table)
|
37
|
-
sql = do_system_execute("SHOW CREATE TABLE
|
39
|
+
sql = do_system_execute("SHOW CREATE TABLE `#{table}`")['data'].try(:first).try(:first)
|
38
40
|
{ options: sql.gsub(/^(?:.*?)ENGINE = (.*?)$/, '\\1') }
|
39
41
|
end
|
40
42
|
|
41
|
-
# @todo copied from ActiveRecord::ConnectionAdapters::SchemaStatements v5.2.2
|
42
|
-
# Why version column type of String, but insert to Integer?
|
43
|
-
def assume_migrated_upto_version(version, migrations_paths)
|
44
|
-
migrations_paths = Array(migrations_paths)
|
45
|
-
version = version.to_i
|
46
|
-
sm_table = quote_table_name(ActiveRecord::SchemaMigration.table_name)
|
47
|
-
|
48
|
-
migrated = ActiveRecord::SchemaMigration.all_versions.map(&:to_i)
|
49
|
-
versions = migration_context.migration_files.map do |file|
|
50
|
-
migration_context.parse_migration_filename(file).first.to_i
|
51
|
-
end
|
52
|
-
|
53
|
-
unless migrated.include?(version)
|
54
|
-
do_execute( "INSERT INTO #{sm_table} (version) VALUES (#{quote(version.to_s)})", 'SchemaMigration', format: nil)
|
55
|
-
end
|
56
|
-
|
57
|
-
inserting = (versions - migrated).select { |v| v < version }
|
58
|
-
if inserting.any?
|
59
|
-
if (duplicate = inserting.detect { |v| inserting.count(v) > 1 })
|
60
|
-
raise "Duplicate migration #{duplicate}. Please renumber your migrations to resolve the conflict."
|
61
|
-
end
|
62
|
-
if supports_multi_insert?
|
63
|
-
do_system_execute insert_versions_sql(inserting.map(&:to_s))
|
64
|
-
else
|
65
|
-
inserting.each do |v|
|
66
|
-
do_system_execute insert_versions_sql(v)
|
67
|
-
end
|
68
|
-
end
|
69
|
-
end
|
70
|
-
end
|
71
|
-
|
72
43
|
# Not indexes on clickhouse
|
73
44
|
def indexes(table_name, name = nil)
|
74
45
|
[]
|
@@ -86,21 +57,22 @@ module ActiveRecord
|
|
86
57
|
end
|
87
58
|
end
|
88
59
|
|
89
|
-
|
90
|
-
|
91
|
-
def apply_format(sql, format)
|
92
|
-
format ? "#{sql} FORMAT #{format}" : sql
|
93
|
-
end
|
94
|
-
|
95
|
-
def do_execute(sql, name = nil, format: 'JSONCompact')
|
60
|
+
def do_execute(sql, name = nil, format: 'JSONCompact', settings: {})
|
96
61
|
log(sql, "#{adapter_name} #{name}") do
|
97
62
|
formatted_sql = apply_format(sql, format)
|
98
|
-
|
63
|
+
request_params = @config || {}
|
64
|
+
res = @connection.post("/?#{request_params.merge(settings).to_param}", formatted_sql)
|
99
65
|
|
100
66
|
process_response(res)
|
101
67
|
end
|
102
68
|
end
|
103
69
|
|
70
|
+
private
|
71
|
+
|
72
|
+
def apply_format(sql, format)
|
73
|
+
format ? "#{sql} FORMAT #{format}" : sql
|
74
|
+
end
|
75
|
+
|
104
76
|
def process_response(res)
|
105
77
|
case res.code.to_i
|
106
78
|
when 200
|
@@ -109,6 +81,8 @@ module ActiveRecord
|
|
109
81
|
raise ActiveRecord::ActiveRecordError,
|
110
82
|
"Response code: #{res.code}:\n#{res.body}"
|
111
83
|
end
|
84
|
+
rescue JSON::ParserError
|
85
|
+
res.body
|
112
86
|
end
|
113
87
|
|
114
88
|
def log_with_debug(sql, name = nil)
|
@@ -121,7 +95,11 @@ module ActiveRecord
|
|
121
95
|
end
|
122
96
|
|
123
97
|
def create_table_definition(*args)
|
124
|
-
|
98
|
+
if ActiveRecord::version >= Gem::Version.new('6')
|
99
|
+
Clickhouse::TableDefinition.new(self, *args)
|
100
|
+
else
|
101
|
+
Clickhouse::TableDefinition.new(*args)
|
102
|
+
end
|
125
103
|
end
|
126
104
|
|
127
105
|
def new_column_from_field(table_name, field)
|
@@ -130,13 +108,17 @@ module ActiveRecord
|
|
130
108
|
default = field[3]
|
131
109
|
default_value = extract_value_from_default(default)
|
132
110
|
default_function = extract_default_function(default_value, default)
|
133
|
-
|
111
|
+
if ActiveRecord::version >= Gem::Version.new('6')
|
112
|
+
ClickhouseColumn.new(field[0], default_value, type_metadata, field[1].include?('Nullable'), default_function)
|
113
|
+
else
|
114
|
+
ClickhouseColumn.new(field[0], default_value, type_metadata, field[1].include?('Nullable'), table_name, default_function)
|
115
|
+
end
|
134
116
|
end
|
135
117
|
|
136
118
|
protected
|
137
119
|
|
138
120
|
def table_structure(table_name)
|
139
|
-
result = do_system_execute("DESCRIBE TABLE
|
121
|
+
result = do_system_execute("DESCRIBE TABLE `#{table_name}`", table_name)
|
140
122
|
data = result['data']
|
141
123
|
|
142
124
|
return data unless data.empty?
|
@@ -2,7 +2,6 @@
|
|
2
2
|
|
3
3
|
require 'clickhouse-activerecord/arel/visitors/to_sql'
|
4
4
|
require 'clickhouse-activerecord/arel/table'
|
5
|
-
require 'active_record/connection_adapters/abstract_adapter'
|
6
5
|
require 'active_record/connection_adapters/clickhouse/oid/date'
|
7
6
|
require 'active_record/connection_adapters/clickhouse/oid/date_time'
|
8
7
|
require 'active_record/connection_adapters/clickhouse/oid/big_integer'
|
@@ -19,6 +18,7 @@ module ActiveRecord
|
|
19
18
|
config = config.symbolize_keys
|
20
19
|
host = config[:host] || 'localhost'
|
21
20
|
port = config[:port] || 8123
|
21
|
+
ssl = config[:ssl].present? ? config[:ssl] : port == 443
|
22
22
|
|
23
23
|
if config.key?(:database)
|
24
24
|
database = config[:database]
|
@@ -26,11 +26,26 @@ module ActiveRecord
|
|
26
26
|
raise ArgumentError, 'No database specified. Missing argument: database.'
|
27
27
|
end
|
28
28
|
|
29
|
-
ConnectionAdapters::ClickhouseAdapter.new(
|
29
|
+
ConnectionAdapters::ClickhouseAdapter.new(logger, [host, port, ssl], { user: config[:username], password: config[:password], database: database }.compact, config)
|
30
30
|
end
|
31
31
|
end
|
32
32
|
end
|
33
33
|
|
34
|
+
class Relation
|
35
|
+
|
36
|
+
# Replace for only ClickhouseAdapter
|
37
|
+
def reverse_order!
|
38
|
+
orders = order_values.uniq
|
39
|
+
orders.reject!(&:blank?)
|
40
|
+
if self.connection.is_a?(ConnectionAdapters::ClickhouseAdapter) && orders.empty? && !primary_key
|
41
|
+
self.order_values = %w(date created_at).select {|c| column_names.include?(c) }.map{|c| arel_attribute(c).desc }
|
42
|
+
else
|
43
|
+
self.order_values = reverse_sql_order(orders)
|
44
|
+
end
|
45
|
+
self
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
34
49
|
module TypeCaster
|
35
50
|
class Map
|
36
51
|
def is_view
|
@@ -78,17 +93,34 @@ module ActiveRecord
|
|
78
93
|
include Clickhouse::SchemaStatements
|
79
94
|
|
80
95
|
# Initializes and connects a Clickhouse adapter.
|
81
|
-
def initialize(
|
82
|
-
super(
|
96
|
+
def initialize(logger, connection_parameters, config, full_config)
|
97
|
+
super(nil, logger)
|
83
98
|
@connection_parameters = connection_parameters
|
84
99
|
@config = config
|
85
|
-
@debug = debug
|
100
|
+
@debug = full_config[:debug] || false
|
101
|
+
@full_config = full_config
|
86
102
|
|
87
103
|
@prepared_statements = false
|
104
|
+
if ActiveRecord::version == Gem::Version.new('6.0.0')
|
105
|
+
@prepared_statement_status = Concurrent::ThreadLocalVar.new(false)
|
106
|
+
end
|
88
107
|
|
89
108
|
connect
|
90
109
|
end
|
91
110
|
|
111
|
+
# Support SchemaMigration from v5.2.2 to v6+
|
112
|
+
def schema_migration # :nodoc:
|
113
|
+
if ActiveRecord::version >= Gem::Version.new('6')
|
114
|
+
super
|
115
|
+
else
|
116
|
+
ActiveRecord::SchemaMigration
|
117
|
+
end
|
118
|
+
end
|
119
|
+
|
120
|
+
def migrations_paths
|
121
|
+
@full_config[:migrations_paths] || 'db/migrate_clickhouse'
|
122
|
+
end
|
123
|
+
|
92
124
|
def arel_visitor # :nodoc:
|
93
125
|
ClickhouseActiverecord::Arel::Visitors::ToSql.new(self)
|
94
126
|
end
|
@@ -145,7 +177,11 @@ module ActiveRecord
|
|
145
177
|
end
|
146
178
|
|
147
179
|
def column_name_for_operation(operation, node) # :nodoc:
|
148
|
-
|
180
|
+
if ActiveRecord::version >= Gem::Version.new('6')
|
181
|
+
visitor.compile(node)
|
182
|
+
else
|
183
|
+
column_name_from_arel_node(node)
|
184
|
+
end
|
149
185
|
end
|
150
186
|
|
151
187
|
# Executes insert +sql+ statement in the context of this connection using
|
@@ -166,16 +202,24 @@ module ActiveRecord
|
|
166
202
|
|
167
203
|
# Create a new ClickHouse database.
|
168
204
|
def create_database(name)
|
169
|
-
sql = "CREATE DATABASE #{quote_table_name(name)}"
|
205
|
+
sql = apply_cluster "CREATE DATABASE #{quote_table_name(name)}"
|
170
206
|
log_with_debug(sql, adapter_name) do
|
171
207
|
res = @connection.post("/?#{@config.except(:database).to_param}", "CREATE DATABASE #{quote_table_name(name)}")
|
172
208
|
process_response(res)
|
173
209
|
end
|
174
210
|
end
|
175
211
|
|
212
|
+
def create_table(table_name, comment: nil, **options)
|
213
|
+
super(
|
214
|
+
apply_cluster(table_name),
|
215
|
+
comment: comment,
|
216
|
+
**options
|
217
|
+
)
|
218
|
+
end
|
219
|
+
|
176
220
|
# Drops a ClickHouse database.
|
177
221
|
def drop_database(name) #:nodoc:
|
178
|
-
sql = "DROP DATABASE IF EXISTS #{quote_table_name(name)}"
|
222
|
+
sql = apply_cluster "DROP DATABASE IF EXISTS #{quote_table_name(name)}"
|
179
223
|
log_with_debug(sql, adapter_name) do
|
180
224
|
res = @connection.post("/?#{@config.except(:database).to_param}", sql)
|
181
225
|
process_response(res)
|
@@ -183,7 +227,7 @@ module ActiveRecord
|
|
183
227
|
end
|
184
228
|
|
185
229
|
def drop_table(table_name, options = {}) # :nodoc:
|
186
|
-
do_execute "DROP TABLE#{' IF EXISTS' if options[:if_exists]} #{quote_table_name(table_name)}"
|
230
|
+
do_execute apply_cluster "DROP TABLE#{' IF EXISTS' if options[:if_exists]} #{quote_table_name(table_name)}"
|
187
231
|
end
|
188
232
|
|
189
233
|
protected
|
@@ -195,8 +239,15 @@ module ActiveRecord
|
|
195
239
|
private
|
196
240
|
|
197
241
|
def connect
|
198
|
-
|
199
|
-
|
242
|
+
@connection = Net::HTTP.start(@connection_parameters[0], @connection_parameters[1], use_ssl: @connection_parameters[2], verify_mode: OpenSSL::SSL::VERIFY_NONE)
|
243
|
+
end
|
244
|
+
|
245
|
+
def cluster
|
246
|
+
@full_config[:cluster]
|
247
|
+
end
|
248
|
+
|
249
|
+
def apply_cluster(sql)
|
250
|
+
cluster ? "#{sql} ON CLUSTER #{cluster}" : sql
|
200
251
|
end
|
201
252
|
end
|
202
253
|
end
|
@@ -3,7 +3,7 @@ module ClickhouseActiverecord
|
|
3
3
|
|
4
4
|
def table(table, stream)
|
5
5
|
stream.puts " # TABLE: #{table}"
|
6
|
-
stream.puts " # SQL: #{@connection.do_system_execute("SHOW CREATE TABLE
|
6
|
+
stream.puts " # SQL: #{@connection.do_system_execute("SHOW CREATE TABLE `#{table.gsub(/^\.inner\./, '')}`")['data'].try(:first).try(:first)}"
|
7
7
|
super(table.gsub(/^\.inner\./, ''), stream)
|
8
8
|
end
|
9
9
|
end
|
@@ -31,13 +31,27 @@ module ClickhouseActiverecord
|
|
31
31
|
create
|
32
32
|
end
|
33
33
|
|
34
|
+
def structure_dump(*args)
|
35
|
+
tables = connection.execute("SHOW TABLES FROM #{@configuration['database']}")['data'].flatten
|
36
|
+
|
37
|
+
File.open(args.first, 'w:utf-8') do |file|
|
38
|
+
tables.each do |table|
|
39
|
+
next if table.match(/\.inner/)
|
40
|
+
file.puts connection.execute("SHOW CREATE TABLE #{table}")['data'].try(:first).try(:first).gsub("#{@configuration['database']}.", '') + ";\n\n"
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def structure_load(*args)
|
46
|
+
File.read(args.first).split(";\n\n").each { |sql| connection.execute(sql) }
|
47
|
+
end
|
48
|
+
|
34
49
|
def migrate
|
35
50
|
check_target_version
|
36
51
|
|
37
52
|
verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] != "false" : true
|
38
53
|
scope = ENV["SCOPE"]
|
39
54
|
verbose_was, ActiveRecord::Migration.verbose = ActiveRecord::Migration.verbose, verbose
|
40
|
-
binding.pry
|
41
55
|
connection.migration_context.migrate(target_version) do |migration|
|
42
56
|
scope.blank? || scope == migration.scope
|
43
57
|
end
|
@@ -6,6 +6,20 @@ class ClickhouseMigrationGenerator < ActiveRecord::Generators::MigrationGenerato
|
|
6
6
|
def create_migration_file
|
7
7
|
set_local_assigns!
|
8
8
|
validate_file_name!
|
9
|
-
migration_template @migration_template, "
|
9
|
+
migration_template @migration_template, File.join(db_migrate_path, "#{file_name}.rb")
|
10
|
+
end
|
11
|
+
|
12
|
+
private
|
13
|
+
|
14
|
+
def db_migrate_path
|
15
|
+
if defined?(Rails.application) && Rails.application
|
16
|
+
configured_migrate_path || default_migrate_path
|
17
|
+
else
|
18
|
+
default_migrate_path
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
def default_migrate_path
|
23
|
+
"db/migrate_clickhouse"
|
10
24
|
end
|
11
25
|
end
|
data/lib/tasks/clickhouse.rake
CHANGED
@@ -2,6 +2,70 @@
|
|
2
2
|
|
3
3
|
namespace :clickhouse do
|
4
4
|
|
5
|
+
task prepare_schema_migration_table: :environment do
|
6
|
+
cluster, database, replica = ActiveRecord::Base.connection_config.values_at(:cluster, :database, :replica)
|
7
|
+
return if cluster.nil?
|
8
|
+
|
9
|
+
connection = ActiveRecord::Base.connection
|
10
|
+
key_options = connection.internal_string_options_for_primary_key
|
11
|
+
block = Proc.new do |t|
|
12
|
+
t.string :version, key_options
|
13
|
+
end
|
14
|
+
distributed_table_name = ".#{ActiveRecord::SchemaMigration.table_name}_distributed"
|
15
|
+
unless connection.table_exists?(distributed_table_name)
|
16
|
+
options = { id: false }
|
17
|
+
if replica
|
18
|
+
shard = replica.is_a?(String) ? replica : '{shard}'
|
19
|
+
options[:options] = <<-SQL
|
20
|
+
ReplicatedMergeTree('/clickhouse/tables/{cluster}/#{shard}/#{database}.`#{distributed_table_name}`', '{replica}')
|
21
|
+
PARTITION BY version ORDER BY (version) SETTINGS index_granularity = 8192
|
22
|
+
SQL
|
23
|
+
end
|
24
|
+
connection.create_table("`#{distributed_table_name}`", options, &block)
|
25
|
+
end
|
26
|
+
unless connection.table_exists?(ActiveRecord::SchemaMigration.table_name)
|
27
|
+
connection.create_table(
|
28
|
+
ActiveRecord::SchemaMigration.table_name,
|
29
|
+
id: false,
|
30
|
+
options: "Distributed(#{cluster},#{database},`#{distributed_table_name}`,sipHash64(version))",
|
31
|
+
&block
|
32
|
+
)
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
task prepare_internal_metadata_table: :environment do
|
37
|
+
cluster, database, replica = ActiveRecord::Base.connection_config.values_at(:cluster, :database, :replica)
|
38
|
+
return if cluster.nil?
|
39
|
+
|
40
|
+
connection = ActiveRecord::Base.connection
|
41
|
+
key_options = connection.internal_string_options_for_primary_key
|
42
|
+
block = Proc.new do |t|
|
43
|
+
t.string :key, key_options
|
44
|
+
t.string :value
|
45
|
+
t.timestamps
|
46
|
+
end
|
47
|
+
distributed_table_name = ".#{ActiveRecord::InternalMetadata.table_name}_distributed"
|
48
|
+
unless connection.table_exists?(distributed_table_name)
|
49
|
+
options = { id: false }
|
50
|
+
if replica
|
51
|
+
shard = replica.is_a?(String) ? replica : '{shard}'
|
52
|
+
options[:options] = <<-SQL
|
53
|
+
ReplicatedMergeTree('/clickhouse/tables/{cluster}/#{shard}/#{database}.`#{distributed_table_name}`', '{replica}')
|
54
|
+
PARTITION BY toDate(created_at) ORDER BY (created_at) SETTINGS index_granularity = 8192
|
55
|
+
SQL
|
56
|
+
end
|
57
|
+
connection.create_table("`#{distributed_table_name}`", options, &block)
|
58
|
+
end
|
59
|
+
unless connection.table_exists?(ActiveRecord::InternalMetadata.table_name)
|
60
|
+
connection.create_table(
|
61
|
+
ActiveRecord::InternalMetadata.table_name,
|
62
|
+
id: false,
|
63
|
+
options: "Distributed(#{cluster},#{database},`#{distributed_table_name}`,sipHash64(created_at))",
|
64
|
+
&block
|
65
|
+
)
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
5
69
|
task load_config: :environment do
|
6
70
|
ENV['SCHEMA'] = "db/clickhouse_schema.rb"
|
7
71
|
ActiveRecord::Migrator.migrations_paths = ["db/migrate_clickhouse"]
|
@@ -27,6 +91,18 @@ namespace :clickhouse do
|
|
27
91
|
|
28
92
|
end
|
29
93
|
|
94
|
+
namespace :structure do
|
95
|
+
desc 'Load database structure'
|
96
|
+
task load: [:load_config, 'db:check_protected_environments'] do
|
97
|
+
ClickhouseActiverecord::Tasks.new(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"]).structure_load("#{Rails.root}/db/clickhouse_structure.sql")
|
98
|
+
end
|
99
|
+
|
100
|
+
desc 'Dump database structure'
|
101
|
+
task dump: [:load_config, 'db:check_protected_environments'] do
|
102
|
+
ClickhouseActiverecord::Tasks.new(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"]).structure_dump("#{Rails.root}/db/clickhouse_structure.sql")
|
103
|
+
end
|
104
|
+
end
|
105
|
+
|
30
106
|
desc 'Creates the database from DATABASE_URL or config/database.yml'
|
31
107
|
task create: [:load_config] do
|
32
108
|
ActiveRecord::Tasks::DatabaseTasks.create(ActiveRecord::Base.configurations["#{Rails.env}_clickhouse"])
|
@@ -49,7 +125,7 @@ namespace :clickhouse do
|
|
49
125
|
end
|
50
126
|
|
51
127
|
desc 'Migrate the clickhouse database'
|
52
|
-
task migrate: :load_config do
|
128
|
+
task migrate: [:load_config, :prepare_schema_migration_table, :prepare_internal_metadata_table] do
|
53
129
|
Rake::Task['db:migrate'].execute
|
54
130
|
end
|
55
131
|
end
|
metadata
CHANGED
@@ -1,22 +1,19 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: clickhouse-activerecord
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.13
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sergey Odintsov
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2020-09-11 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
-
- - "~>"
|
18
|
-
- !ruby/object:Gem::Version
|
19
|
-
version: '1.13'
|
20
17
|
- - ">="
|
21
18
|
- !ruby/object:Gem::Version
|
22
19
|
version: 1.13.4
|
@@ -24,9 +21,6 @@ dependencies:
|
|
24
21
|
prerelease: false
|
25
22
|
version_requirements: !ruby/object:Gem::Requirement
|
26
23
|
requirements:
|
27
|
-
- - "~>"
|
28
|
-
- !ruby/object:Gem::Version
|
29
|
-
version: '1.13'
|
30
24
|
- - ">="
|
31
25
|
- !ruby/object:Gem::Version
|
32
26
|
version: 1.13.4
|
@@ -34,14 +28,14 @@ dependencies:
|
|
34
28
|
name: activerecord
|
35
29
|
requirement: !ruby/object:Gem::Requirement
|
36
30
|
requirements:
|
37
|
-
- - "
|
31
|
+
- - ">="
|
38
32
|
- !ruby/object:Gem::Version
|
39
33
|
version: '5.2'
|
40
34
|
type: :runtime
|
41
35
|
prerelease: false
|
42
36
|
version_requirements: !ruby/object:Gem::Requirement
|
43
37
|
requirements:
|
44
|
-
- - "
|
38
|
+
- - ">="
|
45
39
|
- !ruby/object:Gem::Version
|
46
40
|
version: '5.2'
|
47
41
|
- !ruby/object:Gem::Dependency
|
@@ -153,8 +147,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
153
147
|
- !ruby/object:Gem::Version
|
154
148
|
version: '0'
|
155
149
|
requirements: []
|
156
|
-
|
157
|
-
rubygems_version: 2.5.2.3
|
150
|
+
rubygems_version: 3.0.1
|
158
151
|
signing_key:
|
159
152
|
specification_version: 4
|
160
153
|
summary: ClickHouse ActiveRecord
|