pg_party 1.0.1 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f1a1587885a9bcf43b96be821b8d36a23ebdef34d09776f94f85e6ccf2a85c46
4
- data.tar.gz: b64190282c4062cde23812646ee7138436633a2f7875d56f1a6c84d5df62c35d
3
+ metadata.gz: f0edf666f9d4e7d3a7af82990be55d58b970949c5a549c1d8f1e1aeb2c066b09
4
+ data.tar.gz: 8c1164c72ffec0f00a892f9f4cf7028969a7aa39d056f4aae5e05674651a6142
5
5
  SHA512:
6
- metadata.gz: 3b2e01424e5cb531dbd44b4f572182ba1351a72577402e7ef567943f249f84b05b9ed9cf6af7ecc7c7d9b63eaf7203560cbd22e63399d972aef68852095bcd46
7
- data.tar.gz: 29ebaa353d92c39d8bb6880a7211a0594bb8fec6a2a9d4486c8936f62e3fa387b14258a154859029e413e2cef514240a6b0ec74d816f1b8c71f373546281247e
6
+ metadata.gz: 60baccd0471cd1628714d9d10e918d5b3be2708d07850bedf8751323cd91737321f93441fec5a576346de409580c3a7f2bb99a384c7edb0b1c70b0e96a2be84b
7
+ data.tar.gz: 12126fee3f2eae1b3a70771c79a07cab254b061c3b11ed20fbc8095f7521db05672219e93437f7a3cbb062b4ed95dd56aa2f09ae8e032b7a26cfaab7f6878def
data/README.md CHANGED
@@ -48,6 +48,50 @@ $ gem install pg_party
48
48
  Note that the gemspec does not require `pg`, as some model methods _may_ work for other databases.
49
49
  Migration methods will be unavailable unless `pg` is installed.
50
50
 
51
+ ## Configuration
52
+
53
+ These values can be accessed and set via `PgParty.config` and `PgParty.configure`.
54
+
55
+ - `caching`
56
+ - Whether to cache currently attached partitions and anonymous model classes
57
+ - Default: `true`
58
+ - `caching_ttl`
59
+ - Length of time (in seconds) that cache entries are considered valid
60
+ - Default: `-1` (never expire cache entries)
61
+ - `schema_exclude_partitions`
62
+ - Whether to exclude child partitions in `rake db:structure:dump`
63
+ - Default: `true`
64
+ - `create_template_tables`
65
+ - Whether to create template tables by default. Use the `template:` option when creating partitioned tables to override this default.
66
+ - Default: `true`
67
+ - `create_with_primary_key`
68
+ - Whether to add primary key constraints to partitioned (parent) tables by default.
69
+ * This is not supported for Postgres 10 but is recommended for Postgres 11
70
+ * Primary key constraints must include all partition keys, for example: `primary_key: [:id, :created_at], partition_key: :created_at`
71
+ * Partition keys cannot use expressions
72
+ * Can be overridden via the `create_with_primary_key:` option when creating partitioned tables
73
+ - Default: `false`
74
+ - `include_subpartitions_in_partition_list`
75
+ - Whether to include nested subpartitions in the result of `YourModelClass.partiton_list` mby default.
76
+ You can always pass the `include_subpartitions:` option to override this.
77
+ - Default: `false` (for backward compatibility)
78
+
79
+ Note that caching is done in-memory for each process of an application. Attaching / detaching partitions _will_ clear the cache, but only for the process that initiated the request. For multi-process web servers, it is recommended to use a TTL or disable caching entirely.
80
+
81
+ ### Example
82
+
83
+ ```ruby
84
+ # in a Rails initializer
85
+ PgParty.configure do |c|
86
+ c.caching_ttl = 60
87
+ c.schema_exclude_partitions = false
88
+ c.include_subpartitions_in_partition_list = true
89
+ # Postgres 11+ users starting fresh may want to use below options
90
+ c.create_template_tables = false
91
+ c.create_with_primary_key = true
92
+ end
93
+ ```
94
+
51
95
  ## Usage
52
96
 
53
97
  ### Migrations
@@ -62,25 +106,63 @@ These methods are available in migrations as well as `ActiveRecord::Base#connect
62
106
  - `create_list_partition`
63
107
  - Create partitioned table using the _list_ partitioning method
64
108
  - Required args: `table_name`, `partition_key:`
109
+ - `create_hash_partition` (Postgres 11+)
110
+ - Create partitioned table using the _hash_ partitioning method
111
+ - Required args: `table_name`, `partition_key:`
65
112
  - `create_range_partition_of`
66
113
  - Create partition in _range_ partitioned table with partition key between _range_ of values
67
114
  - Required args: `table_name`, `start_range:`, `end_range:`
115
+ - Create a subpartition by specifying a `partition_type:` of `:range`, `:list`, or `:hash` and a `partition_key:`
68
116
  - `create_list_partition_of`
69
117
  - Create partition in _list_ partitioned table with partition key in _list_ of values
70
118
  - Required args: `table_name`, `values:`
119
+ - Create a subpartition by specifying a `partition_type:` of `:range`, `:list`, or `:hash` and a `partition_key:`
120
+ - `create_hash_partition_of` (Postgres 11+)
121
+ - Create partition in _hash_ partitioned table for partition keys with hashed values having a specific remainder
122
+ - Required args: `table_name`, `modulus:`, `remainder`
123
+ - Create a subpartition by specifying a `partition_type:` of `:range`, `:list`, or `:hash` and a `partition_key:`
124
+ - Note that all partitions in a _hash_ partitioned table should have the same modulus. See [Examples](#examples) for more info.
125
+ - `create_default_partition_of` (Postgres 11+)
126
+ - Create a default partition for values not falling in the range or list constraints of any other partitions
127
+ - Required args: `table_name`
71
128
  - `attach_range_partition`
72
129
  - Attach existing table to _range_ partitioned table with partition key between _range_ of values
73
130
  - Required args: `parent_table_name`, `child_table_name`, `start_range:`, `end_range:`
74
131
  - `attach_list_partition`
75
132
  - Attach existing table to _list_ partitioned table with partition key in _list_ of values
76
133
  - Required args: `parent_table_name`, `child_table_name`, `values:`
134
+ - `attach_hash_partition` (Postgres 11+)
135
+ - Attach existing table to _hash_ partitioned table with partition key hashed values having a specific remainder
136
+ - Required args: `parent_table_name`, `child_table_name`, `modulus:`, `remainder`
137
+ - `attach_default_partition` (Postgres 11+)
138
+ - Attach existing table as the _default_ partition
139
+ - Required args: `parent_table_name`, `child_table_name`
77
140
  - `detach_partition`
78
141
  - Detach partition from both _range and list_ partitioned tables
79
142
  - Required args: `parent_table_name`, `child_table_name`
80
143
  - `create_table_like`
81
144
  - Clone _any_ existing table
82
145
  - Required args: `table_name`, `new_table_name`
83
-
146
+ - `partitions_for_table_name`
147
+ - List all attached partitions for a given table
148
+ - Required args: `table_name`, `include_subpartitions:` (true or false)
149
+ - `parent_for_table_name`
150
+ - Fetch the parent table for a partition
151
+ - Required args: `table_name`
152
+ - Pass optional `traverse: true` to return the top-level table in the hierarchy (for subpartitions)
153
+ - Returns `nil` if the table is not a partition / has no parent
154
+ - `table_partitioned?`
155
+ - Returns true if the table is partitioned (false for non-partitioned tables and partitions themselves)
156
+ - Required args: `table_name`
157
+ - `add_index_on_all_partitions`
158
+ - Recursively add an index to all partitions and subpartitions of `table_name` using Postgres's ADD INDEX CONCURRENTLY
159
+ algorithm which adds the index in a non-blocking manner.
160
+ - Required args: `table_name`, `column_name` (all `add_index` arguments are supported)
161
+ - Use the `in_threads:` option to add indexes in parallel threads when there are many partitions. A value of 2 to 4
162
+ may be reasonable for tables with many large partitions and hosts with 4+ CPUs/cores.
163
+ - Use `disable_ddl_transaction!` in your migration to disable transactions when using this command with `in_threads:`
164
+ or `algorithm: :concurrently`.
165
+
84
166
  #### Examples
85
167
 
86
168
  Create _range_ partitioned table on `created_at::date` with two partitions:
@@ -131,20 +213,130 @@ class CreateSomeListRecord < ActiveRecord::Migration[5.1]
131
213
  create_list_partition_of \
132
214
  :some_list_records,
133
215
  values: 101..200
216
+
217
+ # default partition support is available in Postgres 11 or higher
218
+ create_default_partition_of \
219
+ :some_list_records
220
+ end
221
+ end
222
+ ```
223
+
224
+ Create _hash_ partitioned table on `id` with two partitions (Postgres 11+ required):
225
+ * A hash partition can be used to spread keys evenly(ish) across partitions
226
+ * `modulus:` should always equal the total number of partitions planned for the table
227
+ * `remainder:` is an integer which should be in the range of 0 to modulus-1
228
+
229
+ ```ruby
230
+ class CreateSomeHashRecord < ActiveRecord::Migration[5.1]
231
+ def up
232
+ # symbol is used for partition keys referring to individual columns
233
+ # create_with_primary_key: true, template: false on Postgres 11 will rely on PostgreSQL's native partition schema
234
+ # management vs this gem's template tables
235
+ create_hash_partition :some_hash_records, partition_key: :id, create_with_primary_key: true, template: false do |t|
236
+ t.text :some_value
237
+ t.timestamps
238
+ end
239
+
240
+ # without name argument, child partition created as "some_list_records_<hash>"
241
+ create_hash_partition_of \
242
+ :some_hash_records,
243
+ modulus: 2,
244
+ remainder: 0
245
+
246
+ # without name argument, child partition created as "some_list_records_<hash>"
247
+ create_hash_partition_of \
248
+ :some_hash_records,
249
+ modulus: 2,
250
+ remainder: 1
251
+ end
252
+ end
253
+ ```
254
+
255
+ Advanced example with subpartitioning: Create _list_ partitioned table on `id` subpartitioned by _range_ on `created_at`
256
+ with default partitions
257
+ * We can use Postgres 11's support for primary keys vs template tables, using the composite primary key `[:id, :created_at]`
258
+ to ensure all partition keys are present in the primary key
259
+ * Default partitions are only supported in Postgres 11+
260
+
261
+ ```ruby
262
+ class CreateSomeListSubpartitionedRecord < ActiveRecord::Migration[5.1]
263
+ def up
264
+ # when specifying a composite primary key, the primary keys must be specified as columns
265
+ create_list_partition \
266
+ :some_list_subpartitioned_records,
267
+ partition_key: [:id],
268
+ primary_key: [:id, :created_at],
269
+ template: false,
270
+ create_with_primary_key: true do |t|
271
+
272
+ t.integer :id
273
+ t.text :some_value
274
+ t.timestamps
275
+ end
276
+
277
+ create_default_partition_of \
278
+ :some_list_subpartitioned_records,
279
+ name: :some_list_subpartitioned_records_default,
280
+ partition_type: :range,
281
+ partition_key: :created_at
282
+
283
+ create_range_partition_of \
284
+ :some_list_subpartitioned_records_default,
285
+ name: :some_list_subpartitioned_records_default_2019,
286
+ start_range: '2019-01-01',
287
+ end_range: '2019-12-31T23:59:59'
288
+
289
+ create_default_partition_of \
290
+ :some_list_subpartitioned_records_default
291
+
292
+ create_list_partition_of \
293
+ :some_list_subpartitioned_records,
294
+ name: :some_list_subpartitioned_records_1,
295
+ values: 1..100,
296
+ partition_type: :range,
297
+ partition_key: :created_at
298
+
299
+ create_range_partition_of \
300
+ :some_list_subpartitioned_records_1,
301
+ name: :some_list_subpartitioned_records_1_2019,
302
+ start_range: '2019-01-01',
303
+ end_range: '2019-12-31T23:59:59'
304
+
305
+ create_default_partition_of
306
+ :some_list_subpartitioned_records_1
307
+
308
+ create_list_partition_of \
309
+ :some_list_subpartitioned_records,
310
+ name: :some_list_subpartitioned_records_2,
311
+ values: 101..200,
312
+ partition_type: :range,
313
+ partition_key: :created_at
314
+
315
+ create_range_partition_of \
316
+ :some_list_subpartitioned_records_2,
317
+ name: :some_list_subpartitioned_records_2_2019,
318
+ start_range: '2019-01-01',
319
+ end_range: '2019-12-31T23:59:59'
320
+
321
+ create_default_partition_of \
322
+ :some_list_subpartitioned_records_2
134
323
  end
135
324
  end
136
325
  ```
137
326
 
327
+ #### Template Tables
138
328
  Unfortunately, PostgreSQL 10 doesn't support indexes on partitioned tables.
139
329
  However, individual _partitions_ can have indexes.
140
330
  To avoid explicit index creation for _every_ new partition, we've introduced the idea of template tables.
141
331
  For every call to `create_list_partition` and `create_range_partition`, a clone `<table_name>_template` is created.
142
332
  Indexes, constraints, etc. created on the template table will propagate to new partitions in calls to `create_list_partition_of` and `create_range_partition_of`:
333
+ * Subpartitions will correctly clone from template tables if a template table exists for the top-level ancestor
334
+ * When using Postgres 11 or higher, you may wish to disable template tables and use the native features instead, see [Configuration](#configuration)
143
335
 
144
336
  ```ruby
145
337
  class CreateSomeListRecord < ActiveRecord::Migration[5.1]
146
338
  def up
147
- # template table creation is enabled by default - use "template: false" to opt-out
339
+ # template table creation is enabled by default - use "template: false" or the config option to opt-out
148
340
  create_list_partition :some_list_records, partition_key: :id do |t|
149
341
  t.integer :some_foreign_id
150
342
  t.text :some_value
@@ -167,6 +359,8 @@ class CreateSomeListRecord < ActiveRecord::Migration[5.1]
167
359
  end
168
360
  ```
169
361
 
362
+ #### Attaching Existing Tables as Partitions
363
+
170
364
  Attach an existing table to a _range_ partitioned table:
171
365
 
172
366
  ```ruby
@@ -194,6 +388,20 @@ class AttachListPartition < ActiveRecord::Migration[5.1]
194
388
  end
195
389
  ```
196
390
 
391
+ Attach an existing table to a _hash_ partitioned table:
392
+
393
+ ```ruby
394
+ class AttachHashPartition < ActiveRecord::Migration[5.1]
395
+ def up
396
+ attach_hash_partition \
397
+ :some_hash_records,
398
+ :some_existing_table,
399
+ modulus: 2,
400
+ remainder: 1
401
+ end
402
+ end
403
+ ```
404
+
197
405
  Detach a partition from any partitioned table:
198
406
 
199
407
  ```ruby
@@ -204,6 +412,31 @@ class DetachPartition < ActiveRecord::Migration[5.1]
204
412
  end
205
413
  ```
206
414
 
415
+ #### Safely cascading `add_index` commands
416
+ Postgres 11+ will automatically cascade CREATE INDEX operations to partitions and subpartitions, however
417
+ CREATE INDEX CONCURRENTLY is not supported, meaning table locks will be taken on each table while the new index is built.
418
+ Postgres 10 provides no way to cascade index creation natively.
419
+ * The `add_index_on_all_partitions` method solves for these limitations by recursively creating the specified
420
+ index on all partitions and subpartitions. Index names on individual partitions will include a hash suffix to avoid conflicts.
421
+ * On Postgres 11+, the created indices are correctly attached to an index on the parent table
422
+ * On Postgres 10, if you are using [Template Tables](#template-tables-for-postgres-10), you will want to add the index to the template table separately.
423
+ * This command can also be used on subpartitions to cascade targeted indices starting at one level of the table hierarchy
424
+
425
+ ```ruby
426
+ class AddSomeValueIndexToSomeListRecord < ActiveRecord::Migration[5.1]
427
+ # add_index_on_all_partitions with in_threads option may not be used within a transaction
428
+ # (also, algorithm: :concurrently cannot be used within a transaction)
429
+ disable_ddl_transaction!
430
+
431
+ def up
432
+ add_index :some_records_template, :some_value # Only if using Postgres 10 with template tables
433
+
434
+ # Pass the `in_threads:` option to create indices in parallel across multiple Postgres connections
435
+ add_index_on_all_partitions :some_records, :some_value, algorithm: :concurrently, in_threads: 4
436
+ end
437
+ end
438
+ ```
439
+
207
440
  For more examples, take a look at the Combustion schema definition and integration specs:
208
441
 
209
442
  - https://github.com/rkrage/pg_party/blob/master/spec/dummy/db/schema.rb
@@ -224,12 +457,15 @@ Class methods available to _all_ ActiveRecord models:
224
457
  - `list_partition_by`
225
458
  - Configure a model backed by a _list_ partitioned table
226
459
  - Required arg: `key` (partition key column) or block returning partition key expression
460
+ - `hash_partition_by`
461
+ - Configure a model backed by a _hash_ partitioned table
462
+ - Required arg: `key` (partition key column) or block returning partition key expression
227
463
 
228
464
  Class methods available to both _range and list_ partitioned models:
229
465
 
230
466
  - `partitions`
231
467
  - Retrieve a list of currently attached partitions
232
- - No arguments
468
+ - Optional `include_subpartitions:` argument to include all subpartitions in the returned list
233
469
  - `in_partition`
234
470
  - Retrieve an ActiveRecord model scoped to an individual partition
235
471
  - Required arg: `child_table_name`
@@ -254,6 +490,16 @@ Class methods available to _list_ partitioned models:
254
490
  - `partition_key_in`
255
491
  - Query for records where partition key in _list_ of values
256
492
  - Required arg: list of `values`
493
+
494
+
495
+ Class methods available to _hash_ partitioned models:
496
+
497
+ - `create_partition`
498
+ - Dynamically create new partition with hashed partition key divided by _modulus_ equals _remainder_
499
+ - Required arg: `modulus:`, `remainder:`
500
+ - `partition_key_in`
501
+ - Query for records where partition key in _list_ of values (method operates the same as for _list_ partitions above)
502
+ - Required arg: list of `values`
257
503
 
258
504
  #### Examples
259
505
 
@@ -275,6 +521,15 @@ class SomeListRecord < ApplicationRecord
275
521
  end
276
522
  ```
277
523
 
524
+ Configure model backed by a _hash_ partitioned table to get access to the methods described above:
525
+
526
+ ```ruby
527
+ class SomeHashRecord < ApplicationRecord
528
+ # symbol is used for partition keys referring to individual columns
529
+ hash_partition_by :id
530
+ end
531
+ ```
532
+
278
533
  Dynamically create new partition from _range_ partitioned model:
279
534
 
280
535
  ```ruby
@@ -289,19 +544,26 @@ Dynamically create new partition from _list_ partitioned model:
289
544
  SomeListRecord.create_partition(values: 200..300)
290
545
  ```
291
546
 
547
+ Dynamically create new partition from _hash_ partitioned model:
548
+
549
+ ```ruby
550
+ # additional options include: "name:" and "primary_key:"
551
+ SomeHashRecord.create_partition(modulus: 2, remainder: 1)
552
+ ```
553
+
292
554
  For _range_ partitioned model, query for records where partition key in _range_ of values:
293
555
 
294
556
  ```ruby
295
557
  SomeRangeRecord.partition_key_in("2019-06-08", "2019-06-10")
296
558
  ```
297
559
 
298
- For _list_ partitioned model, query for records where partition key in _list_ of values:
560
+ For _list_ and _hash_ partitioned models, query for records where partition key in _list_ of values:
299
561
 
300
562
  ```ruby
301
563
  SomeListRecord.partition_key_in(1, 2, 3, 4)
302
564
  ```
303
565
 
304
- For both _range and list_ partitioned models, query for records matching partition key:
566
+ For all partitioned models, query for records matching partition key:
305
567
 
306
568
  ```ruby
307
569
  SomeRangeRecord.partition_key_eq(Date.current)
@@ -309,15 +571,15 @@ SomeRangeRecord.partition_key_eq(Date.current)
309
571
  SomeListRecord.partition_key_eq(100)
310
572
  ```
311
573
 
312
- For both _range and list_ partitioned models, retrieve currently attached partitions:
574
+ For all partitioned models, retrieve currently attached partitions:
313
575
 
314
576
  ```ruby
315
577
  SomeRangeRecord.partitions
316
578
 
317
- SomeListRecord.partitions
579
+ SomeListRecord.partitions(include_subpartitions: true) # Include nested subpartitions
318
580
  ```
319
581
 
320
- For both _range and list_ partitioned models, retrieve ActiveRecord model scoped to individual partition:
582
+ For both all partitioned models, retrieve ActiveRecord model scoped to individual partition:
321
583
 
322
584
  ```ruby
323
585
  SomeRangeRecord.in_partition(:some_range_records_partition_name)
@@ -325,9 +587,35 @@ SomeRangeRecord.in_partition(:some_range_records_partition_name)
325
587
  SomeListRecord.in_partition(:some_list_records_partition_name)
326
588
  ```
327
589
 
590
+ To create _range_ partitions by month for previous, current and next months it's possible to use this example. To automate creation of partitions, run `Log.maintenance` every day with cron:
591
+
592
+ ```ruby
593
+ class Log < ApplicationRecord
594
+ range_partition_by { '(created_at::date)' }
595
+
596
+ def self.maintenance
597
+ partitions = [Date.today.prev_month, Date.today, Date.today.next_month]
598
+
599
+ partitions.each do |day|
600
+ name = Log.partition_name_for(day)
601
+ next if ActiveRecord::Base.connection.table_exists?(name)
602
+ Log.create_partition(
603
+ name: name,
604
+ start_range: day.beginning_of_month,
605
+ end_range: day.next_month.beginning_of_month
606
+ )
607
+ end
608
+ end
609
+
610
+ def self.partition_name_for(day)
611
+ "logs_y#{day.year}_m#{day.month}"
612
+ end
613
+ end
614
+ ```
615
+
328
616
  For more examples, take a look at the model integration specs:
329
617
 
330
- - https://github.com/rkrage/pg_party/tree/documentation/spec/integration/model
618
+ - https://github.com/rkrage/pg_party/tree/master/spec/integration/model
331
619
 
332
620
  ## Development
333
621
 
@@ -1,8 +1,28 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "pg_party/version"
4
+ require "pg_party/config"
5
+ require "pg_party/cache"
4
6
  require "active_support"
5
7
 
8
+ module PgParty
9
+ @config = Config.new
10
+ @cache = Cache.new
11
+
12
+ class << self
13
+ attr_reader :config, :cache
14
+
15
+ def configure(&blk)
16
+ blk.call(config)
17
+ end
18
+
19
+ def reset
20
+ @config = Config.new
21
+ @cache = Cache.new
22
+ end
23
+ end
24
+ end
25
+
6
26
  ActiveSupport.on_load(:active_record) do
7
27
  require "pg_party/model/methods"
8
28
 
@@ -14,10 +34,11 @@ ActiveSupport.on_load(:active_record) do
14
34
  PgParty::Adapter::AbstractMethods
15
35
  )
16
36
 
17
- require "pg_party/hacks/schema_cache"
37
+ require "active_record/tasks/postgresql_database_tasks"
38
+ require "pg_party/hacks/postgresql_database_tasks"
18
39
 
19
- ActiveRecord::ConnectionAdapters::SchemaCache.include(
20
- PgParty::Hacks::SchemaCache
40
+ ActiveRecord::Tasks::PostgreSQLDatabaseTasks.prepend(
41
+ PgParty::Hacks::PostgreSQLDatabaseTasks
21
42
  )
22
43
 
23
44
  begin