pg_party 1.0.1 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f1a1587885a9bcf43b96be821b8d36a23ebdef34d09776f94f85e6ccf2a85c46
-  data.tar.gz: b64190282c4062cde23812646ee7138436633a2f7875d56f1a6c84d5df62c35d
+  metadata.gz: f0edf666f9d4e7d3a7af82990be55d58b970949c5a549c1d8f1e1aeb2c066b09
+  data.tar.gz: 8c1164c72ffec0f00a892f9f4cf7028969a7aa39d056f4aae5e05674651a6142
 SHA512:
-  metadata.gz: 3b2e01424e5cb531dbd44b4f572182ba1351a72577402e7ef567943f249f84b05b9ed9cf6af7ecc7c7d9b63eaf7203560cbd22e63399d972aef68852095bcd46
-  data.tar.gz: 29ebaa353d92c39d8bb6880a7211a0594bb8fec6a2a9d4486c8936f62e3fa387b14258a154859029e413e2cef514240a6b0ec74d816f1b8c71f373546281247e
+  metadata.gz: 60baccd0471cd1628714d9d10e918d5b3be2708d07850bedf8751323cd91737321f93441fec5a576346de409580c3a7f2bb99a384c7edb0b1c70b0e96a2be84b
+  data.tar.gz: 12126fee3f2eae1b3a70771c79a07cab254b061c3b11ed20fbc8095f7521db05672219e93437f7a3cbb062b4ed95dd56aa2f09ae8e032b7a26cfaab7f6878def

data/README.md

@@ -48,6 +48,50 @@ $ gem install pg_party
 Note that the gemspec does not require `pg`, as some model methods _may_ work for other databases.
 Migration methods will be unavailable unless `pg` is installed.
 
+## Configuration
+
+These values can be accessed and set via `PgParty.config` and `PgParty.configure`.
+
+- `caching`
+  - Whether to cache currently attached partitions and anonymous model classes
+  - Default: `true`
+- `caching_ttl`
+  - Length of time (in seconds) that cache entries are considered valid
+  - Default: `-1` (never expire cache entries)
+- `schema_exclude_partitions`
+  - Whether to exclude child partitions in `rake db:structure:dump`
+  - Default: `true`
+- `create_template_tables`
+  - Whether to create template tables by default. Use the `template:` option when creating partitioned tables to override this default.
+  - Default: `true`
+- `create_with_primary_key`
+  - Whether to add primary key constraints to partitioned (parent) tables by default.
+    * This is not supported for Postgres 10 but is recommended for Postgres 11
+    * Primary key constraints must include all partition keys, for example: `primary_key: [:id, :created_at], partition_key: :created_at`
+    * Partition keys cannot use expressions
+    * Can be overridden via the `create_with_primary_key:` option when creating partitioned tables
+  - Default: `false`
+- `include_subpartitions_in_partition_list`
+  - Whether to include nested subpartitions in the result of `YourModelClass.partiton_list` mby default.
+    You can always pass the `include_subpartitions:` option to override this.
+  - Default: `false` (for backward compatibility)
+
+Note that caching is done in-memory for each process of an application. Attaching / detaching partitions _will_ clear the cache, but only for the process that initiated the request. For multi-process web servers, it is recommended to use a TTL or disable caching entirely.
+
+### Example
+
+```ruby
+# in a Rails initializer
+PgParty.configure do |c|
+  c.caching_ttl = 60
+  c.schema_exclude_partitions = false
+  c.include_subpartitions_in_partition_list = true
+  # Postgres 11+ users starting fresh may want to use below options
+  c.create_template_tables = false
+  c.create_with_primary_key = true
+end
+```
+
 ## Usage
 
 ### Migrations
@@ -62,25 +106,63 @@ These methods are available in migrations as well as `ActiveRecord::Base#connect
 - `create_list_partition`
   - Create partitioned table using the _list_ partitioning method
   - Required args: `table_name`, `partition_key:`
+- `create_hash_partition` (Postgres 11+)
+  - Create partitioned table using the _hash_ partitioning method
+  - Required args: `table_name`, `partition_key:`
 - `create_range_partition_of`
   - Create partition in _range_ partitioned table with partition key between _range_ of values
   - Required args: `table_name`, `start_range:`, `end_range:`
+  - Create a subpartition by specifying a `partition_type:` of `:range`, `:list`, or `:hash` and a `partition_key:`
 - `create_list_partition_of`
   - Create partition in _list_ partitioned table with partition key in _list_ of values
   - Required args: `table_name`, `values:`
+  - Create a subpartition by specifying a `partition_type:` of `:range`, `:list`, or `:hash` and a `partition_key:`
+- `create_hash_partition_of` (Postgres 11+)
+  - Create partition in _hash_ partitioned table for partition keys with hashed values having a specific remainder
+  - Required args: `table_name`, `modulus:`, `remainder`
+  - Create a subpartition by specifying a `partition_type:` of `:range`, `:list`, or `:hash` and a `partition_key:`
+  - Note that all partitions in a _hash_ partitioned table should have the same modulus. See [Examples](#examples) for more info.
+- `create_default_partition_of` (Postgres 11+)
+  - Create a default partition for values not falling in the range or list constraints of any other partitions
+  - Required args: `table_name`
 - `attach_range_partition`
   - Attach existing table to _range_ partitioned table with partition key between _range_ of values
   - Required args: `parent_table_name`, `child_table_name`, `start_range:`, `end_range:`
 - `attach_list_partition`
   - Attach existing table to _list_ partitioned table with partition key in _list_ of values
   - Required args: `parent_table_name`, `child_table_name`, `values:`
+- `attach_hash_partition` (Postgres 11+)
+  - Attach existing table to _hash_ partitioned table with partition key hashed values having a specific remainder
+  - Required args: `parent_table_name`, `child_table_name`, `modulus:`, `remainder`
+- `attach_default_partition` (Postgres 11+)
+  - Attach existing table as the _default_ partition
+  - Required args: `parent_table_name`, `child_table_name`
 - `detach_partition`
   - Detach partition from both _range and list_ partitioned tables
   - Required args: `parent_table_name`, `child_table_name`
 - `create_table_like`
   - Clone _any_ existing table
   - Required args: `table_name`, `new_table_name`
-
+- `partitions_for_table_name`
+  - List all attached partitions for a given table
+  - Required args: `table_name`, `include_subpartitions:` (true or false)
+- `parent_for_table_name`
+  - Fetch the parent table for a partition
+  - Required args: `table_name`
+  - Pass optional `traverse: true` to return the top-level table in the hierarchy (for subpartitions)
+  - Returns `nil` if the table is not a partition / has no parent
+- `table_partitioned?`
+  - Returns true if the table is partitioned (false for non-partitioned tables and partitions themselves)
+  - Required args: `table_name`
+- `add_index_on_all_partitions`
+  - Recursively add an index to all partitions and subpartitions of `table_name` using Postgres's ADD INDEX CONCURRENTLY
+    algorithm which adds the index in a non-blocking manner.
+  - Required args: `table_name`, `column_name` (all `add_index` arguments are supported)
+  - Use the `in_threads:` option to add indexes in parallel threads when there are many partitions. A value of 2 to 4
+    may be reasonable for tables with many large partitions and hosts with 4+ CPUs/cores.
+  - Use `disable_ddl_transaction!` in your migration to disable transactions when using this command with `in_threads:`
+    or `algorithm: :concurrently`.
+    
 #### Examples
 
 Create _range_ partitioned table on `created_at::date` with two partitions:
@@ -131,20 +213,130 @@ class CreateSomeListRecord < ActiveRecord::Migration[5.1]
      create_list_partition_of \
        :some_list_records,
        values: 101..200
+    
+    # default partition support is available in Postgres 11 or higher
+     create_default_partition_of \
+       :some_list_records
+  end
+end
+```
+
+Create _hash_ partitioned table on `id` with two partitions (Postgres 11+ required):
+  * A hash partition can be used to spread keys evenly(ish) across partitions
+  * `modulus:` should always equal the total number of partitions planned for the table
+  * `remainder:` is an integer which should be in the range of 0 to modulus-1
+
+```ruby
+class CreateSomeHashRecord < ActiveRecord::Migration[5.1]
+  def up
+    # symbol is used for partition keys referring to individual columns
+    # create_with_primary_key: true, template: false on Postgres 11 will rely on PostgreSQL's native partition schema
+    # management vs this gem's template tables
+    create_hash_partition :some_hash_records, partition_key: :id, create_with_primary_key: true, template: false do |t|
+      t.text :some_value
+      t.timestamps
+    end
+
+    # without name argument, child partition created as "some_list_records_<hash>"
+    create_hash_partition_of \
+      :some_hash_records,
+      modulus: 2,
+      remainder: 0
+
+    # without name argument, child partition created as "some_list_records_<hash>"
+    create_hash_partition_of \
+      :some_hash_records,
+      modulus: 2,
+      remainder: 1
+  end
+end
+```
+
+Advanced example with subpartitioning: Create _list_ partitioned table on `id` subpartitioned by _range_ on `created_at`
+with default partitions
+* We can use Postgres 11's support for primary keys vs template tables, using the composite primary key `[:id, :created_at]`
+  to ensure all partition keys are present in the primary key
+* Default partitions are only supported in Postgres 11+
+
+```ruby
+class CreateSomeListSubpartitionedRecord < ActiveRecord::Migration[5.1]
+  def up
+    # when specifying a composite primary key, the primary keys must be specified as columns
+    create_list_partition \
+      :some_list_subpartitioned_records,
+      partition_key: [:id],
+      primary_key: [:id, :created_at],
+      template: false,
+      create_with_primary_key: true do |t|
+
+      t.integer :id
+      t.text :some_value
+      t.timestamps
+    end
+
+    create_default_partition_of \
+      :some_list_subpartitioned_records,
+      name: :some_list_subpartitioned_records_default,
+      partition_type: :range,
+      partition_key: :created_at
+
+    create_range_partition_of \
+      :some_list_subpartitioned_records_default,
+      name: :some_list_subpartitioned_records_default_2019,
+      start_range: '2019-01-01',
+      end_range: '2019-12-31T23:59:59'
+    
+    create_default_partition_of \
+      :some_list_subpartitioned_records_default
+
+    create_list_partition_of \
+      :some_list_subpartitioned_records,
+      name: :some_list_subpartitioned_records_1,
+      values: 1..100,
+      partition_type: :range,
+      partition_key: :created_at
+  
+    create_range_partition_of \
+      :some_list_subpartitioned_records_1,
+      name: :some_list_subpartitioned_records_1_2019,
+      start_range: '2019-01-01',
+      end_range: '2019-12-31T23:59:59'
+
+    create_default_partition_of
+      :some_list_subpartitioned_records_1
+
+     create_list_partition_of \
+       :some_list_subpartitioned_records,
+       name: :some_list_subpartitioned_records_2,
+       values: 101..200,
+       partition_type: :range,
+       partition_key: :created_at
+
+    create_range_partition_of \
+      :some_list_subpartitioned_records_2,
+      name: :some_list_subpartitioned_records_2_2019,
+      start_range: '2019-01-01',
+      end_range: '2019-12-31T23:59:59'
+
+    create_default_partition_of \
+      :some_list_subpartitioned_records_2
   end
 end
 ```
 
+#### Template Tables
 Unfortunately, PostgreSQL 10 doesn't support indexes on partitioned tables.
 However, individual _partitions_ can have indexes.
 To avoid explicit index creation for _every_ new partition, we've introduced the idea of template tables.
 For every call to `create_list_partition` and `create_range_partition`, a clone `<table_name>_template` is created.
 Indexes, constraints, etc. created on the template table will propagate to new partitions in calls to `create_list_partition_of` and `create_range_partition_of`:
+* Subpartitions will correctly clone from template tables if a template table exists for the top-level ancestor
+* When using Postgres 11 or higher, you may wish to disable template tables and use the native features instead, see [Configuration](#configuration)
 
 ```ruby
 class CreateSomeListRecord < ActiveRecord::Migration[5.1]
   def up
-    # template table creation is enabled by default - use "template: false" to opt-out
+    # template table creation is enabled by default - use "template: false" or the config option to opt-out
     create_list_partition :some_list_records, partition_key: :id do |t|
       t.integer :some_foreign_id
       t.text :some_value
@@ -167,6 +359,8 @@ class CreateSomeListRecord < ActiveRecord::Migration[5.1]
 end
 ```
 
+#### Attaching Existing Tables as Partitions
+
 Attach an existing table to a _range_ partitioned table:
 
 ```ruby
@@ -194,6 +388,20 @@ class AttachListPartition < ActiveRecord::Migration[5.1]
 end
 ```
 
+Attach an existing table to a _hash_ partitioned table:
+
+```ruby
+class AttachHashPartition < ActiveRecord::Migration[5.1]
+  def up
+    attach_hash_partition \
+      :some_hash_records,
+      :some_existing_table,
+      modulus: 2,
+      remainder: 1
+  end
+end
+```
+
 Detach a partition from any partitioned table:
 
 ```ruby
@@ -204,6 +412,31 @@ class DetachPartition < ActiveRecord::Migration[5.1]
 end
 ```
 
+#### Safely cascading `add_index` commands
+Postgres 11+ will automatically cascade CREATE INDEX operations to partitions and subpartitions, however
+CREATE INDEX CONCURRENTLY is not supported, meaning table locks will be taken on each table while the new index is built.
+Postgres 10 provides no way to cascade index creation natively.
+* The `add_index_on_all_partitions` method solves for these limitations by recursively creating the specified
+  index on all partitions and subpartitions. Index names on individual partitions will include a hash suffix to avoid conflicts.
+* On Postgres 11+, the created indices are correctly attached to an index on the parent table
+* On Postgres 10, if you are using [Template Tables](#template-tables-for-postgres-10), you will want to add the index to the template table separately.
+* This command can also be used on subpartitions to cascade targeted indices starting at one level of the table hierarchy
+
+```ruby
+class AddSomeValueIndexToSomeListRecord < ActiveRecord::Migration[5.1]
+  # add_index_on_all_partitions with in_threads option may not be used within a transaction
+  # (also, algorithm: :concurrently cannot be used within a transaction)
+  disable_ddl_transaction!
+
+  def up
+    add_index :some_records_template, :some_value # Only if using Postgres 10 with template tables
+
+    # Pass the `in_threads:` option to create indices in parallel across multiple Postgres connections
+    add_index_on_all_partitions :some_records, :some_value, algorithm: :concurrently, in_threads: 4
+  end
+end
+```
+
 For more examples, take a look at the Combustion schema definition and integration specs:
 
 - https://github.com/rkrage/pg_party/blob/master/spec/dummy/db/schema.rb
@@ -224,12 +457,15 @@ Class methods available to _all_ ActiveRecord models:
 - `list_partition_by`
   - Configure a model backed by a _list_ partitioned table
   - Required arg: `key` (partition key column) or block returning partition key expression
+- `hash_partition_by`
+  - Configure a model backed by a _hash_ partitioned table
+  - Required arg: `key` (partition key column) or block returning partition key expression
 
 Class methods available to both _range and list_ partitioned models:
 
 - `partitions`
   - Retrieve a list of currently attached partitions
-  - No arguments
+  - Optional `include_subpartitions:` argument to include all subpartitions in the returned list
 - `in_partition`
   - Retrieve an ActiveRecord model scoped to an individual partition
   - Required arg: `child_table_name`
@@ -254,6 +490,16 @@ Class methods available to _list_ partitioned models:
 - `partition_key_in`
   - Query for records where partition key in _list_ of values
   - Required arg: list of `values`
+  
+
+Class methods available to _hash_ partitioned models:
+
+- `create_partition`
+  - Dynamically create new partition with hashed partition key divided by _modulus_ equals _remainder_
+  - Required arg: `modulus:`, `remainder:`
+- `partition_key_in`
+  - Query for records where partition key in _list_ of values (method operates the same as for _list_ partitions above)
+  - Required arg: list of `values`
 
 #### Examples
 
@@ -275,6 +521,15 @@ class SomeListRecord < ApplicationRecord
 end
 ```
 
+Configure model backed by a _hash_ partitioned table to get access to the methods described above:
+
+```ruby
+class SomeHashRecord < ApplicationRecord
+  # symbol is used for partition keys referring to individual columns
+  hash_partition_by :id
+end
+```
+
 Dynamically create new partition from _range_ partitioned model:
 
 ```ruby
@@ -289,19 +544,26 @@ Dynamically create new partition from _list_ partitioned model:
 SomeListRecord.create_partition(values: 200..300)
 ```
 
+Dynamically create new partition from _hash_ partitioned model:
+
+```ruby
+# additional options include: "name:" and "primary_key:"
+SomeHashRecord.create_partition(modulus: 2, remainder: 1)
+```
+
 For _range_ partitioned model, query for records where partition key in _range_ of values:
 
 ```ruby
 SomeRangeRecord.partition_key_in("2019-06-08", "2019-06-10")
 ```
 
-For _list_ partitioned model, query for records where partition key in _list_ of values:
+For _list_ and _hash_ partitioned models, query for records where partition key in _list_ of values:
 
 ```ruby
 SomeListRecord.partition_key_in(1, 2, 3, 4)
 ```
 
-For both _range and list_ partitioned models, query for records matching partition key:
+For all partitioned models, query for records matching partition key:
 
 ```ruby
 SomeRangeRecord.partition_key_eq(Date.current)
@@ -309,15 +571,15 @@ SomeRangeRecord.partition_key_eq(Date.current)
 SomeListRecord.partition_key_eq(100)
 ```
 
-For both _range and list_ partitioned models, retrieve currently attached partitions:
+For all partitioned models, retrieve currently attached partitions:
 
 ```ruby
 SomeRangeRecord.partitions
 
-SomeListRecord.partitions
+SomeListRecord.partitions(include_subpartitions: true) # Include nested subpartitions
 ```
 
-For both _range and list_ partitioned models, retrieve ActiveRecord model scoped to individual partition:
+For both all partitioned models, retrieve ActiveRecord model scoped to individual partition:
 
 ```ruby
 SomeRangeRecord.in_partition(:some_range_records_partition_name)
@@ -325,9 +587,35 @@ SomeRangeRecord.in_partition(:some_range_records_partition_name)
 SomeListRecord.in_partition(:some_list_records_partition_name)
 ```
 
+To create _range_ partitions by month for previous, current and next months it's possible to use this example. To automate creation of partitions, run `Log.maintenance` every day with cron:
+
+```ruby
+class Log < ApplicationRecord
+  range_partition_by { '(created_at::date)' }
+
+  def self.maintenance
+    partitions = [Date.today.prev_month, Date.today, Date.today.next_month]
+
+    partitions.each do |day|
+      name = Log.partition_name_for(day)
+      next if ActiveRecord::Base.connection.table_exists?(name)
+      Log.create_partition(
+        name: name,
+        start_range: day.beginning_of_month,
+        end_range: day.next_month.beginning_of_month
+      )
+    end
+  end
+
+  def self.partition_name_for(day)
+    "logs_y#{day.year}_m#{day.month}"
+  end
+end
+```
+
 For more examples, take a look at the model integration specs:
 
-- https://github.com/rkrage/pg_party/tree/documentation/spec/integration/model
+- https://github.com/rkrage/pg_party/tree/master/spec/integration/model
 
 ## Development
 

data/lib/pg_party.rb

@@ -1,8 +1,28 @@
 # frozen_string_literal: true
 
 require "pg_party/version"
+require "pg_party/config"
+require "pg_party/cache"
 require "active_support"
 
+module PgParty
+  @config = Config.new
+  @cache = Cache.new
+
+  class << self
+    attr_reader :config, :cache
+
+    def configure(&blk)
+      blk.call(config)
+    end
+
+    def reset
+      @config = Config.new
+      @cache = Cache.new
+    end
+  end
+end
+
 ActiveSupport.on_load(:active_record) do
   require "pg_party/model/methods"
 
@@ -14,10 +34,11 @@ ActiveSupport.on_load(:active_record) do
     PgParty::Adapter::AbstractMethods
   )
 
-  require "pg_party/hacks/schema_cache"
+  require "active_record/tasks/postgresql_database_tasks"
+  require "pg_party/hacks/postgresql_database_tasks"
 
-  ActiveRecord::ConnectionAdapters::SchemaCache.include(
-    PgParty::Hacks::SchemaCache
+  ActiveRecord::Tasks::PostgreSQLDatabaseTasks.prepend(
+    PgParty::Hacks::PostgreSQLDatabaseTasks
   )
 
   begin