chrono_model 0.5.3 → 0.8.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: da72ced6e70c0fc98ba5b78417421557dfd22708
+  data.tar.gz: 6f6c9b32d810cceb820f5e6b502baef95add1c26
+SHA512:
+  metadata.gz: 1f7c0e8f7e1db361fb037295122f03562359ba497495a279e53b9aa7d48cbecac4c2396bd5cff88ea609a5a2a62cc70060d24c9e8047f9482dca27d2633e61de
+  data.tar.gz: 7226296b3c3e0442f9c9795e556711c8a8d7cfc963da315eaaa8f50f55c91146c356ab93c3045646c2d564086ecafb7ee2df912a0082e96942e2391bbdc56653

data/.rspec

@@ -1,2 +1,2 @@
 --color
---format progress
+--format Fuubar

data/.travis.yml

@@ -1,12 +1,19 @@
 rvm:
   - 1.9.3
   - 2.0.0
+  - 2.1.1
 
 services:
   - postgresql
 
 bundler_args: --without development
 
+before_install:
+  - sudo service postgresql stop
+  - sudo apt-get update
+  - sudo apt-get install postgresql-9.3 postgresql-plpython-9.3
+  - sudo service postgresql start 9.3
+
 before_script:
   - psql -c "CREATE DATABASE chronomodel;" -U postgres
 

data/Gemfile

@@ -4,11 +4,20 @@ source 'https://rubygems.org'
 gemspec
 
 group :development do
-  gem 'debugger'
   gem 'pry'
+  gem 'hirb'
+
+  gem(
+    case RUBY_VERSION.to_f
+    when 1.8 then 'ruby-debug'
+    when 1.9 then 'debugger'
+    else          'byebug'
+    end
+  )
 end
 
 group :test do
   gem 'rspec'
   gem 'rake'
+  gem 'fuubar'
 end

data/LICENSE

@@ -1,4 +1,6 @@
-Copyright (c) 2012 Marcello Barnaba <m.barnaba@ifad.org>
+Copyright (c) 2012-2014 Marcello Barnaba <m.barnaba@ifad.org>
+Copyright (c) 2012-2014 Peter J. Brindisi <p.brindisi@ifad.org>
+Copyright (c) 2012-2014 IFAD
 
 MIT License
 

data/README.md

@@ -1,228 +1,331 @@
-# ChronoModel
+# Temporal database system on PostgreSQL using [updatable views][], [table inheritance][] and [INSTEAD OF triggers][]. [![Build Status][build-status-badge]][build-status] [![Dependency Status][deps-status-badge]][deps-status] [![Code Climate][code-analysis-badge]][code-analysis]
 
-A temporal database system on PostgreSQL using
-[table inheritance](http://www.postgresql.org/docs/9.0/static/ddl-inherit.html) and
-[the rule system](http://www.postgresql.org/docs/9.0/static/rules-update.html).
+ChronoModel does what Oracle sells as "Flashback Queries", but with standard
+SQL on free PostgreSQL. Academically speaking, ChronoModel implements a
+[Type-2 Slowly-Changing Dimension][wp-scd-2] with [history tables][wp-scd-4].
 
-This is a data structure for a
-[Slowly-Changing Dimension Type 2](http://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2)
-temporal database, implemented using only [PostgreSQL](http://www.postgresql.org) >= 9.0 features.
+All history keeping happens inside the database system, freeing application
+code from having to deal with it. ChronoModel implements all the required
+features in Ruby on Rails' ORM to leverage the database temporal structure
+beneath.
 
-[![Build Status](https://travis-ci.org/ifad/chronomodel.png?branch=master)](https://travis-ci.org/ifad/chronomodel)
-[![Dependency Status](https://gemnasium.com/ifad/chronomodel.png)](https://gemnasium.com/ifad/chronomodel)
-[![Code Climate](https://codeclimate.com/github/ifad/chronomodel.png)](https://codeclimate.com/github/ifad/chronomodel)
 
-All the history recording is done inside the database system, freeing the application code from
-having to deal with it.
+## Design
 
-The application model is backed by an updatable view that behaves exactly like a plain table, while
-behind the scenes the database redirects the queries to concrete tables using
-[the rule system](http://www.postgresql.org/docs/9.0/static/rules-update.html) for most of the cases,
-and using an `AFTER FOR EACH ROW` trigger only for the `INSERT` case on tables
-having a `SERIAL` primary key.
+The application model is backed by an updatable view in the default `public`
+schema that behaves like a plain table to any database client.  When data in
+manipulated on it, INSTEAD OF [triggers][] redirect the manipulations to
+concrete tables.
 
-Current data is hold in a table in the `temporal` [schema](http://www.postgresql.org/docs/9.0/static/ddl-schemas.html),
-while history in hold in another table in the `history` schema. The latter
-[inherits](http://www.postgresql.org/docs/9.0/static/ddl-inherit.html) from the former, to get
-automated schema updates for free. Partitioning of history is even possible but not implemented
-yet.
+Current data is hold in a table in the `temporal` [schema][], while history in
+hold in a specular one in the `history` schema.  The latter [inherits][] from
+the former, to get automated schema updates and for free and other benefits.
 
-The updatable view is created in the default `public` schema, making it visible to Active Record.
+The current time is taken using [`current_timestamp`][], so that multiple data
+manipulations in the same transaction on the same records always create a
+single history entry (they are _squashed_ together).
 
-All Active Record schema migration statements are decorated with code that handles the temporal
-structure by e.g. keeping the view rules in sync or dropping/recreating it when required by your
-migrations.
+[Partitioning][] of history is also possible: this design [fits the
+requirements][partitioning-excl-constraints] but it's not implemented yet.
 
-Data extraction at a single point in time and even `JOIN`s between temporal and non-temporal data
-is implemented using sub-selects and a `WHERE` generated by the provided `TimeMachine` module to
-be included in your models.
+See [README.sql][] for a SQL example defining the machinery for a simple table.
 
-The `WHERE` is optimized using a spatial GiST index in which time is represented represented by
-boxes and the filtering is done using the overlapping (`&&`) geometry operator.
 
-All timestamps are (forcibly) stored in the UTC time zone, bypassing the `AR::Base.config.default_timezone`
-setting.
+## Active Record integration
 
-See [README.sql](https://github.com/ifad/chronomodel/blob/master/README.sql) for the plain SQL
-defining this temporal schema for a single table.
+All Active Record schema migration statements are decorated with code that
+handles the temporal structure by e.g. keeping the triggers in sync or
+dropping/recreating it when required by your migrations.
+
+Data extraction at a single point in time and even `JOIN`s between temporal and
+non-temporal data is implemented using sub-selects and a `WHERE` generated by
+the provided `TimeMachine` module to be included in your models.
+
+The `WHERE` is optimized using [GiST indexes][] on the `tsrange` defining
+record validity. Overlapping history is prevented through [exclusion
+constraints][] and the [btree_gist][] extension.
+
+All timestamps are _forcibly_ stored in as UTC, bypassing the
+`default_timezone` setting.
 
 
 ## Requirements
 
-* Ruby >= 1.9.2
-* Active Record >= 3.2
-* PostgreSQL >= 9.0
+* Ruby >= 1.9.3
+* Active Record >= 4.0
+* PostgreSQL >= 9.3
+* The `btree_gist` PostgreSQL extension
 
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
-    gem 'chrono_model', :git => 'git://github.com/ifad/chronomodel'
+    gem 'chrono_model', github: 'ifad/chronomodel'
 
 And then execute:
 
     $ bundle
 
 
-## Schema creation
+## Configuration
 
-This library hooks all `ActiveRecord::Migration` methods to make them temporal aware.
+Configure your `config/database.yml` to use the `chronomodel` adapter:
+
+```yaml
+development:
+  adapter: chronomodel
+  username: ...
+```
+
+## Schema creation
 
-The only option added is `:temporal => true` to `create_table`:
+ChronoModel hooks all `ActiveRecord::Migration` methods to make them temporal
+aware.
 
-    create_table :countries, :temporal => true do |t|
-      t.string :common_name
-      t.references :currency
-      # ...
-    end
+```ruby
+create_table :countries, temporal: true do |t|
+  t.string     :common_name
+  t.references :currency
+  # ...
+end
+```
 
-That'll create the _current_, its _history_ child table and the _public_ view.
-Every other housekeeping of the temporal structure is handled behind the scenes
-by the other schema statements. E.g.:
+This creates the _temporal_ table, its inherited _history_ one the _public_
+view and all the trigger machinery. Every other housekeeping of the temporal
+structure is handled behind the scenes by the other schema statements. E.g.:
 
- * `rename_table`  - renames tables, views, sequences, indexes and rules
+ * `rename_table`  - renames tables, views, sequences, indexes and triggers
  * `drop_table`    - drops the temporal table and all dependant objects
- * `add_column`    - adds the column to the current table and updates rules
- * `rename_column` - renames the current table column and updates the rules
- * `remove_column` - removes the current table column and updates the rules
- * `add_index`     - creates the index in the history table as well
- * `remove_index`  - removes the index from the history table as well
+ * `add_column`    - adds the column to the current table and updates triggers
+ * `rename_column` - renames the current table column and updates the triggers
+ * `remove_column` - removes the current table column and updates the triggers
+ * `add_index`     - creates the index in both _temporal_ and _history_ tables
+ * `remove_index`  - removes the index from both tables
 
 
 ## Adding Temporal extensions to an existing table
 
 Use `change_table`:
 
-    change_table :your_table, :temporal => true
+```ruby
+change_table :your_table, temporal: true
+```
 
 If you want to also set up the history from your current data:
 
-    change_table :your_table, :temporal => true, :copy_data => true
+```ruby
+change_table :your_table, temporal: true, copy_data: true
+```
 
 This will create an history record for each record in your table, setting its
-validity from midnight, January 1st, 1 CE. You can set a specific validity
-with the `:validity` option:
+validity from midnight, January 1st, 1 CE. You can set a specific validity with
+the `:validity` option:
+
+```ruby
+change_table :your_table, :temporal => true, :copy_data => true, :validity => '1977-01-01'
+```
+
+Please note that `change_table` requires you to use *old_style* `up` and
+`down` migrations. It cannot work with Rails 3-style `change` migrations.
+
+## Selective Journaling
+
+By default UPDATEs only to the `updated_at` field are not recorded in the
+history.
+
+You can also choose which fields are to be journaled, passing the following
+options to `create_table`:
+
+  * `:journal => %w( fld1 fld2 .. .. )` - record changes in the history only when changing specified fields
+  * `:no_journal => %w( fld1 fld2 .. )` - do not record changes to the specified fields
+  * `:full_journal => true`             - record changes to *all* fields, including `updated_at`.
+
+These options are stored as JSON in the [COMMENT][] area of the public view,
+alongside with the ChronoModel version that created them.
 
-    change_table :your_table, :temporal => true, :copy_data => true, :validity => '1977-01-01'
+This is visible in `psql` if you issue a `\d+`. Example after a test run:
+
+    chronomodel=# \d+
+                                                           List of relations
+     Schema |     Name      |   Type   |    Owner    |    Size    |                           Description
+    --------+---------------+----------+-------------+------------+-----------------------------------------------------------------
+     public | bars          | view     | chronomodel | 0 bytes    | {"temporal":true,"chronomodel":"0.7.0.alpha"}
+     public | foos          | view     | chronomodel | 0 bytes    | {"temporal":true,"chronomodel":"0.7.0.alpha"}
+     public | plains        | table    | chronomodel | 0 bytes    |
+     public | test_table    | view     | chronomodel | 0 bytes    | {"temporal":true,"journal":["foo"],"chronomodel":"0.7.0.alpha"}
 
 
 ## Data querying
 
-A model backed by a temporal view will behave like any other model backed by a
-plain table. If you want to do as-of-date queries, you need to include the
-`ChronoModel::TimeMachine` module in your model.
+Include the `ChronoModel::TimeMachine` module in your model.
 
-    module Country < ActiveRecord::Base
-      include ChronoModel::TimeMachine
+```ruby
+module Country < ActiveRecord::Base
+  include ChronoModel::TimeMachine
 
-      has_many :compositions
-    end
+  has_many :compositions
+end
+```
 
-This will create a `Country::History` model inherited from `Country`, and it
-will make an `as_of` class method available to your model. E.g.:
+This will create a `Country::History` model inherited from `Country`, and add
+an `as_of` class method.
 
-    Country.as_of(1.year.ago)
+```ruby
+Country.as_of(1.year.ago)
+```
 
 Will execute:
 
-    SELECT "countries".* FROM (
-      SELECT "history"."countries".* FROM "history"."countries"
-      WHERE box(
-        point( date_part( 'epoch', '#{1.year.ago.utc}'::timestamp ), 0 ),
-        point( date_part( 'epoch', '#{1.year.ago.utc}'::timestamp ), 0 )
-      ) &&
-      box(
-        point( date_part( 'epoch', "history"."addresses"."valid_from" ), 0 ),
-        point( date_part( 'epoch', "history"."addresses"."valid_to"   ), 0 ),
-      )
-    ) AS "countries"
+```sql
+SELECT "countries".* FROM (
+  SELECT "history"."countries".* FROM "history"."countries"
+  WHERE '#{1.year.ago}' <@ "history"."countries"."validity"
+) AS "countries"
+```
 
-This work on associations using temporal extensions as well:
+The returned `ActiveRecord::Relation` will then hold and pass along the
+timestamp given to the first `.as_of()` call to queries on associated entities.
+E.g.:
 
-    Country.as_of(1.year.ago).first.compositions
+```ruby
+Country.as_of(1.year.ago).first.compositions
+```
 
 Will execute:
 
-    # ... countries history query ...
-    LIMIT 1
+```sql
+SELECT "countries".*, '#{1.year.ago}' AS as_of_time FROM (
+  SELECT "history"."countries".* FROM "history"."countries"
+  WHERE '#{1.year.ago}' <@ "history"."countries"."validity"
+) AS "countries" LIMIT 1
+```
 
-    SELECT * FROM  (
-      SELECT "history"."compositions".* FROM "history"."compositions"
-      WHERE box(
-        point( date_part( 'epoch', '#{above_timestamp}'::timestamp ), 0 ),
-        point( date_part( 'epoch', '#{above_timestamp}'::timestamp ), 0 )
-      ) &&
-      box(
-        point( date_part( 'epoch', "history"."addresses"."valid_from" ), 0 ),
-        point( date_part( 'epoch', "history"."addresses"."valid_to"   ), 0 ),
-      )
-    ) AS "compositions" WHERE country_id = X
+and then, using the above fetched `as_of_time` timestamp, expand to:
 
-And `.joins` works as well:
+```sql
+SELECT * FROM  (
+  SELECT "history"."compositions".* FROM "history"."compositions"
+  WHERE '#{as_of_time}' <@ "history"."compositions"."validity"
+) AS "compositions" WHERE country_id = X
+```
 
-    Country.as_of(1.month.ago).joins(:compositions)
+`.joins` works as well:
 
-Will execute:
+```ruby
+Country.as_of(1.month.ago).joins(:compositions)
+```
 
-    SELECT "countries".* FROM (
-      # .. countries history query ..
-    ) AS "countries" INNER JOIN (
-      # .. compositions history query ..
-    ) AS "compositions" ON compositions.country_id = countries.id
+Expands to:
 
-More methods are provided, see the
-[TimeMachine](https://github.com/ifad/chronomodel/blob/master/lib/chrono_model/time_machine.rb) source
-for more information.
+```sql
+SELECT "countries".* FROM (
+  SELECT "history"."countries".* FROM "history"."countries"
+  WHERE '#{1.month.ago}' <@ "history"."countries"."validity"
+) AS "countries" INNER JOIN (
+  SELECT "history"."compositions".* FROM "history"."compositions"
+  WHERE '#{1.month.ago}' <@ "history"."compositions"."validity"
+) AS "compositions" ON compositions.country_id = countries.id
+```
 
+More methods are provided, see the [TimeMachine][] source for more information.
 
 ## Running tests
 
-You need a running Postgresql instance. Create `spec/config.yml` with the
+You need a running PostgreSQL 9.3 instance. Create `spec/config.yml` with the
 connection authentication details (use `spec/config.yml.example` as template).
 
-Run `rake`. SQL queries are logged to `spec/debug.log`. If you want to see
-them in your output, use `rake VERBOSE=true`.
+You need to connect as  a database superuser, because specs need to create the
+`btree_gist` extension.
+
+Run `rake`. SQL queries are logged to `spec/debug.log`. If you want to see them
+in your output, use `rake VERBOSE=true`.
+
+## Usage with JSON columns
+
+[JSON][json-type] does not provide an [equality operator][json-func].
+As both unnecessary update suppression and selective journaling require
+comparing the OLD and NEW rows fields, this fails by default.
+
+ChronoModel provides a naive JSON equality operator using a naive
+comparison of JSON objects [implemented in pl/python][json-opclass].
+
+To load the opclass you can use the `ChronoModel::Json.create`
+convenience method. If you don't use JSON don't bother doing this.
 
 ## Caveats
 
- * The rules and temporal indexes cannot be saved in schema.rb. The AR
+ * Rails 4 support requires disabling tsrange parsing support, as it
+   [is broken][r4-tsrange-broken] and  [incomplete][r4-tsrange-incomplete]
+   as of now, mainly due to a [design clash with ruby][pg-tsrange-and-ruby].
+
+ * There is (yet) no upgrade path from [v0.5][chronomodel-0.5],
+   (PG 9.0-compatible, `box()` and hacks) to v0.6 and up (9.3-only, `tsrange`
+   and _less_ hacks).
+
+ * The triggers and temporal indexes cannot be saved in schema.rb. The AR
    schema dumper is quite basic, and it isn't (currently) extensible.
    As we're using many database-specific features, Chronomodel forces the
    usage of the `:sql` schema dumper, and included rake tasks override
    `db:schema:dump` and `db:schema:load` to do `db:structure:dump` and
-   `db:structure:load`. Two helper tasks are also added, `db:data:dump`
-   and `db:data:load`.
-
- * `.includes` still doesn't work, but it'll fixed.
-
- * The history queries are very verbose, they should be factored out using a
-   `FUNCTION`.
+   `db:structure:load`.
+   Two helper tasks are also added, `db:data:dump` and `db:data:load`.
 
- * The migration statements extension is implemented using a Man-in-the-middle
-   class that inherits from the PostgreSQL adapter, and that relies on some
-   private APIs. This should be made more maintainable, maybe by requiring
-   the use of `adapter: chronomodel` or `adapter: chrono_postgresql`.
+ * `.includes` is quirky when using `.as_of`.
 
- * Savepoints are disabled, because there is
-   [currently](http://archives.postgresql.org/pgsql-hackers/2012-08/msg01094.php)
-   no way to identify a subtransaction belonging to the current transaction.
-
- * The choice of using subqueries instead of [Common Table Expressions](http://www.postgresql.org/docs/9.0/static/queries-with.html)
+ * The choice of using subqueries instead of [Common Table Expressions][]
    was dictated by the fact that CTEs [currently acts as an optimization
-   fence](http://archives.postgresql.org/pgsql-hackers/2012-09/msg00700.php).
-   If it will be possible [to opt-out of the
-   fence](http://archives.postgresql.org/pgsql-hackers/2012-10/msg00024.php)
-   in the future, they will be probably be used again as they were [in the
-   past](https://github.com/ifad/chronomodel/commit/18f4c4b), because the
-   resulting queries are much more readable, and do not inhibit using
-   `.from()` from ARel.
+   fence][cte-optimization-fence].
+   If it will be possible [to opt-out of the fence][cte-opt-out-fence] in
+   the future, they will be probably be used again as they were [in the
+   past][chronomodel-cte-impl], because the resulting queries were more
+   readable, and do not inhibit using `.from()` on the `AR::Relation`.
 
 
 ## Contributing
 
  1. Fork it
  2. Create your feature branch (`git checkout -b my-new-feature`)
- 3. Commit your changes (`git commit -am 'Added some feature'`)
+ 3. Commit your changes (`git commit -am 'Added some great feature'`)
  4. Push to the branch (`git push origin my-new-feature`)
  5. Create new Pull Request
+
+
+[build-status]: https://travis-ci.org/ifad/chronomodel
+[build-status-badge]: https://travis-ci.org/ifad/chronomodel.png?branch=master
+[deps-status]: https://gemnasium.com/ifad/chronomodel
+[deps-status-badge]: https://gemnasium.com/ifad/chronomodel.png
+[code-analysis]: https://codeclimate.com/github/ifad/chronomodel
+[code-analysis-badge]: https://codeclimate.com/github/ifad/chronomodel.png
+
+[updatable views]: http://www.postgresql.org/docs/9.3/static/sql-createview.html#SQL-CREATEVIEW-UPDATABLE-VIEWS
+[table inheritance]: http://www.postgresql.org/docs/9.3/static/ddl-inherit.html
+[INSTEAD OF triggers]: http://www.postgresql.org/docs/9.3/static/sql-createtrigger.html
+[wp-scd-2]: http://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2
+[wp-scd-4]: http://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_4
+[triggers]: http://www.postgresql.org/docs/9.3/static/trigger-definition.html
+[schema]: http://www.postgresql.org/docs/9.3/static/ddl-schemas.html
+[inherits]: http://www.postgresql.org/docs/9.3/static/ddl-inherit.html
+[`current_timestamp`]: http://www.postgresql.org/docs/9.3/interactive/functions-datetime.html#FUNCTIONS-DATETIME-TABLE
+
+[Partitioning]: http://www.postgresql.org/docs/9.3/static/ddl-partitioning.html)
+[partitioning-excl-constraints]: http://www.postgresql.org/docs/9.3/static/ddl-partitioning.html#DDL-PARTITIONING-CONSTRAINT-EXCLUSION
+[README.sql]: https://github.com/ifad/chronomodel/blob/master/README.sql
+[GiST indexes]: http://www.postgresql.org/docs/9.3/static/gist.html
+[exclusion constraints]: http://www.postgresql.org/docs/9.3/static/sql-createtable.html#SQL-CREATETABLE-EXCLUDE
+[btree_gist]: http://www.postgresql.org/docs/9.3/static/btree-gist.html
+[COMMENT]: http://www.postgresql.org/docs/9.3/static/sql-comment.html
+[TimeMachine]: https://github.com/ifad/chronomodel/blob/master/lib/chrono_model/time_machine.rb
+
+[r4-tsrange-broken]: https://github.com/rails/rails/pull/13793#issuecomment-34608093
+[r4-tsrange-incomplete]: https://github.com/rails/rails/issues/14010)
+[pg-tsrange-and-ruby]: https://bugs.ruby-lang.org/issues/6864
+[chronomodel-0.5]: https://github.com/ifad/chronomodel/tree/c2daa0f
+[Common Table Expressions]: http://www.postgresql.org/docs/9.3/static/queries-with.html
+[cte-optimization-fence]: http://archives.postgresql.org/pgsql-hackers/2012-09/msg00700.php
+[cte-opt-out-fence]: http://archives.postgresql.org/pgsql-hackers/2012-10/msg00024.php
+[chronomodel-cte-impl]: https://github.com/ifad/chronomodel/commit/18f4c4b
+
+[json-type]: http://www.postgresql.org/docs/9.3/static/datatype-json.html
+[json-func]: http://www.postgresql.org/docs/9.3/static/functions-json.html
+[json-opclass]: https://github.com/ifad/chronomodel/blob/master/sql/json_ops.sql