repertoire-faceting 0.7.3 → 0.7.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9f33047f3e67ebe050a869ec98d8f90042c284de
-  data.tar.gz: 66d4aa841d23714ede620a6ea56f2947f107a97c
+  metadata.gz: 6d22b7d7430b564a39b9dd864f7da3fd9fc28003
+  data.tar.gz: 5c51fc9c8de95977c82cf4d1e03ec1d53af3fe2a
 SHA512:
-  metadata.gz: 5d86d643b9c881419b6dfe4f3608a56160193775057fc4f645f4b4367f10416cc5968e7f85a0f8059acc9a6c0f5cdc7ebf01a571d1bdc74e6661aa253f6a15e3
-  data.tar.gz: 890a6a07e726450554e058065f0e1fd2471805b8b4e6a751d87d564b5bc3dee58567176b56e8c445fc3df7f12009a9882d7eb2f854966d7facd0c82dce447675
+  metadata.gz: 270ea2290d57503544d37200f525f875f32798b95bc72d414f2ba2f45eb956ed3bf56690005f8bb93a83f111d43a2f9dcc79752dc78a81adacd77dfdc2085239
+  data.tar.gz: a49a2376295d12ba0f1cb6151ef94bff18ccd1aa352ec75e5189eababc8d8ee33233e88233c5ef0542878397f7a552d2ecf938c7164fb8d7c98132bdfb8652e7

data/FAQ

@@ -20,6 +20,67 @@
 	In cases where you do not have superuser access to the deployment host (e.g. Heroku) and so cannot run "rake db:faceting:extensions:install", you can get use the connection's "faceting_api_sql" method to load the API by hand. See the repertoire-faceting-example application's migrations for a concrete example.
 
 
+*Q* Caching for the faceted browser.
+
+*A* Because the opening page of a faceted search is also the most computationally-intensive to produce (it integrates
+counts across the entire dataset), the faceted browser is radically faster when configured to cache results. As of version
+7.4, the controller mixin sets HTTP cache headers in responses to reasonable defaults.
+
+  The short story is that models should have an updated_at column, which is used to compute cache keys. For models with many
+  items, you may wish to add an index on updated_at.
+
+  The detailed story:
+
+  - Requests without an HTTP etag header are processed normally
+
+  - For results queries, a cache key is built that combines the model's updated_at column with a count of the number of rows
+    in the table. If you use Rails' built-in timestamps, this will expire the cache after updating and deleting items.
+
+  - For facet counts on indexed facets, the most recent index refresh is used to construct a cache key. This ensures caches
+    expire when you call "<Model>.index_facets" to refresh the indices (and not before).
+
+  - For facet counts on unindexed facets, the model table's cache key is used.
+
+  - If the model table has no updated_at column, caching falls back to Rails default. (Your query will execute every time.)
+
+  This arrangement caches the most commonly accessed (and slowest) queries in the user's browser. Hence the first
+  request from a new session will load more slowly than consecutive ones. To speed up access across all sessions,
+  configure your Rails app to use an intermediate server cache, e.g. with Rack::Cache and memcached or Varnish.
+
+  If you choose to over-ride the results web-service in your own controller, you can easily reuse Repertoire Faceting's
+  cache settings by checking the value of <Model>.facet_cache_key:
+
+    class TodoslistController < ApplicationController
+      include Repertoire::Faceting::Controller
+      ...
+      def results
+        filter = params[:filter] || {}
+        if stale?(base.facet_cache_key)
+          @results = base.refine(filter)
+          respond_with @results
+        end
+      end
+      ...
+
+      See https://signalvnoise.com/posts/3113-how-key-based-cache-expiration-works for context.
+
+  *Caveats*  The following caveats apply to Rails HTTP header based caching in general:
+
+  - You must make sure your ActiveRecord associations touch the parent table's updated_at column when updating, or you
+    may receive stale data. For example, if you are faceting over todos stored in a joined table:
+
+    class Todo < ActiveRecord::Base
+      belongs_to :todolist, touch: true
+    end
+
+  - It is recommended to expire all cache keys on deploying a new version of any application. However, as of Rails 4.0.5, there
+    is still no standard convention for versioning apps. It may be possible to automate this in Heroku.  See
+    http://stackoverflow.com/questions/8792716/reflecting-heroku-push-version-within-the-app.
+
+  - Because the default strategy caches items in the user agent's browser, the first load of a faceted browser session will be
+    slower than the rest. Installing a server-side cache like Rack::Cache will alleviate this.
+
+
 == About facet indexing and the signature SQL type
 
 *Q* What's the scalability of this thing?
@@ -32,6 +93,13 @@
 *A* Make sure the facet indices aren't empty.  Running '<Model>.index_facets([])' from the Rails console will drop them all.
 
 
+*Q* My facet count values are out of date with the model.
+
+*A* If your facets are indexed, the indexes must be refreshed from the model table periodically.  Running '<Model>.index_facets'
+(no arguments) in the Rails console will refresh them.  In a production environment, run this periodically for each indexed model
+via a cron task.
+
+
 *Q* Can I facet over multiple models?
 
 *A* Not currently.  However, this may be possible using an ActiveRecord polymorphic relation on the main model.

data/TODO

@@ -16,6 +16,9 @@ DESIRED FEATURES / IMPROVEMENTS.
 
 TODO
 
+-- adapter tests for existence of materialized views too often
+-- need a way to merge version number into etags?
+
 -- add info on CORS to FAQ                                      DONE
 -- ensure pushState respects repertoire.defaults.path_prefix    DONE
 

data/ext/Makefile

@@ -8,7 +8,7 @@
 #
 #-------------------------------------------------------------------------
 
-API_VERSION = 0.7.3
+API_VERSION = 0.7.4
 
 MODULES = signature/signature
 EXTENSION = signature/faceting \

data/ext/bytea/faceting_bytea.control

@@ -2,5 +2,5 @@
  
 comment = 'API for faceted indexing and queries (based on plv8 + bytea bitmaps)'
 requires = 'plv8, plpgsql'
-default_version = '0.7.3'
+default_version = '0.7.4'
 schema = 'facet'

data/ext/signature/faceting.control

@@ -2,5 +2,5 @@
  
 comment = 'API for faceted indexing and queries (based on custom C bitmap type)'
 requires = plpgsql
-default_version = '0.7.3'
+default_version = '0.7.4'
 schema = 'facet'

data/ext/varbit/faceting_varbit.control

@@ -3,5 +3,5 @@
 comment = 'API for faceted indexing and queries (based on builtin VARBIT bit strings)'
 requires = plpgsql
 superuser = false
-default_version = '0.7.3'
+default_version = '0.7.4'
 schema = 'facet'

data/lib/repertoire-faceting/adapters/postgresql_adapter.rb

@@ -65,6 +65,23 @@ module Repertoire
         Float(result)
       end
 
+      #
+      # Methods for detecting table content changes
+      #
+      # (If a later version of PostgreSQL can hashcode a table/timestamp in the system catalog,
+      # switch to use that instead.)
+      #
+      def stat_table(table_name, column="updated_at")
+        sql = "SELECT COUNT(#{column}), MAX(#{column}) AS timestamp FROM #{table_name}"
+        result = select_one(sql)
+        result = HashWithIndifferentAccess.new({
+          :count     => Integer(result["count"]),
+          :timestamp => Time.parse(result["timestamp"])
+        })
+
+        result
+      end
+
       #
       # Methods for running facet value counts
       #

data/lib/repertoire-faceting/controller.rb

@@ -35,8 +35,11 @@ module Repertoire
     #                   when 'alphanumeric' then ["#{facet} ASC"]
     #                   when 'count'        then ["count DESC", "#{facet} ASC"]
     #                 end
-    #       @counts = base.refine(filter).order(sorting).count(facet)
-    #       render :json => @counts.to_a
+    #
+    #       if stale?(base.facet_cache_key, :public => true)
+    #         @counts = base.refine(filter).order(sorting).count(facet)
+    #         render :json => @counts.to_a
+    #       end
     #     end
     #
     module Controller
@@ -44,26 +47,43 @@ module Repertoire
       # Web-service to return value, count pairs for a given facet, given existing filter refinements
       # on other facets in the context.  Over-ride this method if you need to specify additional 
       # query params for faceting.
+      #
+      # Public HTTP cache headers are set, in the following order:
+      #   - by the facet index table (if present)
+      #   - by the facet model table (if it has an updated_at column)
+      #   - otherwise, no HTTP cache header is set
+      #
       def counts
         facet  = params[:facet]
         filter = params[:filter] || {}
         raise "Unkown facet #{facet}" unless base.facet?(facet)
-        
-        @counts = base.refine(filter).count(facet)
 
-        render :json => @counts.to_a
+        if stale?(base.facet_cache_key(facet), :public => true)
+
+          @counts = base.refine(filter).count(facet)
+          render :json => @counts.to_a
+
+        end
       end
 
       # Web-service to return the results of a query, given existing filter requirements.  Over-ride
       # this method if you need to specify additional query parms for faceting results.
+      #
+      # Private HTTP cache headers are set:
+      #   - by the facet model table (if it has an updated_at column)
+      #   - otherwise, no HTTP cache header is set
+      #
       def results
         filter = params[:filter] || {}
-
-        @results = base.refine(filter).to_a
         
-        respond_to do |format|
-          format.html { render @results, :layout => false }
-          format.json { render :json => @results }
+        if stale?(base.facet_cache_key)
+
+          @results = base.refine(filter).to_a
+
+          respond_to do |format|
+            format.html { render @results, :layout => false }
+            format.json { render :json => @results }
+          end
         end
       end
       

data/lib/repertoire-faceting/facets/abstract_facet.rb

@@ -62,6 +62,15 @@ module Repertoire
           indexed_facets.map(&:to_sym).include?(facet_name)
         end
 
+        # Returns a row count and timestamp for the facet's index table,
+        # or nil if it is unindexed.
+        #
+        # If a future PostgreSQL timestamps materialized view refreshes
+        # via the system catalogs, that value should be used instead.
+        def stat_table
+          connection.stat_table(facet_index_name) if facet_indexed?
+        end
+
         protected
 
         # Return a facet's index table name

data/lib/repertoire-faceting/facets/basic_facet.rb

@@ -34,7 +34,7 @@ module Repertoire #:nodoc:
         def create_index
           col = group_values.first
           rel = only(:where, :joins, :group)
-          sql = rel.select(["#{col} AS #{facet_name}", "facet.signature(#{table_name}.#{faceting_id})"]).to_sql
+          sql = rel.select(["#{col} AS #{facet_name}", "facet.signature(#{table_name}.#{faceting_id})", "now() AS updated_at"]).to_sql
 
           connection.create_materialized_view(facet_index_name, sql)
         end

data/lib/repertoire-faceting/facets/nested_facet.rb

@@ -50,12 +50,12 @@ module Repertoire
             end
             rel = rel.group(columns[0..level])
 
-            queries << rel.select(["#{level+1} AS level", "facet.signature(#{table_name}.#{faceting_id})"]).to_sql
+            queries << rel.select(["#{level+1} AS level", "facet.signature(#{table_name}.#{faceting_id})", "now() AS updated_at"]).to_sql
           end
 
           # Root of tree
           empty_cols = columns.map { |col| "NULL AS #{col}"}
-          queries << only(:where).select(empty_cols + ["0 AS level", "facet.signature(#{table_name}.#{faceting_id})"]).to_sql
+          queries << only(:where).select(empty_cols + ["0 AS level", "facet.signature(#{table_name}.#{faceting_id})", "now() AS updated_at"]).to_sql
 
           # Give the fullest index first (i.e. leaves of the tree), so the database
           # can resolve types before encountering any NULL values (i.e. values of

data/lib/repertoire-faceting/model.rb

@@ -11,7 +11,7 @@ module Repertoire
         base.singleton_class.delegate :refine, :minimum, :nils, :reorder, :to_sql, :to => :scoped_all
 
         base.class_attribute(:facets)
-        base.facets = {}
+        base.facets = HashWithIndifferentAccess.new
       end
 
       #
@@ -254,6 +254,41 @@ module Repertoire
           connection.signature_wastage(table_name, signature_column)
         end
 
+        # Returns the row count and most recent update timestamp for a model table; or nil if
+        # there is no updated_at field.
+        def stat_table(timestamp_column = nil)
+          timestamp_column ||= "updated_at"
+          connection.stat_table(table_name) if column_names.include?(timestamp_column)
+        end
+
+        # Return a key suitable for use in HTTP headers, either for the base model table or one of its
+        # facets.
+        #
+        # If the name of an indexed facet is provided, the timestamp of the facet index table is used
+        # to construct the facet's cache key. This ensures that facet counts expire when the facet is
+        # re-indexed (and not before).
+        #
+        # If the name of an unindexed facet is given, a cache key for the entire model table is provided.
+        # This ensures that faceted browsers over live data expire facet count caches when the base
+        # model table is updated.
+        #
+        # Calling facet_cache_key with no arguments returns a cache key for the entire model table. The
+        # key is a combination of the most recent update_at column value and the table row count. If
+        # the model has no updated_at attribute, caching is disabled.
+        #
+        # See the FAQ for additional information.
+        def facet_cache_key(facet = nil)
+          facet = facet.try(:to_s)
+          result = nil
+
+          stats = facets[facet].stat_table if facet_names.include?(facet)
+          stats ||= stat_table
+
+          result = { :etag => stats[:count], :last_modified => stats[:timestamp] } if stats.present?
+
+          result
+        end
+
         # Once clients have migrated to Rails 4, delete and replace with 'all' where this is called
         #
         # c.f. http://stackoverflow.com/questions/18198963/with-rails-4-model-scoped-is-deprecated-but-model-all-cant-replace-it

data/lib/repertoire-faceting/version.rb

@@ -1,5 +1,5 @@
 module Repertoire
   module Faceting #:nodoc:
-    VERSION = "0.7.3"
+    VERSION = "0.7.4"
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: repertoire-faceting
 version: !ruby/object:Gem::Version
-  version: 0.7.3
+  version: 0.7.4
 platform: ruby
 authors:
 - Christopher York
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-06-19 00:00:00.000000000 Z
+date: 2014-06-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -83,9 +83,9 @@ files:
 - ext/bytea/faceting_bytea.control
 - ext/common/util.sql
 - ext/extconf.rb
-- ext/faceting--0.7.2.sql
-- ext/faceting_bytea--0.7.2.sql
-- ext/faceting_varbit--0.7.2.sql
+- ext/faceting--0.7.4.sql
+- ext/faceting_bytea--0.7.4.sql
+- ext/faceting_varbit--0.7.4.sql
 - ext/signature/faceting.control
 - ext/signature/signature.c
 - ext/signature/signature.o