repertoire-faceting 0.5.5 → 0.6.0

data/README

@@ -1,12 +1,12 @@
 === Repertoire Faceting README
 
-Repertoire Faceting is highly scalable and extensible module for creating database-driven faceted browsers in Rails 3. It consists of three components: (1) a native PostgreSQL data type for constructing fast bitset indices over controlled vocabularies; (2) Rails 3 model and controller mixins that add a faceting API to your existing application; and (3) a set of extensible javascript widgets for building user-interfaces. In only 10-15 lines of new code you can implement a fully-functional faceted browser for your existing Rails data models, with scalability out of the box to over 1,000,000 items.
+Repertoire Faceting is highly scalable and extensible module for creating database-driven faceted browsers in Rails 3 & 4. It consists of three components: (1) a native PostgreSQL data type for constructing fast bitset indices over controlled vocabularies; (2) Rails model and controller mixins that add a faceting API to your existing application; and (3) a set of extensible javascript widgets for building user-interfaces. In only 10-15 lines of new code you can implement a fully-functional faceted browser for your existing Rails data models, with scalability out of the box to over 1,000,000 items.
 
 == Features
 
 Several features distinguish Repertoire Faceting from other faceting systems such as Simile Exhibit, Endeca, and Solr.
 
-Repertoire Faceting is an end-to-end solution that works with your existing database schema and Rails models. There's no need to munge your data into a proprietary format, run a separate facet index server, or construct your own user-interface widgets. (Conversely, however, your project needs to use Rails 3, PostgreSQL, and JQuery.)
+Repertoire Faceting is an end-to-end solution that works with your existing database schema and Rails models. There's no need to munge your data into a proprietary format, run a separate facet index server, or construct your own user-interface widgets. (Conversely, however, your project needs to use Rails, PostgreSQL, and JQuery.)
 
 The module works equally well on small and large data sets, which means there's a low barrier to entry but existing projects can grow easily. In 'training wheels' mode, the module produces SQL queries for facet counts directly from declarations in your model so you can get a project up and running quickly. Then, after your dataset grows beyond a thousand items or so, just add indices as necessary. The module detects and uses these automatically, with no changes to your code or additional SQL necessary.
 
@@ -16,7 +16,7 @@ Both facet widgets and indexes are pluggable and extensible. You can subclass th
 
 Similarly, you can write new facet implementations for novel data types, which automatically detect and index appropriate columns. For example, the module has been used to do facet value counts over GIS data points on a map, by drilling down through associated GIS layers using spatial logic relations.
 
-For an out-of-the box example using Repertoire Faceting, which demonstrates the module's visualization and scalability features, see the example application (http://github.com/yorkc/repertoire-faceting-example).
+For an out-of-the box example using Repertoire Faceting, which demonstrates the module's visualization and scalability features, see the example application (http://github.com/christopheryork/repertoire-faceting-example).
 
 
 == Installation
@@ -29,10 +29,9 @@ See the INSTALL document for a description of how to install the module and buil
 You can run the unit tests from the module's root directory. You will need a local PostgreSQL superuser role with your unix username (use 'createuser -Upostgres').
 
   $ bundle install
-  $ rake db:faceting:build    { sudo will prompt for your password }
-  $ rake db:create
+  $ rake db:setup
   $ rake test
-  
+
 
 == Generating documentation
 
@@ -68,9 +67,9 @@ See Repertoire::Faceting::Facets::AbstractFacet
 It is very useful to create a rake task to update your application's indices. In the project's rake task file:
 
   task :reindex => :environment do
-    Painting.update_indexed_facets([:genre, :era])
+    Painting.index_facets([:genre, :era])
   end
-  
+
 Then run 'rake reindex' whenever you need to update indices manually.
 
 *static* If the facet data is unchanging, use a rake task like the one above to create indices manually while developing or deploying.
@@ -84,7 +83,7 @@ Because repertoire-faceting depends on a native shared library loaded by the Pos
 
   <server>$ bundle install --deployment
   <server>$ export RAILS_ENV=production
-  <server>$ rake db:faceting:build
+  <server>$ rake db:faceting:extensions:install
   <server>$ # ... from here, follow normal deployment procedure
 
 
@@ -96,7 +95,7 @@ There are three direct implementations for faceted classifications like this in
 
 *1:* Explicit controlled vocabulary, multiple valued facet
 
-    genres            plays_genres            plays                      
+    genres            plays_genres            plays
   ----+---------    ---------+----------    ----+------------------+---------
    id | name         play_id | genre_id      id | title            | date ...
   ----+---------    ---------+----------    ----+------------------|---------
@@ -112,7 +111,7 @@ There are three direct implementations for faceted classifications like this in
 
 *2:* Implicit vocabulary, multiple valued facet
 
-    plays_genres            plays                      
+    plays_genres            plays
   ---------+----------    ----+------------------+---------
    play_id | genre_id      id | title            | date ...
   ---------+----------    ----+------------------|---------
@@ -127,8 +126,8 @@ There are three direct implementations for faceted classifications like this in
              ...                ....
 
 *3:* Implicit vocabulary, single valued facet
-  
-    plays                      
+
+    plays
   ----+-----------------+---------+---------
    id | title           | genre   | date ...
   ----+-----------------|---------+---------
@@ -140,13 +139,13 @@ There are three direct implementations for faceted classifications like this in
    6 | Comedy of Errors | comedy  |
    7 | Macbeth          | tragedy |
    8 | Hamlet           | tragedy |
-       ...                ....                                                
+       ...                ....
 
 For all of these representations, Repertoire Faceting works by constructing an inverted bitset index from the controlled vocabulary to your central model. Each bit represents a distinct model row (plays.id in this example). 1 indicates the play is in the category, and 0 that it is not:
 
     _plays_genre_facet
   ---------+-----------
-    genre  | signature 
+    genre  | signature
   ---------+-----------
    comedy  | 00001100
    history | 01110000
@@ -156,10 +155,10 @@ For all of these representations, Repertoire Faceting works by constructing an i
 From these bitset "signatures", Repertoire Faceting can easily count the number of member plays for each category, even in combination with other facets and a base query. For example, the bitset signature for all plays whose title contains the search word "Henry" is 0110000. Masking this (via bitwise "and") with each signature in the genre index above, we see that there are 2 histories that match the base search - Henry 4 parts 1 & 2 - a none in the other categories:
 
   ---------+------------------
-    genre  | signature & base 
+    genre  | signature & base
   ---------+------------------
-   comedy  | 00000000         
-   history | 01100000         
+   comedy  | 00000000
+   history | 01100000
    romance | 00000000
    tragedy | 00000000
 
@@ -173,14 +172,198 @@ References on faceted search:
 - http://en.wikipedia.org/wiki/Controlled_vocabulary
 
 
-== Known issues
+== A Quick Tour of API Levels
+
+The Repertoire Faceting module is intended to be a toolkit for building highly-scaleable faceted browsers over data held in relational databases. While it can be used as a black box (see the INSTALL document for a recipe), each API is also designed to be called directly. For example, you might write your own facet widgets in Javascript, using the JSON data feeds from the web service API level.
+
+The API layers are:
+
+  - Javascript widgets              [ see lib/assets/javascripts
+  - JSON web services               [ see Repertoire::Faceting::Controller, Repertoire::Faceting::Routing
+  - Rails model & finders           [ see Repertoire::Faceting::Model
+  - SQL queries and indexes         [ see ext/README.faceting
+
+To the relationships between the APIs clear, here is the same basic facet count query traced through each layer. While the module itself does not always issue exactly the queries listed here, the basic data model is the same. (To experiment with the APIs, run psql or the rails console from the Repertoire Faceting Example application. You may wish to use "SET search_path = public, facet" to bring the faceting schema's namespace into scope.)
+
+*** SQL API ***
+
+The most basic facet count query is a simple SQL aggregation.
+
+  =# SELECT discipline, COUNT(*) FROM nobelists GROUP BY discipline;
+         discipline      | count
+    ---------------------+-------
+     Chemistry           |    12
+     Peace               |     2
+     Economics           |    13
+     Medicine/Physiology |     9
+     Physics             |    27
+    (5 rows)
+
+Here is the facet value index for nobelist.discipline, as described in the prior section of the README.
+
+    =# SELECT * FROM facet.nobelists_discipline_index;
+         discipline      |                            signature
+    ---------------------+------------------------------------------------------------------
+     Physics             | 01011011001100110100100101000011010100001000001110110100110001
+     Chemistry           | 0000010000001100000000000000100000001011000101000100101
+     Economics           | 0010000001000000000101001000010010100000001000000000000000111001
+     Medicine/Physiology | 000000001000000010100010001100000000000001000000000000010000001
+     Peace               | 000000000000000000000000000000000000010000001
+    (5 rows)
+
+The faceting API's count() function returns the number of set bits in a signature. The same query, using a facet value index:
+
+    =# SELECT discipline, facet.count(signature) FROM facet.nobelists_discipline_index;
+         discipline      | count
+    ---------------------+-------
+     Physics             |    27
+     Chemistry           |    12
+     Economics           |    13
+     Medicine/Physiology |     9
+     Peace               |     2
+    (5 rows)
+
+One of the cardinal virtues of faceted search is that facet value counts show the "landscape" of data surrounding a base query. For example, here is a raw facet value count using "Robert" and the base query.
+
+    =# SELECT discipline, COUNT(*) FROM nobelists WHERE name LIKE 'Robert%' GROUP BY discipline;
+     discipline | count
+    ------------+-------
+     Chemistry  |     2
+     Physics    |     1
+     Economics  |     5
+    (3 rows)
+
+    (* Keep in mind a proper data model would use full-text index here.)
+
+To run facet value counts, we first gather a signature representing the base query, then use it as a mask over each entry in the facet value index. Here is a representative base query in raw SQL:
+
+    =# SELECT id, name, discipline, _packed_id FROM nobelists WHERE name LIKE 'Robert%';
+         id     |         name          | discipline | _packed_id
+    ------------+-----------------------+------------+------------
+       57839852 | Robert Burns Woodward | Chemistry  |         39
+      506489850 | Robert B. Laughlin    | Physics    |         40
+      920398821 | Robert M. Solow       | Economics  |         42
+       54824727 | Robert S. Mulliken    | Chemistry  |         43
+      309696094 | Robert J. Aumann      | Economics  |         58
+      249288376 | Robert C. Merton      | Economics  |         59
+      889316300 | Robert Engle          | Economics  |         60
+     1039451971 | Robert A. Mundell     | Economics  |         63
+    (8 rows)
+
+We can use the faceting API aggregator to read this result into a signature. (Note we use the serial id column, since the primary key is quite sparse.)
+
+    =# SELECT facet.signature(_packed_id) FROM nobelists WHERE name LIKE 'Robert%';
+                                signature
+    ------------------------------------------------------------------
+     0000000000000000000000000000000000000001101100000000000000111001
+    (1 row)
+
+Combining this base mask bitwise with each of the signatures in the facet value indices allows us to quickly calculate counts for very large datasets. (*For clarity we access the faceting namespace directly and use a subquery).
+
+    =# SET search_path TO public, facet;
+    =# SELECT discipline, facet.count(index.signature & base.signature) FROM
+          (SELECT facet.signature(_packed_id) FROM nobelists              WHERE name LIKE 'Robert%') AS base,
+          facet.nobelists_discipline_index                                                           AS index;
+
+
+         discipline      | count
+    ---------------------+-------
+     Physics             |     1
+     Chemistry           |     2
+     Economics           |     5
+     Medicine/Physiology |     0
+     Peace               |     0
+    (5 rows)
+
+If other facet values have been refined, they are also collected into signatures and used as masks.
+
+    =# SELECT discipline, facet.count(index.signature & base.signature & refine.signature) FROM
+          (SELECT facet.signature(_packed_id) FROM nobelists              WHERE name LIKE 'Robert%') AS base,
+          (SELECT signature                   FROM nobelists_degree_index WHERE degree = 'Ph.D.')    AS refine,
+          facet.nobelists_discipline_index                                                           AS index;
+
+In this fashion, facet count queries can be reduced to a single table scan over the model for the base query, plus an index table scan for each facet that has been refined.
+
+Each of the PostgreSQL API bindings implements these same operators, but over a different bitmap value type.
+
+*** ActiveRecord API ***
+
+  [ See Repertoire::Faceting::Model for full details ]
 
-- Running the unit tests issues warnings about a circular require. These can be ignored.
+The ActiveRecord API is built around the observation that our basic facet value count query is built-in to Rails:
 
+    > Nobelist.count(:discipline)
+    => {"Physics"=>27, "Economics"=>13, "Chemistry"=>12, "Medicine/Physiology"=>9, "Peace"=>2}
 
-== PostgreSQL C extensions
+When the Repertoire Faceting module is loaded and facets declared in the model, the same query will read the facet index instead. Execute the query in the console, and you will see the SQL generated reads the facet value index rather than the model table.
 
-Repertoire Faceting adds a native data type supporting bitwise operations and population count to PostgreSQL. For API details, see ext/signature.sql.IN.
+    > Nobelist.index_facets([:discipline, :degree])
+    > Nobelist.count(:discipline)
+    => {"Physics"=>27, "Economics"=>13, "Chemistry"=>12, "Medicine/Physiology"=>9, "Peace"=>2}
+
+Facets act just like Rails scoped queries, so you can use ActiveRecord's native syntax to specify a base query.
+
+    > Nobelist.where("name LIKE 'Robert%'").count(:discipline)
+    => {"Economics"=>5, "Chemistry"=>2, "Physics"=>1}
+
+Use refine() to specify facet value refinements on other attribtues.
+
+    > Nobelist.where("name LIKE 'Robert%'").refine(:degree => 'Ph.D.').count(:discipline)
+    => {"Economics"=>3, "Physics"=>1}
+
+You will see from the SQL query that the faceting module is detecting which model column to use as an id key, then reading the facet value indices wherever possible. The result is similar to the final SQL API example above.
+
+If you use refine() without count(), the module will use facet value indices to calculate the list of final results.
+
+    > Nobelist.where("name LIKE 'Robert%'").refine(:degree => 'Ph.D.')
+    => ...
+
+Note that because facets are assumed to be multi-valued, refine() is different from a normal ActiveRecord where() clause. In rails an equivalent query would be:
+
+    > Nobelist.where("name LIKE 'Robert%'").joins(:affiliations).where('affiliations.degree' => 'Ph.D.')
+    => ...
+
+When the number of rows in the model table is large and many facets come into play, using refine() can yield a performance gain over the straight query.
+
+*** Web services API ***
+
+The web services API exposes two JSON feeds, one that returns facet value counts given a set of refinements and another that returns the actual list of results. Your Rails controller constructs the base query, and the faceting webservice handles the surrounding facet refinements. For example, one of the faceting example application's controllers declares a query similar to this:
+
+class NobelistsController < ApplicationController
+  ...
+  def base
+    search = "#{params[:search]}%"
+    Nobelist.where(["name ILIKE ?", search])
+  end
+
+After including "faceting_for :nobelists" in the router, you can query the indexer by facet name, base query, and refinement filter:
+
+    $ curl -g "http://localhost:3000/nobelists/counts/discipline"
+    [["Physics",27],["Economics",13],["Chemistry",12],["Medicine/Physiology",9],["Peace",2]]
+
+    $ curl -g "http://localhost:3000/nobelists/counts/discipline?search=Robert"
+    [["Economics",5],["Chemistry",2],["Physics",1]]
+
+    $ curl -g "http://localhost:3000/nobelists/counts/discipline?filter[degree][]=Ph.D.&search=Robert"
+    [["Economics",3],["Physics",1]]
+
+Or you can issue a refinement query to get the results list:
+
+    $ curl -g "http://localhost:3000/nobelists/?filter[degree][]=Ph.D.&search=Robert"
+    => ...
+
+
+== Appendix. PostgreSQL in-database Faceting API
+
+Several bindings for the in-database faceting API are provided. In order of capability, they are:
+
+- signature        C language, requires superuser permissions
+- bytea            Javascript language, requires plv8 extension
+- varbit           No language or superuser requirements
+
+In general, if you have superuser permissions you should build and install the C-language (signature) API, as it is more scalable than the others, at no cost.
+
+All the Repertoire Faceting APIs add functionality for bitwise operations and population counts to PostgreSQL. For API details, see the ext directory.
 
 Signature: an auto-sizing bitset with the following functions
 
@@ -189,14 +372,12 @@ Signature: an auto-sizing bitset with the following functions
 - members(a)          => { set of integers corresponding to set bits }
 
 - sig_in, sig_out     => { mandatory I/O functions }
-- sig_and(a, b)    	=> a & b
-- sig_or(a, b)     	=> a | b
-- sig_xor(a)       	=> ~a
-- sig_length(a)	    => { number of bits in a }
-- sig_min(a)       	=> { lowest 1 in a, a.length }
+- sig_and(a, b)    	  => a & b
+- sig_or(a, b)     	  => a | b
+- sig_length(a)	      => { number of bits in a }
 - sig_get(a, i)       => { ith bit of a, or 0 }
 - sig_set(a, i, n)    => { sets ith bit of a to n }
-- sig_resize(a, n)    => { resizes a to hold n bits }
+- sig_resize(a, n)    => { resizes a to hold at least n bits }
 
 Bitwise signature operators:  &, |
 
@@ -206,12 +387,10 @@ Bitwise aggregates:
 - collect(signature)  => 'or' signature results together
 - filter(signature)   => 'and' signature results together
 
-Helper functions:
-
-- renumber_table(table, column, threshold)
+Helper aggregates:
 
-Check what percentage of signature bits constructed from the specified table and column would be wasted. If higher than the threshold, drop and re-add the column with a packed, contiguous integer id.
+- wastage(INT) -> REAL
 
-- signature_wastage(table, column)
+Aggregator that examines a table's primary key column, checking what proportion of signature bits constructed from the table would be wasted. If the proportion of wasted bits to valid bits is high, you should consider adding a new serial column.
 
-Returns a real number representing the percentage of bits that would be wasted in signatures constructed from the specified column.
+The Rails API introspects signature wastage before any facet indexing operation, and adds or removes a new serial column (called _packed_id) as necessary.

data/TODO

@@ -5,9 +5,9 @@ DESIRED FEATURES / IMPROVEMENTS.
      Adding support is a matter of defining the adapter and moving signature()
      calls into the postgresql adapter ]
 
--- modify widgets to multiplex ajax calls to work around 2 call limit 
+-- modify widgets to multiplex ajax calls to work around 2 call limit
    in many browsers
-   [ design: fetch() queues ajax calls; update() implementations request 
+   [ design: fetch() queues ajax calls; update() implementations request
      queue  and merge current webservice call with ones already in the queue.
      on controller side, receive multiple facet names, iterate, and bundle.
      fetch() then unbundles the results can dispatches them to appropriate
@@ -16,13 +16,26 @@ DESIRED FEATURES / IMPROVEMENTS.
 
 TODO
 
-- deploy to menzinga, document process
-- make sure works with postgresql-crontab
+-- gemcutter release of new faceting gem
+-- revise example app to bundle gem                             DONE
+--   redeploy to bytea / varbit targets                         DONE
+
+-- reference to the ActiveRecord API in the README              DONE
+-- revise the README to describe API layers & use               DONE
+
+-- clean up population in postgresql adapter?                   NO
+-- clean up throughout the facet definitions                    PARTIAL
+-- test harness
+   - test all extensions in order, skipping if cannot load      DONE
+-- simplify the in-database API
+   - use materialized views instead of dropping and adding tables   DONE
+   - move signature_wastage into an aggregate function          DONE
+   - remove the renumber function                               DONE
+   - the only remaining function should be expand_nesting       MOVED
+-- Makefile is verbose; use patterns                            NO
 - check formatting of all docs                                 DONE
-
 - make rake tasks smarter about detecting whether to run       NOT NECESSARY
 - make rake tasks automatically choose task dep on db type     NOT TO DO
-
 - make sure example app can be set up and deployed via rake    DONE
 
 - README
@@ -34,7 +47,7 @@ TODO
   * installing postgresql extensions                            DONE
   * migrations for indexing                                     DONE
   * updating indices (a) postgresql-crontab (b) rake crontab    DONE
-- generate routes that don't conflict with resource routes 
+- generate routes that don't conflict with resource routes
   (rails thinks /nobelists/results is the nobelist named 'results')	  NOT TO DO
 - FAQ
   * "not grouped error"

data/ext/Makefile

@@ -1,27 +1,37 @@
 #-------------------------------------------------------------------------
 #
 # Makefile--
-# Makefile for Repertoire bitset signature type
+# Makefile for Repertoire faceting API
 #
 # By default, this builds against an existing PostgreSQL installation
 # (the one identified by whichever pg_config is first in your path).
 #
 #-------------------------------------------------------------------------
- 
-MODULES = signature
-DATA_built = signature.sql uninstall_signature.sql
-# DOCS = README.signature                           # TODO. postgres 8.3 & 8.4 look for doc files in different places;
-                                                    #       temporarily commented out until we have migrated
 
-SHLIB_LINK = $(BE_DLLLIBS)
- 
+API_VERSION = 0.6.0
+
+MODULES = signature/signature
+EXTENSION = signature/faceting \
+            bytea/faceting_bytea \
+            varbit/faceting_varbit
+DATA_built = faceting--$(API_VERSION).sql \
+             faceting_bytea--$(API_VERSION).sql \
+             faceting_varbit--$(API_VERSION).sql
+DOCS = README.faceting
+
 PG_CONFIG = pg_config
 PGXS := $(shell $(PG_CONFIG) --pgxs)
-include $(PGXS)
 
-#-- Repertoire specific targets
-signature.sql:
-	cp signature.sql.IN signature.sql
+-include $(PGXS)
+# the dash above means loading pgxs may fail silently on Heroku, but
+# the API bindings will still be built:
+api_bindings : $(DATA_built)
+
+faceting--$(API_VERSION).sql: signature/signature.sql common/util.sql
+	cat signature/signature.sql common/util.sql > $@
+
+faceting_bytea--$(API_VERSION).sql: bytea/bytea.sql common/util.sql
+	cat bytea/bytea.sql common/util.sql > $@
 
-uninstall_signature.sql:
-	cp uninstall_signature.sql.IN uninstall_signature.sql
+faceting_varbit--$(API_VERSION).sql: varbit/varbit.sql common/util.sql
+	cat varbit/varbit.sql common/util.sql > $@

data/ext/README.faceting

@@ -0,0 +1,51 @@
+--
+--
+-- In-database support for Repertoire faceting module.
+--
+-- These libraries add scalable faceted indexing to the PostgreSQL database.
+--
+-- Basic approach is similar to other faceted browsers (Solr, Exhibit): an inverted bitmap index
+-- allows fast computation of facet value counts, given a base context and prior facet refinements.
+-- Bitsets can also be used to compute the result set of items.
+--
+-- There are three bindings for the API. The first extends PostgreSQL with a new bitset datatype
+-- written in C (called 'signature'). This version provides scaleable faceting up to 1,000,000 items
+-- and beyond, but requires control over the PostgreSQL server instance to build and load the C
+-- extensions.
+--
+-- The second is implemented using PostgreSQL's built-in VARBIT data type, and scales to a rough
+-- limit of about 30,000 items. It works in exactly the same way as the 'signature' data type above,
+-- but is about a factor of 5-10 slower. However, it does not require administrative control over
+-- the database server to install or use and so is suited to shared host deployment.
+--
+-- The third uses PostgreSQL's built-in BYTEA data type, processed via the Google Javascript
+-- language binding plv8 (https://code.google.com/p/plv8js/wiki/PLV8). Scalability and performance
+-- are unknown, but should be similar to the native C 'signature' type. However, the server needs
+-- to have the PLV8 language extension installed.
+--
+-- Only one binding of the API needs to be loaded at any time.  Each consists of:
+--
+--   (a) functions for accessing the bitset data types. These are used to store inverted indices from
+--   facet values to item ids. Functions are provided for doing refinements and counts on items
+--   with a given facet value.
+--
+--   (b) facilities for adding a packed (continuous) id sequence to the main item table.  Packed ids
+--   are used in the facet value indexes.
+--
+--   (c) utility functions for creating/updating packed ids and facet value index tables, e.g. in
+--   a crontab task.
+--
+-- The API bindings can each be built as a PostgreSQL extension, and then loaded and dropped using
+-- CREATE EXTENSION <faceting|faceting_bytea|faceting_varbit> and DROP EXTENSION ...
+--
+-- For hosts without administrative access, the individual sql files can be sourced directly.
+--
+-- Installation (in a Rails app)
+--
+--    $ cd repertoire-faceting
+--    $ rake db:faceting:extensions:install
+--
+-- Installation (PostgreSQL APIs only)
+--
+--    $ cd repertoire-faceting/ext
+--    $ make; sudo make install