intel 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b03f995035ea2892df23d4d49479c10d8e2e6fe5
-  data.tar.gz: 1770eb2f64a96b5751fbafcfce6c87aab2f31225
+  metadata.gz: a6afd8d515e1b008debd378f41f709f6a9e59b20
+  data.tar.gz: e514f5ffc47f3e45bdb1ab72b68117d50201e981
 SHA512:
-  metadata.gz: 8827a71c28cba3c68a2b76dea14573cd8f7420ee4239dc55f8cbc087943153315b2efe8be03b99c647250ed6677e8bcbb72119e0653561b1be92254f082ed9b2
-  data.tar.gz: 08955ef22b11378624daffc0fec7291cd4778881d2e84daf68b43c9cb6fc793db4eff29ca9461a59869e6536cedd2b5c27e6e39b0c81f19c7ae4843992272d3b
+  metadata.gz: ba504e4d707d5bdd3aa542443bb3769418bf4e13aa5d1ddef9cc855f1aeca7e99faea8034c66c12a7f0ba4dd42d57d4aa3696a890408d3d71048c4af3ca2a151
+  data.tar.gz: ba7dae63eaa56c1380733f31acde069c9e1a84be927af85dd1109abb44b358c7c1e2aeccbb2823d1c8d6062e0c63fd8dbea92ffdfbce5c5c73efb0231cffc34c

data/README.md

@@ -1,10 +1,16 @@
 # Intel
 
-Search analytics made easy
+:monkey_face: Search analytics made easy
 
-- track searches and conversions
-- watch searches in real-time [coming soon]
-- view searches with low conversions or no results [coming soon]
+[See it in action](http://intel-demo.herokuapp.com/)
+
+- view searches in real-time
+- track conversions week over week
+- monitor the performance of top searches
+
+:cupid: An amazing companion to [Searchkick](https://github.com/ankane/searchkick)
+
+Works with Rails 3.1+ and any search engine, including Elasticsearch, Sphinx, and Solr
 
 ## Get Started
 
@@ -14,43 +20,136 @@ Add this line to your application’s Gemfile:
 gem "intel"
 ```
 
-Create a table to keep track of searches.
+And run the generator. This creates a migration to store searches.
 
 ```sh
 rails generate intel:install
 rake db:migrate
 ```
 
-Use the `track` option to track searches.
+Next, add the dashboard to your `config/routes.rb`.
+
+```ruby
+mount Intel::Engine, at: "admin/intel"
+```
+
+Be sure to protect the endpoint in production - see the [Authentication](#authentication) section for ways to do this.
+
+### Track Searches
+
+Track searches by creating a record in the database.
+
+```ruby
+Intel::Search.create(
+  search_type: "Item", # typically the model name
+  query: "apple",
+  results_count: 12
+)
+```
+
+With [Searchkick](https://github.com/ankane/searchkick), you can use the `track` option to do this automatically.
 
 ```ruby
 Item.search "apple", track: true
 ```
 
-Want to track more attributes?  Just add them to the `searches` table with migrations and pass the values to the `track` option.
+If you want to track more attributes, add them to the `intel_searches` table.  Then, pass the values to the `track` option.
+
+```ruby
+Item.search "apple", track: {user_id: 1, source: "web"}
+```
+
+It’s that easy.
+
+### Track Conversions
+
+Tracking conversions is super important.
+
+First, define your conversion metric. This is specific to your application.
+
+Next, when a user searches, keep track of the search id.
+
+```ruby
+@items = Item.search "apple", track: true
+@items.search.id # returns search id
+```
+
+When a user converts, mark it.
 
 ```ruby
-Item.search "apple", track: {user_id: true, source: "web"}
+search = Intel::Search.find params[:search_id]
+search.converted_at = Time.now
+search.save
 ```
 
-It’s that easy!
+Better yet, record the result that converted.
 
-## View the Data [coming soon]
+```ruby
+item = Item.find params[:item_id]
+search.convertable = item
+search.save
+```
+
+The item will appear in the live stream. Add the `intel_name` method to your model to change what is displayed.
+
+```ruby
+class Item < ActiveRecord::Base
+  def intel_name
+    title # use the title method
+  end
+end
+```
+
+### Authentication
+
+Don’t forget to protect the dashboard in production.
+
+#### Basic Authentication
+
+Set the following variables in your environment or an initializer.
+
+```ruby
+ENV["INTEL_USERNAME"] = "andrew"
+ENV["INTEL_PASSWORD"] = "secret"
+```
 
-Add the dashboards to your `config/routes.rb`.
+#### Devise
 
 ```ruby
-mount Intel::Engine => "searches"
+authenticate :user, lambda{|user| user.admin? } do
+  mount Intel::Engine, at: "admin/intel"
+end
 ```
 
-Be sure to protect the endpoint in production.
+### Customize
 
-[show example]
+#### Time Zone
+
+To change the time zone, create an initializer `config/initializers/intel.rb` with:
+
+```ruby
+Intel.time_zone = "Pacific Time (US & Canada)" # defaults to Time.zone
+```
+
+#### Top Searches
+
+Change the number of top searches shown with:
+
+```ruby
+Intel.top_searches = 500 # defaults to 100
+```
+
+## TODO
+
+- customize views
+- group similar queries
+- track pagination, facets, sorting, etc
 
 ## Contributing
 
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+Everyone is encouraged to help improve this project. Here are a few ways you can help:
+
+- [Report bugs](https://github.com/ankane/intel/issues)
+- Fix bugs and [submit pull requests](https://github.com/ankane/intel/pulls)
+- Write, clarify, or fix documentation
+- Suggest or add new feature

data/app/controllers/intel/searches_controller.rb

@@ -0,0 +1,72 @@
+module Intel
+  class SearchesController < ActionController::Base
+    layout "intel/application"
+
+    http_basic_authenticate_with name: ENV["INTEL_USERNAME"], password: ENV["INTEL_PASSWORD"] if ENV["INTEL_PASSWORD"]
+
+    before_filter :set_time_zone
+    before_filter :set_search_types
+    before_filter :set_search_type, only: [:index, :overview]
+    before_filter :set_time_range, only: [:index, :overview]
+    before_filter :set_searches, only: [:index, :overview]
+
+    def index
+      if params[:sort] == "conversion_rate"
+        @searches.sort_by!{|s| [s["conversion_rate"].to_f, s["query"]] }
+      end
+    end
+
+    def overview
+      relation = Intel::Search.where(search_type: params[:search_type])
+      @searches_by_week = relation.group_by_week(:created_at, Time.zone, @time_range).count
+      @conversions_by_week = relation.where("converted_at is not null").group_by_week(:created_at, Time.zone, @time_range).count
+      @top_searches = @searches.first(5)
+      @bad_conversion_rate = @searches.sort_by{|s| [s["conversion_rate"].to_f, s["query"]] }.first(5).select{|s| s["conversion_rate"] < 50 }
+      @conversion_rate_by_week = {}
+      @searches_by_week.each do |week, searches_count|
+        @conversion_rate_by_week[week] = searches_count > 0 ? (100.0 * @conversions_by_week[week] / searches_count).round : 0
+      end
+    end
+
+    def stream
+    end
+
+    # suspiciously similar to bootstrap 3
+    COLORS = %w[5bc0de d9534f 5cb85c f0ad4e]
+
+    def recent
+      @searches = Intel::Search.order("created_at desc").limit(10)
+      @color = {}
+      @search_types.each_with_index do |search_type, i|
+        @color[search_type] = COLORS[i % COLORS.size]
+      end
+      render layout: false
+    end
+
+    protected
+
+    def set_search_types
+      @search_types = Intel::Search.uniq.pluck(:search_type).sort
+    end
+
+    def set_search_type
+      @search_type = params[:search_type].to_s
+    end
+
+    def set_time_zone
+      @time_zone = Intel.time_zone
+    end
+
+    def set_time_range
+      @time_range = 8.weeks.ago.in_time_zone(@time_zone).beginning_of_week(:sunday)..Time.now
+    end
+
+    def set_searches
+      @limit = params[:limit] || Intel.top_searches
+      @searches = Intel::Search.connection.select_all(Intel::Search.select("normalized_query, COUNT(*) as searches_count, COUNT(converted_at) as conversions_count, AVG(results_count) as avg_results_count").where(created_at: @time_range, search_type: @search_type).group("normalized_query").order("searches_count desc, normalized_query asc").limit(@limit).to_sql).to_a
+      @searches.each do |search|
+        search["conversion_rate"] = 100 * search["conversions_count"].to_i / search["searches_count"].to_f
+      end
+    end
+  end
+end

data/app/views/intel/searches/index.html.erb

@@ -0,0 +1,27 @@
+<h1>
+  <%= @search_type %> Searches
+  <small style="color: #999; font-weight: normal;">Top <%= @limit %></small>
+</h1>
+
+<table>
+  <thead>
+    <tr>
+      <th>Query</th>
+      <th class="num" style="width: 20%;"><%= link_to "Searches", params.except(:sort), class: ("active" if params[:sort] != "conversion_rate") %></th>
+      <th class="num" style="width: 20%;">Conversions</th>
+      <th class="num" style="width: 20%;"><%= link_to "%", params.merge(sort: "conversion_rate"), class: ("active" if params[:sort] == "conversion_rate") %></th>
+      <th class="num" style="width: 20%;">Avg Results</th>
+    </tr>
+  </thead>
+  <tbody>
+    <% @searches.each do |search| %>
+      <tr>
+        <td><%= search["normalized_query"] %></td>
+        <td class="num"><%= search["searches_count"] %></td>
+        <td class="num"><%= search["conversions_count"] %></td>
+        <td class="num"><%= number_to_percentage search["conversion_rate"].to_f, precision: 0 %></td>
+        <td class="num"><%= search["avg_results_count"].to_f.round %></td>
+      </tr>
+    <% end %>
+  </tbody>
+</table>

data/app/views/intel/searches/overview.html.erb

@@ -0,0 +1,48 @@
+<h1><%= @search_type %> Overview</h1>
+
+<div class="grid">
+  <div class="col-1-2">
+    <table>
+      <thead>
+        <tr>
+          <th>Top Searches</th>
+          <th class="num"><%= link_to "View all", intel.searches_path(search_type: params[:search_type]) %></th>
+        </tr>
+      </thead>
+      <tbody>
+        <% @top_searches.each do |row| %>
+          <tr>
+            <td><%= row["normalized_query"] %></td>
+            <td class="num"><%= row["searches_count"] %></td>
+          </tr>
+        <% end %>
+      </tbody>
+    </table>
+  </div>
+  <div class="col-1-2">
+    <table>
+      <thead>
+        <tr>
+          <th>Low Conversions</th>
+          <th class="num"><%= link_to "View all", intel.searches_path(search_type: params[:search_type], sort: "conversion_rate") %></th>
+        </tr>
+      </thead>
+      <tbody>
+        <% @bad_conversion_rate.each do |row| %>
+          <tr>
+            <td><%= row["normalized_query"] %></td>
+            <td class="num"><%= number_to_percentage row["conversion_rate"], precision: 0 %></td>
+          </tr>
+        <% end %>
+      </tbody>
+    </table>
+  </div>
+</div>
+
+<h2>Conversion Rate</h2>
+
+<%= line_chart @conversion_rate_by_week %>
+
+<h2>Volume</h2>
+
+<%= line_chart [{name: "Searches", data: @searches_by_week}, {name: "Conversions", data: @conversions_by_week}] %>

data/app/views/intel/searches/recent.html.erb

@@ -0,0 +1,20 @@
+<% @searches.each do |search| %>
+  <tr>
+    <td style="width: 15%;">
+      <%= link_to search.search_type, intel.overview_searches_path(search_type: search.search_type), class: "type-link", style: "background-color: ##{@color[search.search_type]};" %>
+    </td>
+    <td><%= search.query %></td>
+    <td style="width: 35%;">
+      <% if search.converted_at %>
+        <span style="color: #5cb85c;">
+          <span style="font-weight: bold;">✓</span>
+          <%= search.convertable ? (search.convertable.respond_to?(:intel_name) ? search.convertable.intel_name : "#{search.convertable_type} #{search.convertable_id}") : "Converted" %>
+        </span>
+      <% end %>
+    </td>
+    <td style="width: 20%;" class="num">
+      <%= time_ago_in_words search.created_at %> ago
+      <div style="color: #999;"><%= pluralize search.results_count, "result" %></div>
+    </td>
+  </tr>
+<% end %>

data/app/views/intel/searches/stream.html.erb

@@ -0,0 +1,13 @@
+<h1>Live Stream</h1>
+
+<table id="stream"></table>
+
+<script>
+  function fetchRecentSearches() {
+    $("#stream").load("<%= intel.searches_recent_path %>");
+    setTimeout(fetchRecentSearches, 5 * 1000);
+  }
+  $( function() {
+    fetchRecentSearches();
+  });
+</script>

data/app/views/layouts/intel/application.html.erb

@@ -0,0 +1,291 @@
+<!DOCTYPE html>
+<html>
+  <head>
+    <title>Intel</title>
+
+    <meta charset="utf-8" />
+
+    <style>
+      body {
+        font-family: "Helvetica Neue", Arial, Helvetica, sans-serif;
+        margin: 0;
+        padding: 20px;
+        font-size: 14px;
+        line-height: 1.4;
+        color: #333;
+      }
+
+      a, a:visited, a:active {
+        color: #428bca;
+        text-decoration: none;
+      }
+
+      a:hover {
+        text-decoration: underline;
+      }
+
+      table {
+        width: 100%;
+        border-collapse: collapse;
+        border-spacing: 0;
+        margin-bottom: 20px;
+      }
+
+      th {
+        text-align: left;
+        border-bottom: solid 2px #ddd;
+      }
+
+      h1 {
+        font-size: 20px;
+      }
+
+      h2 {
+        font-size: 16px;
+      }
+
+      ul {
+        list-style-type: none;
+        padding: 0;
+      }
+
+      table td, table th {
+        padding: 8px;
+      }
+
+      td {
+        border-top: solid 1px #ddd;
+      }
+
+      #brand {
+        font-size: 18px;
+        line-height: 20px;
+        font-weight: bold;
+        color: #999;
+      }
+
+      a.active {
+        color: #5cb85c;
+      }
+
+      a.type-link {
+         color: #fff;
+         padding: 8px;
+         border-radius: 4px;
+      }
+
+      a.type-link:hover {
+        text-decoration: none;
+      }
+
+      #header {
+        border-bottom: solid 1px #ddd;
+        padding-bottom: 20px;
+      }
+
+      .nav, .nav li {
+        display: inline;
+      }
+
+      .nav li {
+        margin-right: 20px;
+      }
+
+      #stream {
+        border-bottom: solid 1px #ddd;
+      }
+
+      .num {
+        text-align: right;
+      }
+
+      .container {
+        max-width: 800px;
+        margin-left: auto;
+        margin-right: auto;
+      }
+
+      /*
+        Simple Grid
+        Learn More - http://dallasbass.com/simple-grid-a-lightweight-responsive-css-grid/
+        Project Page - http://thisisdallas.github.com/Simple-Grid/
+        Author - Dallas Bass
+        Site - dallasbass.com
+      */
+
+      *, *:after, *:before {
+        -webkit-box-sizing: border-box;
+        -moz-box-sizing: border-box;
+        box-sizing: border-box;
+      }
+
+      body {
+        margin: 0px;
+      }
+
+      [class*='col-'] {
+        float: left;
+        padding-right: 20px;
+      }
+
+      [class*='col-']:last-of-type {
+        padding-right: 0px;
+      }
+
+      .grid {
+        width: 100%;
+
+        margin: 0 auto;
+        overflow: hidden;
+      }
+
+      .grid:after {
+        content: "";
+        display: table;
+        clear: both;
+      }
+
+      .grid-pad {
+        padding: 20px 0 0px 20px;
+      }
+
+      .grid-pad > [class*='col-']:last-of-type {
+        padding-right: 20px;
+      }
+
+      .push-right {
+        float: right;
+      }
+
+      /* Content Columns */
+
+      .col-1-1 {
+        width: 100%;
+      }
+      .col-2-3, .col-8-12 {
+        width: 66.66%;
+      }
+
+      .col-1-2, .col-6-12 {
+        width: 50%;
+      }
+
+      .col-1-3, .col-4-12 {
+        width: 33.33%;
+      }
+
+      .col-1-4, .col-3-12 {
+        width: 25%;
+      }
+
+      .col-1-5 {
+        width: 20%;
+      }
+
+      .col-1-6, .col-2-12 {
+        width: 16.667%;
+      }
+
+      .col-1-7 {
+        width: 14.28%;
+      }
+
+      .col-1-8 {
+        width: 12.5%;
+      }
+
+      .col-1-9 {
+        width: 11.1%;
+      }
+
+      .col-1-10 {
+        width: 10%;
+      }
+
+      .col-1-11 {
+        width: 9.09%;
+      }
+
+      .col-1-12 {
+        width: 8.33%
+      }
+
+      /* Layout Columns */
+
+      .col-11-12 {
+        width: 91.66%
+      }
+
+      .col-10-12 {
+        width: 83.333%;
+      }
+
+      .col-9-12 {
+        width: 75%;
+      }
+
+      .col-5-12 {
+        width: 41.66%;
+      }
+
+      .col-7-12 {
+        width: 58.33%
+      }
+
+      @media handheld, only screen and (max-width: 767px) {
+
+
+        .grid {
+          width: 100%;
+          min-width: 0;
+          margin-left: 0px;
+          margin-right: 0px;
+          padding-left: 0px;
+          padding-right: 0px;
+        }
+
+        [class*='col-'] {
+          width: auto;
+          float: none;
+          margin-left: 0px;
+          margin-right: 0px;
+          margin-top: 10px;
+          margin-bottom: 10px;
+          padding-right: 0px;
+          padding-left: 0px;
+        }
+      }
+    </style>
+
+    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
+    <%= javascript_include_tag "//www.google.com/jsapi", "chartkick" %>
+  </head>
+  <body>
+    <div class="container">
+      <% if !@skip_header %>
+        <div id="header">
+          <div class="grid">
+            <div class="col-1-2">
+              <ul class="nav">
+                <li id="brand">Intel</li>
+                <li><%= link_to "Live Stream", intel.root_path, class: ("active" if !@search_type) %></li>
+                <% @search_types.each do |search_type| %>
+                  <li><%= link_to search_type, intel.overview_searches_path(search_type: search_type), class: ("active" if @search_type == search_type) %></li>
+                <% end %>
+              </ul>
+            </div>
+
+            <div class="col-1-2" style="text-align: right;">
+              <% if %w[overview index].include?(params[:action]) and @time_range %>
+                <%= @time_range.first.strftime("%b %-e, %Y") %> to <%= @time_range.last.strftime("%b %-e, %Y") %>
+                <span style="color: #999;"><%= @time_zone.name.sub(" (US & Canada)", "") %></span>
+              <% end %>
+            </div>
+          </div>
+        </div>
+      <% end %>
+
+      <%= yield %>
+    </div>
+  </body>
+</html>

data/config/routes.rb

@@ -0,0 +1,7 @@
+Intel::Engine.routes.draw do
+  resources :searches, only: [:index] do
+    get "overview", on: :collection
+  end
+  get "searches/recent"
+  root to: "searches#stream"
+end

data/intel.gemspec

@@ -18,7 +18,8 @@ Gem::Specification.new do |spec|
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
-  spec.add_dependency "searchkick"
+  spec.add_dependency "chartkick"
+  spec.add_dependency "groupdate"
 
   spec.add_development_dependency "bundler", "~> 1.3"
   spec.add_development_dependency "rake"

data/lib/generators/intel/templates/install.rb

@@ -1,10 +1,19 @@
 class <%= migration_class_name %> < ActiveRecord::Migration
   def change
-    create_table :searches do |t|
-      t.string :searchable_type
+    create_table :intel_searches do |t|
+      t.string :search_type
       t.string :query
+      t.string :normalized_query
       t.integer :results_count
       t.timestamp :created_at
+      t.integer :convertable_id
+      t.string :convertable_type
+      t.timestamp :converted_at
     end
+
+    add_index :intel_searches, [:created_at]
+    add_index :intel_searches, [:search_type, :created_at]
+    add_index :intel_searches, [:search_type, :normalized_query, :created_at], name: "index_intel_searches_on_search_type_and_normalized_query_and_cr" # autogenerated name is too long
+    add_index :intel_searches, [:convertable_id, :convertable_type]
   end
 end

data/lib/intel.rb

@@ -1,14 +1,34 @@
-require "searchkick"
 require "intel/search"
 require "intel/track"
 require "intel/engine"
 require "intel/version"
 
-module Searchkick
-  module Search
-    include Intel::Track
+require "chartkick"
+require "groupdate"
 
-    alias_method :search_without_track, :search
-    alias_method :search, :search_with_track
+module Intel
+  # time zone
+  mattr_reader :time_zone
+  def self.time_zone=(time_zone)
+    @@time_zone = time_zone.is_a?(String) ? ActiveSupport::TimeZone.new(time_zone) : time_zone
+  end
+
+  # top searches
+  mattr_accessor :top_searches
+  self.top_searches = 100
+end
+
+if defined?(Searchkick)
+  module Searchkick
+    module Search
+      include Intel::Track
+
+      alias_method :search_without_track, :search
+      alias_method :search, :search_with_track
+    end
+
+    class Results
+      attr_accessor :search
+    end
   end
 end

data/lib/intel/engine.rb

@@ -1,4 +1,5 @@
 module Intel
   class Engine < ::Rails::Engine
+    isolate_namespace Intel
   end
 end

data/lib/intel/search.rb

@@ -1,4 +1,18 @@
 module Intel
   class Search < ActiveRecord::Base
+    belongs_to :convertable, polymorphic: true
+
+    before_save :set_normalized_query
+
+    def converted?
+      converted_at.present?
+    end
+
+    protected
+
+    def set_normalized_query
+      self.normalized_query = query.downcase if query
+    end
+
   end
 end

data/lib/intel/track.rb

@@ -6,7 +6,7 @@ module Intel
 
       if options[:track]
         attributes = options[:track] == true ? {} : options[:track]
-        Intel::Search.create({searchable_type: self.name, query: term, results_count: results.total_count}.merge(attributes))
+        results.search = Intel::Search.create({search_type: self.name, query: term, results_count: results.total_count}.merge(attributes))
       end
 
       results

data/lib/intel/version.rb

@@ -1,3 +1,3 @@
 module Intel
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

metadata

@@ -1,17 +1,31 @@
 --- !ruby/object:Gem::Specification
 name: intel
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-10-24 00:00:00.000000000 Z
+date: 2013-10-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: searchkick
+  name: chartkick
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: groupdate
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -64,6 +78,13 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
+- app/controllers/intel/searches_controller.rb
+- app/views/intel/searches/index.html.erb
+- app/views/intel/searches/overview.html.erb
+- app/views/intel/searches/recent.html.erb
+- app/views/intel/searches/stream.html.erb
+- app/views/layouts/intel/application.html.erb
+- config/routes.rb
 - intel.gemspec
 - lib/generators/intel/install_generator.rb
 - lib/generators/intel/templates/install.rb