redis_analytics 0.1.0

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in redis_analytics.gemspec
+gemspec

data/Gemfile.lock

@@ -0,0 +1,57 @@
+PATH
+  remote: .
+  specs:
+    redis_analytics (0.1.0)
+      activesupport (>= 3.2.0)
+      browser (>= 0.1.6)
+      geoip (>= 1.2.1)
+      json (>= 1.7.7)
+      rack (>= 1.4.0)
+      redis (>= 3.0.2)
+      sinatra (>= 1.3.3)
+
+GEM
+  remote: http://rubygems.org/
+  specs:
+    activesupport (3.2.13)
+      i18n (= 0.6.1)
+      multi_json (~> 1.0)
+    browser (0.1.6)
+    diff-lcs (1.1.3)
+    geoip (1.2.1)
+    i18n (0.6.1)
+    json (1.7.7)
+    metaclass (0.0.1)
+    mocha (0.12.7)
+      metaclass (~> 0.0.1)
+    multi_json (1.7.1)
+    rack (1.5.2)
+    rack-protection (1.5.0)
+      rack
+    rack-test (0.6.2)
+      rack (>= 1.0)
+    rake (10.0.3)
+    redis (3.0.2)
+    rspec (2.11.0)
+      rspec-core (~> 2.11.0)
+      rspec-expectations (~> 2.11.0)
+      rspec-mocks (~> 2.11.0)
+    rspec-core (2.11.1)
+    rspec-expectations (2.11.3)
+      diff-lcs (~> 1.1.3)
+    rspec-mocks (2.11.3)
+    sinatra (1.3.3)
+      rack (~> 1.3, >= 1.3.6)
+      rack-protection (~> 1.2)
+      tilt (~> 1.3, >= 1.3.3)
+    tilt (1.3.6)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  mocha (>= 0.12.7)
+  rack-test (>= 0.6.2)
+  rake (>= 10.0.3)
+  redis_analytics!
+  rspec (>= 2.11.0)

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2013 Schubert Cardozo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,98 @@
+# redis_analytics  [![Build Status](https://travis-ci.org/saturnine/redis_analytics.png?branch=master)](https://travis-ci.org/saturnine/redis_analytics)
+
+## What is redis_analytics?
+
+A gem that provides a Redis based web analytics solution for your rack-compliant apps
+
+## Why should I use it?
+
+It gives you detailed analytics about visitors, unique visitors, browsers, OS, visitor recency, traffic sources and more
+
+## Does it have a cool dashboard?
+
+Yes, It uses the excellent [Morris.js](http://www.oesmith.co.uk/morris.js/) for the main dashboard and [Highcharts](http://www.highcharts.com) for drawing the various detailed graphs
+
+![Screenshot](https://github.com/saturnine/redis_analytics/raw/master/screenshot.png)
+
+## OK, so how do I install it?
+
+`gem install redis_analytics`
+
+or in your `Gemfile`
+
+```ruby
+gem 'redis_analytics'
+```
+
+Make sure your redis server is running! Redis configuration is outside the scope of this README, but
+check out the [Redis documentation](http://redis.io/documentation).
+
+## How do I enable tracking in my rack-compliant app?
+
+### Step 1: Load the redis_analytics library and configure it
+
+```ruby
+# this is not required unless you use :require => false in your Gemfile
+require 'redis_analytics'
+
+# configure your redis connection (this is mandatory) and namespace (this is optional)
+Rack::RedisAnalytics.configure do |configuration|
+  configuration.redis_connection = Redis.new(:host => 'localhost', :port => '6379')
+  configuration.redis_namespace = 'ra'
+end
+```
+### Step 2: Use the Tracker rack middleware (NOT REQUIRED FOR RAILS)
+
+```ruby
+# in Sinatra you would do...
+use Rack::RedisAnalytics::Tracker
+```
+
+For rails the middleware is added automatically, so you do not need to add it manually using `config.middleware.use`
+
+## Where do I view the dashboard?
+
+### Option 1: Set a dashboard endpoint in your configuration
+
+```ruby
+Rack::RedisAnalytics.configure do |configuration|
+  configuration.dashboard_endpoint = '/dashboard'
+end
+```
+
+and navigate to [http://localhost:3000/dashboard](http://localhost:3000/dashboard) assuming your rack-compliant app is hosted at [http://localhost:3000](http://localhost:3000)
+
+### Option 2: Simply run the bundled Sinatra application binary
+
+`redis_analytics_dashboard --redis-host 127.0.0.1 --redis-port 6379 --redis-namespace ra`
+
+and navigate to [http://localhost:4567](http://localhost:4567)
+
+## What if I have multiple rails apps that I want to track as one?
+
+In the configuration, keep the value of redis_namespace the same across all your rails apps
+
+```ruby
+Rack::RedisAnalytics.configure do |configuration|
+  configuration.redis_connection = Redis.new(:host => 'localhost', :port => '6379')
+  configuration.redis_namespace = 'mywebsite.org'
+end
+```
+
+## Why is the Geolocation tracking giving me wrong results?
+
+IP based Geolocation works using [MaxMind's](http://www.maxmind.com) GeoLite database. The free version is not as accurate as their commercial version. 
+Also it is recommended to regularly get an updated binary of 'GeoLite Country' database from [here](http://dev.maxmind.com/geoip/geolite) and extract the GeoIP.dat file into a local directory.
+You will then need to point to the GeoIP.dat file in your configuration.
+
+```ruby
+Rack::RedisAnalytics.configure do |configuration|
+  configuration.redis_connection = Redis.new(:host => 'localhost', :port => '6379')
+  configuration.redis_namespace = 'mywebsite.org'
+  configuration.geo_ip_data_path = '/path/to/GeoIP.dat'
+end
+```
+
+## Copyright
+
+Copyright (c) 2012-2013 Schubert Cardozo. See LICENSE for further details.

data/Rakefile

@@ -0,0 +1,19 @@
+# require 'bundler/gem_tasks'
+require 'rake'
+require 'rspec/core'
+require 'rspec/core/rake_task'
+
+# encoding: UTF-8
+require 'rubygems'
+begin
+  require 'bundler/setup'
+rescue LoadError
+  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
+end
+
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rspec_opts = ['--backtrace']
+end
+
+task :default => :spec

data/bin/redis_analytics_dashboard

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+path = File.expand_path '../../lib', __FILE__
+$:.unshift(path) if File.directory?(path) && !$:.include?(path)
+require 'redis_analytics/dashboard'
+args = Hash[*ARGV] rescue {}
+if args.key?('--redis-host') and args.key?('--redis-port')
+  Rack::RedisAnalytics.configure do |c|
+    c.redis_connection = Redis.new(:host => args['--redis-host'], :port => args['--redis-port'], :db => args['--redis-db'])
+    c.redis_namespace = args['--redis-namespace'] if args['--redis-namespace']
+  end
+  Rack::RedisAnalytics::Dashboard.set(:port, args['-p']) if args['-p']
+  Rack::RedisAnalytics::Dashboard.run!
+else
+  puts "Usage: redis_analytics_dashboard --redis-host 127.0.0.1 --redis-port 6379 --redis-db 0 --redis-namespace test_namespace"
+end

data/lib/redis_analytics.rb

@@ -0,0 +1,22 @@
+require 'rack'
+require 'redis'
+require 'browser'
+require 'sinatra'
+require 'geoip' 
+
+require 'redis_analytics'
+require 'redis_analytics/version'
+require 'redis_analytics/configuration'
+require 'redis_analytics/analytics'
+require 'redis_analytics/time_ext'
+require 'redis_analytics/helpers'
+require 'redis_analytics/dashboard'
+
+require 'redis_analytics/tracker'
+require 'redis_analytics/railtie' if defined? Rails::Railtie
+
+module Rack
+  module RedisAnalytics
+    extend Configuration
+  end
+end

data/lib/redis_analytics/analytics.rb

@@ -0,0 +1,231 @@
+# -*- coding: utf-8 -*-
+require 'digest/md5'
+module Rack
+  module RedisAnalytics
+    
+    class Analytics
+      
+      PAGEVIEWS = [['/health', 'o', 'd', 't'], ['/track', 'o', 'd', 't']]
+
+      TRANSACTIONS = {
+        "flights_search" => ["/flights/search", :o, :d],
+        "flights_results" => ["/flights/results", :o, :d],
+        "flights_book_step_1" => ["/flights/itinerary/:itinerary_id/info"],
+        "flights_book_step_2" => ["/flights/itinerary/:itinerary_id/traveller"],
+        "flights_book_step_3" => ["/flights/itinerary/:itinerary_id/pay"],
+        "flights_book_done" => ["/flights/itinerary/:itinerary_id/confirmation"],
+        "products" => ["/products/:id/review", :user_id]
+      }
+
+      REFERRERS = ['google', 'bing', 'yahoo', 'cleartrip', 'github']
+
+      def initialize(app)
+        @app = app
+        @redis_key_prefix = "#{RedisAnalytics.redis_namespace}:"
+      end
+      
+      # def call(env)
+      #   dup.call!(env)
+      # end
+
+      def call(env)
+        @env = env
+        @request  = Request.new(env)
+        status, headers, body = @app.call(env)
+        @response = Rack::Response.new(body, status, headers)
+        record if @response.ok? and @response.content_type =~ /^text\/html/
+        @response.finish
+      end
+
+      def parse_path(path)
+        keys = []
+        spc = %w{. + ( )}
+        pattern =
+          path.to_str.gsub(/((:\w+)|[\*#{spc.join}])/) do |match|
+          case match
+          # when "*"
+          #   keys << 'splat'
+          #   "(.*?)"
+          when *spc
+            Regexp.escape(match)
+          else
+            keys << $2[1..-1]
+            "([^/?&#]+)"
+          end
+        end
+        [/^#{pattern}$/, keys]
+      end
+
+      def for_each_time_range(t)
+        RedisAnalytics.redis_key_timestamps.map{|x, y| [t.strftime(x), y]}.each do |ts, expire|
+          yield(ts, expire) # returns an array of the redis methods to call 
+          # r.each do |method, args|
+          #   RedisAnalytics.redis_connection.send(method, *args)
+          #   RedisAnalytics.redis_connection.expire(args[1]) if expire # assuming args[1] is always the key that is being operated on.. will this always work?
+          # end
+        end
+      end
+
+      def record
+        t = Time.now 
+
+        # record pageviews
+        path = @request.path
+        params = @request.params
+        if i = PAGEVIEWS.index{|x| x[0] == path}
+          page = PAGEVIEWS[i]
+          params.select{|x, y| page[1..-1].include?(x)}.each do |k, v|
+            for_each_time_range(t) do |ts, expire|
+              h = Digest::MD5.hexdigest(v)
+              RedisAnalytics.redis_connection.hset("#{@redis_key_prefix}page", h, v)
+              RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}page", expire) if expire
+
+              RedisAnalytics.redis_connection.incr("#{@redis_key_prefix}page_#{i}_#{page.index(k)}_#{h}:#{ts}")
+              RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}page_#{i}_#{page.index(k)}_#{h}:#{ts}", expire) if expire
+            end
+          end
+        end
+        
+        returning_visitor = @request.cookies[RedisAnalytics.returning_user_cookie_name]
+        recent_visitor = @request.cookies[RedisAnalytics.visit_cookie_name]
+
+        vcn_seq, rucn_seq = nil
+        visit_start_time =visit_end_time = first_visit_time = t.to_i
+
+        if not returning_visitor and not recent_visitor
+          rucn_seq = RedisAnalytics.redis_connection.incr("#{@redis_key_prefix}unique_visits")
+          vcn_seq = RedisAnalytics.redis_connection.incr("#{@redis_key_prefix}visits")
+          visit(t, :rucn_seq => rucn_seq)
+          new_visit(t)
+          visit_time(t, t.to_i)
+          page_view(t)
+        elsif returning_visitor and not recent_visitor
+          vcn_seq = RedisAnalytics.redis_connection.incr("#{@redis_key_prefix}visits")
+          rucn_seq, first_visit_time, last_visit_time = returning_visitor.split('.')
+          visit(t, :last_visit_time => last_visit_time.to_i, :rucn_seq => rucn_seq)
+          page_view(t)
+        elsif returning_visitor and recent_visitor
+          rucn_seq, vcn_seq, visit_start_time, visit_end_time = recent_visitor.split('.')
+          rucn_seq, first_visit_time = returning_visitor.split('.')
+          visit_time(t, visit_end_time.to_i)
+          page_view(t, visit_start_time.to_i == visit_end_time.to_i)
+        elsif not returning_visitor and recent_visitor
+          rucn_seq, vcn_seq, visit_start_time, visit_end_time = recent_visitor.split('.')
+          visit_time(t, visit_end_time.to_i)
+          page_view(t, visit_start_time.to_i == visit_end_time.to_i)
+        end
+
+        # create the recent visit cookie
+        @response.set_cookie(RedisAnalytics.visit_cookie_name, {:value => "#{rucn_seq}.#{vcn_seq}.#{visit_start_time}.#{t.to_i}", :expires => t + (RedisAnalytics.visit_timeout.to_i * 60 )})
+
+        # create the permanent cookie (2 years)
+        @response.set_cookie(RedisAnalytics.returning_user_cookie_name, {:value => "#{rucn_seq}.#{first_visit_time}.#{t.to_i}", :expires => t + (60 * 60 * 24 * 5)}) # 5 hours temp
+
+        puts "TIME = [#{t}]"
+        puts "VISIT = #{vcn_seq}"
+        puts "UNIQUE VISIT = #{rucn_seq}"
+      end
+      
+      def new_visit(t)
+        for_each_time_range(t) do |ts, expire|
+          RedisAnalytics.redis_connection.incr("#{@redis_key_prefix}new_visits:#{ts}")
+          RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}new_visits:#{ts}",expire) if expire
+        end
+      end
+
+      def visit(t, options = {})
+        last_visit_time = options[:last_visit_time] || nil
+        rucn_seq = options[:rucn_seq] || nil
+        geo_country_code = nil
+        referrer = nil
+        ua = nil
+        
+        # Geo IP Country code fetch
+        if defined?(GeoIP)
+          begin
+            g = GeoIP.new(RedisAnalytics.geo_ip_data_path)
+            geo_country_code = g.country(@request.ip).to_hash[:country_code2]
+            puts "Tracking IP #{@request.ip} using #{g.inspect} => GOT #{geo_country_code}"
+          rescue Exception => e
+            puts "Warning: Unable to fetch country info #{e}"
+          end
+        end
+
+        # Referrer regex decode
+        if @request.referrer
+          REFERRERS.each do |referrer|
+            # this will track x.google.mysite.com as google so its buggy, fix the regex
+            if m = @request.referrer.match(/^(https?:\/\/)?([a-zA-Z0-9\.\-]+\.)?(#{referrer})\.([a-zA-Z\.]+)(:[0-9]+)?(\/.*)?$/)
+              "REFERRER => #{m.to_a[3]}"
+              referrer = m.to_a[3]
+            else
+              referrer = 'other'
+            end
+          end
+        else
+          referrer = 'organic'
+        end
+
+        # User agent
+        ua = Browser.new(:ua => @request.user_agent, :accept_language => 'en-us')
+        
+        for_each_time_range(t) do |ts, expire|
+
+          # increment the total visits
+          RedisAnalytics.redis_connection.incr("#{@redis_key_prefix}visits:#{ts}")
+          RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}visits:#{ts}", expire) if expire
+
+          if rucn_seq
+            unique = RedisAnalytics.redis_connection.sadd("#{@redis_key_prefix}unique_visits:#{ts}", rucn_seq)
+            RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}unique_visits:#{ts}", expire) if expire
+            
+            # geo ip tracking
+            if geo_country_code and geo_country_code =~ /^[A-Z]{2}$/
+              RedisAnalytics.redis_connection.zincrby("#{@redis_key_prefix}ratio_country:#{ts}", 1, geo_country_code)
+              RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}ratio_country:#{ts}", expire) if expire
+            end
+            
+            # referrer tracking 
+            RedisAnalytics.redis_connection.zincrby("#{@redis_key_prefix}ratio_referrers:#{ts}", 1, referrer)
+          end
+          
+          # tracking for visitor recency
+          if last_visit_time
+            days_since_last_visit = ((t.to_i - last_visit_time)/(24*3600)).round
+            RedisAnalytics.redis_connection.zincrby("#{@redis_key_prefix}ratio_recency:#{ts}", 1, days_since_last_visit)
+            RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}ratio_recency:#{ts}", expire) if expire
+          end
+          
+          RedisAnalytics.redis_connection.zincrby("#{@redis_key_prefix}ratio_browsers:#{ts}", 1,  ua.name)
+          RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}ratio_browsers:#{ts}", expire) if expire
+
+          RedisAnalytics.redis_connection.zincrby("#{@redis_key_prefix}ratio_platforms:#{ts}", 1, ua.platform.to_s)
+          RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}ratio_platforms:#{ts}", expire) if expire
+          
+          RedisAnalytics.redis_connection.zincrby("#{@redis_key_prefix}ratio_devices:#{ts}", 1, ua.mobile? ? 'M' : 'D')
+          RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}ratio_devices:#{ts}", expire) if expire
+        end        
+      end
+
+      def visit_time(t, visit_end_time = nil)
+        for_each_time_range(t) do |ts, expire|
+          RedisAnalytics.redis_connection.incrby("#{@redis_key_prefix}visit_time:#{ts}", t.to_i - visit_end_time)
+          RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}visit_time:#{ts}", expire) if expire
+        end
+      end
+
+      # 2nd pageview in a visit
+      def page_view(t, second_page_view = false)
+        for_each_time_range(t) do |ts, expire|
+          RedisAnalytics.redis_connection.incr("#{@redis_key_prefix}page_views:#{ts}")
+          RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}page_views:#{ts}", expire) if expire
+
+          RedisAnalytics.redis_connection.incr("#{@redis_key_prefix}second_page_views:#{ts}") if second_page_view
+          RedisAnalytics.redis_connection.expire("#{@redis_key_prefix}second_page_views:#{ts}", expire) if second_page_view and expire
+        end
+      end
+
+
+    end
+  end
+end