web_analytics_discovery 2.0

data/README.md

@@ -0,0 +1,133 @@
+# web_analytics_discovery
+<!--[![Gem Version](https://badge.fury.io/rb/web_analytics_discovery.png)](http://badge.fury.io/rb/web_analytics_discovery)-->
+[![Build Status](https://travis-ci.org/GreyCat/web_analytics_discovery.svg?branch=master)](https://travis-ci.org/GreyCat/web_analytics_discovery)
+[![Dependency Status](https://gemnasium.com/GreyCat/web_analytics_discovery.svg)](https://gemnasium.com/GreyCat/web_analytics_discovery)
+[![Code Climate](https://codeclimate.com/github/GreyCat/web_analytics_discovery/badges/gpa.svg)](https://codeclimate.com/github/GreyCat/web_analytics_discovery)
+<!--[![Coverage Status](https://coveralls.io/repos/GreyCat/web_analytics_discovery/badge.png?branch=master)](https://coveralls.io/r/GreyCat/web_analytics_discovery)-->
+<!--[![Security Status](http://rails-brakeman.com/GreyCat/web_analytics_discovery.png)](http://rails-brakeman.com/GreyCat/web_analytics_discovery)-->
+
+This gem provides a set of tools for discovery and export of data from
+popular web analytics tools.
+
+The supported web analytics systems are:
+
+* Alexa
+* Google Analytics
+* LiveInternet
+* Mail.ru
+* Openstat
+* Quantcast
+* Rambler Top100
+* Yandex Metrika
+
+## The problem
+
+Given a particular site URL (i.e. `http://example.com/`), we'd like to
+know audience statistics on that particular site (i.e. how many unique
+people visit this site per day, per week, per month, how many page views
+do they do, etc).
+
+## The solution
+
+Many sites use web analytics tools to measure audience stats. Quite
+often, these statistics are even available for public, although one needs to know:
+
+* which particular web analytics system a given site uses
+* what is this site's ID in that web analytics system
+
+Answering these question usually requires tedious manual process:
+
+* Look up site's HTML code
+* Locate JavaScript code / tags / calls to web analytics system
+* Identify this system
+* Identify site's ID in the code / calls
+* Go to web analytics's system site or API and get desired statistics
+
+This gem tries automate these tasks, looking up all the info and
+retrieving information from web analytics systems. Exported data can
+be accessed in simple tabular form or programmatically, as a hash,
+using API.
+
+## Installation
+
+### From RubyGems repository
+
+* Make sure you have Ruby and RubyGems
+* Just run `gem install web_analytics_discovery`
+
+### Manually from source
+
+* Clone this repository / download snapshot
+* `gem build web_analytics_discovery.gemspec`
+* `gem install --local ./web_analytics_discovery-*.gem` (usually as
+  root, if you need system-wide installation)
+
+## Basic usage
+
+For basic usage, a simple executable `web_analytics_discover` is
+provided and installed during gem installation. It can be run with one
+or several URLs as command-line arguments and it will produce a simple
+summary table for each of the URLs.
+
+Example:
+
+    $ web_analytics_discover http://kp.ru/
+                        |                      id|      v/day|      s/day|     pv/day|      v/mon|      s/mon|     pv/mon
+    alexa               |                   kp.ru|        N/A|        N/A|    1477599|    6825125|        N/A|   44974428
+    googleanalytics     |           UA-23870775-1|        N/A|        N/A|        N/A|        N/A|        N/A|        N/A
+    liveinternet        |                        |     597956|     745757|    1787863|   10585641|   21308436|   49775501
+    mailru              |                  294001|     756600|        N/A|    2230674|   15086634|        N/A|   73738178
+    openstat            |                 2026010|     983579|    1195306|    2823114|   14757845|   28953554|   69970669
+    quantcast           |                wd:ru.kp|        N/A|        N/A|        N/A|      36300|        N/A|        N/A
+    rambler             |                   17841|    1048235|    1287761|    3015270|   15550162|   31307958|   75869606
+    yandexmetrika       |                 1051362|     259987|     310983|     727833|        N/A|        N/A|   22153416
+
+## API usage
+
+One can easily use web analytics discovery using simple API. Every web
+analytics service is supported by a separate class named after that
+service in `WebAnalyticsDiscovery` module:
+
+* `Alexa`
+* `GoogleAnalytics`
+* `LiveInternet`
+* `MailRu`
+* `Openstat`
+* `Quantcast`
+* `Rambler`
+* `YandexMetrika`
+
+One can use it like that:
+
+    require 'web_analytics_discovery'
+    d = WebAnalyticsDiscovery::MailRu.new
+    result = d.run('http://kp.ru/')
+
+`result` will look like that:
+
+    {:id=>294001,
+     :visitors_day=>756600,
+     :pv_day=>2230674,
+     :visitors_week=>3365344,
+     :pv_week=>13102096,
+     :visitors_mon=>15086634,
+     :pv_mon=>73738178}
+
+Some values might be missing if it's not possible to retrieve them
+from a given service.
+
+## Licensing and usage
+
+Copyright (C) 2013-2014  Mikhail Yakshin <greycat@altlinux.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as
+published by the Free Software Foundation, either version 3 of the
+License, or (at your option) any later version.
+
+This program is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+Affero General Public License for more details.
+
+Please consult LICENSE file for more details and full license text.

data/Rakefile

@@ -0,0 +1,7 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new
+
+task :default => :spec
+task :test => :spec

data/bin/web_analytics_discover

@@ -0,0 +1,77 @@
+#!/usr/bin/env ruby
+
+require 'fileutils'
+require 'uri'
+require 'optparse'
+
+require 'web_analytics_discovery'
+include WebAnalyticsDiscovery
+
+class AnalyticsGrabber
+	def initialize
+		@services = {}
+		SERVICES.each_pair { |name, klass|
+			begin
+				@services[name] = klass.new
+			rescue Exception => ex
+				warn "Unable to start analytics service #{name}"
+				$stderr.puts ex.message
+				$stderr.puts ex.backtrace.join("\n")
+			end
+		}
+	end
+
+	def run(url)
+		r = {}
+		@services.each_pair { |name, service|
+			begin
+				r[name] = service.run(url)
+			rescue Exception => ex
+				warn "Exception querying analytics service #{name}"
+				$stderr.puts ex.message
+				$stderr.puts ex.backtrace.join("\n")
+			end
+		}
+		r
+	end
+
+	def pp(url)
+		r = run(url)
+		print_line ['', 'id', 'v/day', 's/day', 'pv/day', 'v/mon', 's/mon', 'pv/mon']
+		r.keys.sort.each { |service|
+			res = r[service]
+			next unless res
+			print_line [
+				service,
+				res[:id],
+				res[:visitors_day],
+				res[:visits_day],
+				res[:pv_day],
+				res[:visitors_mon],
+				res[:visits_mon],
+				res[:pv_mon],
+			]
+		}
+	end
+
+	def print_line(a)
+		printf '%-20s', a.shift
+		printf '|%24s', a.shift
+		a.each { |x|
+			printf '|%11s', x || 'N/A'
+		}
+		puts
+	end
+end
+
+options = {}
+OptionParser.new { |opts|
+	opts.banner = "Usage: #{__FILE__} [options] <urls>"
+
+#	opts.on('-v', '--[no-]verbose', 'Verbose logging') { |v| options[:verbose] = v }
+
+	opts.on_tail('-h', '--help', 'Show this message') { puts opts; exit }
+}.parse!
+
+ag = AnalyticsGrabber.new
+ARGV.each { |a| ag.pp(a) }

data/lib/web_analytics_discovery.rb

@@ -0,0 +1,23 @@
+require 'web_analytics_discovery/version'
+
+require 'web_analytics_discovery/grabber/alexa'
+require 'web_analytics_discovery/grabber/googleanalytics'
+require 'web_analytics_discovery/grabber/liveinternet'
+require 'web_analytics_discovery/grabber/mailru'
+require 'web_analytics_discovery/grabber/openstat'
+require 'web_analytics_discovery/grabber/quantcast'
+require 'web_analytics_discovery/grabber/rambler'
+require 'web_analytics_discovery/grabber/tns'
+require 'web_analytics_discovery/grabber/yandexmetrika'
+
+module WebAnalyticsDiscovery
+	# Special trickery to get a map of {:service_name => ClassThatImplementsServiceExtraction} magic
+	SERVICES = Hash[constants.map { |x|
+		possible_class = const_get(x)
+		if possible_class.class == Class
+			[x.to_s.downcase.to_sym, possible_class]
+		else
+			nil
+		end
+	}.delete_if { |v| v.nil? }]
+end

data/lib/web_analytics_discovery/grabber/alexa.rb

@@ -0,0 +1,33 @@
+#!/usr/bin/env ruby
+
+require 'uri'
+require 'web_analytics_discovery/grabberutils'
+
+module WebAnalyticsDiscovery
+class Alexa
+	include GrabberUtils
+
+	def run(url)
+		uri = URI.parse(url)
+		host = uri.host
+		r = {}
+		doc = download("http://www.alexa.com/siteinfo/#{host}#trafficstats")
+
+		# Try to extract certified metrics
+		r[:visitors_day], r[:pv_day], r[:visitors_mon], r[:pv_mon] = grab_certified_metrics(doc)
+
+		# Grab ID for clarity's sake
+		if doc =~ /<img src="http:\/\/traffic\.alexa\.com\/graph\?.*&u=([^"]+)">/
+			r[:id] = $1
+		end
+		return r
+	end
+
+	def grab_certified_metrics(doc)
+		r = []
+		doc.gsub(/<strong class="metrics-data">([0-9,]+)<\/strong>/) { r << $1 }
+		r.map! { |x| x.gsub(/,/, '').to_i }
+		return r
+	end
+end
+end

data/lib/web_analytics_discovery/grabber/googleanalytics.rb

@@ -0,0 +1,29 @@
+require 'web_analytics_discovery/grabberutils'
+
+module WebAnalyticsDiscovery
+class GoogleAnalytics
+	include GrabberUtils
+
+	def run(url)
+		@page = download(url)
+		run_id(find_id)
+	end
+
+	def find_id
+		case @page
+		when /_gat\._getTracker\(["']([^"']+)["']\)/
+			$1
+		when /_gaq\.push\(\[['"]_setAccount['"], ['"]([^"']+)['"]\]\)/
+			$1
+		else
+			nil
+		end
+	end
+
+	def run_id(id)
+		return nil unless id
+		r = {:id => id}
+		return r
+	end
+end
+end

data/lib/web_analytics_discovery/grabber/liveinternet.rb

@@ -0,0 +1,61 @@
+# -*- coding: utf-8 -*-
+
+require 'uri'
+require 'web_analytics_discovery/grabberutils'
+
+module WebAnalyticsDiscovery
+class LiveInternet
+	include GrabberUtils
+
+	def run(url)
+		@url = url
+		@page = download(url)
+		run_id(find_id)
+	end
+
+	def find_id
+		case @page
+		when /new Image\(\)\.src = "\/\/counter\.yadro\.ru\/hit;([^?"]+)\?/
+			$1
+		else
+			# Use hostname as a last resort measure
+			URI.parse(@url).host
+		end
+	end
+
+	def run_id(host)
+		r = {:id => host}
+
+		doc = download("http://www.liveinternet.ru/stat/#{host}/index.csv")
+		r[:pv_day], r[:visits_day], r[:visitors_day] = grab_psv(doc, 4)
+
+		# Bail out early if no LiveInternet data available
+		return r unless r[:pv_day]
+
+		doc = download("http://www.liveinternet.ru/stat/#{host}/index.csv?period=week;total=yes")
+		r[:pv_week], r[:visits_week], r[:visitors_week] = grab_psv(doc, 2)
+
+		doc = download("http://www.liveinternet.ru/stat/#{host}/index.csv?period=month;total=yes")
+		r[:pv_mon], r[:visits_mon], r[:visitors_mon] = grab_psv(doc, 2)
+
+		return r
+	end
+
+	private
+	def grab_psv(doc, col)
+		r = [nil, nil, nil]
+		doc.split(/\n/).each { |l|
+			c = l.split(/;/)
+			case c[0]
+			when '"Просмотры"'
+				r[0] = c[col].to_i
+			when '"Сессии"'
+				r[1] = c[col].to_i
+			when '"Посетители"'
+				r[2] = c[col].to_i
+			end
+		}
+		return r
+	end
+end
+end

data/lib/web_analytics_discovery/grabber/mailru.rb

@@ -0,0 +1,89 @@
+# -*- coding: utf-8 -*-
+
+require 'web_analytics_discovery/grabberutils'
+
+module WebAnalyticsDiscovery
+class MailRu
+	include GrabberUtils
+
+	def run(url)
+		@page = download(url)
+		run_id(find_id)
+	end
+
+	def find_id
+		case @page
+		when /<a [^>]*href="http:\/\/top\.mail\.ru\/jump\?from=(\d+)".*>\s*<img src="http:\/\/.*.top.mail.ru\/counter/m,
+			/<img src=['"]?http:\/\/top\.list\.ru\/counter\?id=(\d+)/,
+			/<img src=['"]?http:\/\/.*top\.mail\.ru\/counter\?js=na;id=(\d+)/,
+			/_tmr.push\(\{id:\s*['"](\d+)['"]/
+			$1.to_i
+		else
+			nil
+		end
+	end
+
+	def run_id(id)
+		return nil unless id
+		r = {:id => id}
+
+		#doc = download("http://top.mail.ru/visits?id=#{id}")
+
+		# Analyze daily report
+		doc = download("http://top.mail.ru/visits.csv?id=#{id}&period=0&date=&back=30&", 'windows-1251').split(/\n/)
+		return run_id_html_rating(r, id) if doc.empty?
+		doc = doc[4..-1]
+
+		sum_v = 0
+		sum_pv = 0
+		doc.each { |l|
+			#"Дата";"Посетители";"Новые посетители";"Ядро";"Хосты";"Просмотры";"Глубина"
+			date, v, new_v, core_v, hosts, pv, depth = l.split(/;/)
+			sum_v += v.to_i
+			sum_pv += pv.to_i
+		}
+
+		r[:visitors_day] = sum_v / doc.size
+		r[:pv_day] = sum_pv / doc.size
+
+		# Analyze weekly report
+		doc = download("http://top.mail.ru/visits.csv?id=#{id}&period=1&date=&back=98&", 'windows-1251').split(/\n/)
+		return r if doc.empty?
+		date, v, new_v, core_v, hosts, pv, depth = doc[4].split(/;/)
+		r[:visitors_week] = v.to_i
+		r[:pv_week] = pv.to_i
+
+		# Analyze monthly report
+		doc = download("http://top.mail.ru/visits.csv?id=#{id}&period=2&date=&back=395&", 'windows-1251').split(/\n/)
+		return r if doc.empty?
+		date, v, new_v, core_v, hosts, pv, depth = doc[4].split(/;/)
+		r[:visitors_mon] = v.to_i
+		r[:pv_mon] = pv.to_i
+
+		return r
+	end
+
+	# Parse semi-closed rating when normal full CSV export is not available
+	def run_id_html_rating(r, id)
+		doc = download("http://top.mail.ru/rating?id=#{id}", 'windows-1251')
+
+		today = []
+		doc.gsub(/<td class="l_col">Сегодня<\/td>.*?<td class="r_col"><b>([0-9,]+)<\/b>/m) { today << $1.gsub(/,/, '').to_i }
+
+		week = []
+		doc.gsub(/<td class="l_col">Неделя<\/td>.*?<td class="r_col"><b>([0-9,]+)<\/b>/m) { week << $1.gsub(/,/, '').to_i }
+
+		month = []
+		doc.gsub(/<td class="l_col">Месяц<\/td>.*?<td class="r_col"><b>([0-9,]+)<\/b>/m) { month << $1.gsub(/,/, '').to_i }
+
+		# Non-normal number of matches? That's weird, bail out
+		return r unless today.length == 3 and week.length == 3 and month.length == 3
+
+		r[:visitors_day], r[:pv_day], r[:ip_day] = today
+		r[:visitors_week], r[:pv_week], r[:ip_week] = week
+		r[:visitors_mon], r[:pv_mon], r[:ip_mon] = month
+
+		return r
+	end
+end
+end