panmind-usage-tracker 0.4.0 → 1.0.1

data/Gemfile

@@ -0,0 +1,15 @@
+source :rubygems
+
+gem 'bson_ext'
+gem 'mongo'
+gem 'couchrest' 
+gem 'activesupport', '~>3.0.3'
+gem 'json' 
+gem 'eventmachine' 
+gem 'freegenie-em-spec', '0.2.3'
+gem 'rspec'
+
+
+
+
+

data/README.md

@@ -5,54 +5,48 @@ What is it?
 ===========
 
  1. A `Rack::Middleware` that sends selected parts of the request environment to an UDP socket
- 2. An `EventMachine` daemon that opens an UDP socket and sends out received data to CouchDB
- 3. A set of CouchDB map-reduce views, for analysis
-
+ 2. An `EventMachine` daemon that opens an UDP socket and sends out received data to a database
 
 Does it work?
 =============
 
-Yes, but the release is still incomplete, because currently tests are too
-tied to Panmind logic.
-
+Yes, we are using it in production. 
 If you can help in complete the test suite, it is much appreciated :-).
 
 Deploying
 =========
 
- * Add the usage_tracker gem to your Gemfile and require the middleware
+ * Add the usage\_tracker gem to your Gemfile and require the middleware
 
-   gem 'usage_tracker', :require => 'usage_tracker/middleware'
+   gem 'usage\_tracker', :require => 'usage\_tracker/middleware'
 
- * Add the Middleware to your application:
+ * Configure the middleware and plug it into your application:
 
+    UsageTracker::Middleware.config(:host => '192.168.1.20', :port => '8840')
     Your::Application.config.middleware.use UsageTracker::Middleware
 
- * The daemon can be started manually with the following command, inside a Rails.root:
+ * Install the gem on the target machine and run it with this command:
 
      $ usage_tracker [environment]
 
+   If you run it into a Rails.root it will log and write pids in canonical dirs.
+
    `environment` is optional and will default to "development" if no command line
-   option nor the RAILS_ENV environment variable are set.
+   option nor the RAILS\_ENV environment variable are set.
 
    or can be put under Upstart using the provided configuration file located in
    `config/usage_tracker_upstart.conf`. Check it out and modify it to suit your needs.
 
-   The daemon logs to `log/usage_tracker.log` and rotates its logs when receives
-   the USR1 signal.
+   The daemon logs to `usage_tracker.log` if the log directory exists and rotates its 
+   logs when receives the USR1 signal. 
 
- * The daemon writes its pid into tmp/pids/usage_tracker.pid
+ * The daemon writes its pid into usage\_tracker.pid
 
- * The daemon connects to a Couch database named `usage_tracker` running on `localhost`,
-   default port `5984/TCP`, and listens on `localhost`, port `5985/UDP` by default.
-   You can change these settings via a `config/usage_tracker.yml` file. See the example
-   in the `config` directory of the gem distribution.
-
- * The CouchDB instance must be running, the database is created (and updated)
-   if necessary.
+ * The daemon can be configured to work with couchdb or mongodb adapter. Look at the 
+   sample configuration file for hints.
 
  * If the daemon cannot start, e.g. because of unavailable database or listening
-   address, it will print a diagnostig message to STDERR, log to usage_tracker.log
+   address, it will print a diagnostig message to STDERR, log to usage\_tracker.log
    and exit with status of 1.
 
  * The daemon exits gracefully if it receives the INT or the TERM signals.
@@ -60,6 +54,15 @@ Deploying
 Testing
 =======
 
-The current test suite, brutally extracted from Panmind codebase, is in the
-`middleware_test.rb` file at the root of the Gem distribution. It is of no
-use except Panmind, but it's a start for writing new ones. Please help! :-)
+Our will is to test the Evented code in isolation using rspec and em-rspec gem. 
+Tests are still incomplete. You can start the running: 
+
+> bundle exec rspec spec 
+
+All required gems for testing should be installed running: 
+
+> bundle install
+
+About the middleware, it's probably better for you to test that in your own 
+app's integration test suite. 
+

data/Rakefile

@@ -8,18 +8,21 @@ begin
 
   Jeweler::Tasks.new do |gemspec|
     gemspec.name        = 'panmind-usage-tracker'
-    gemspec.summary     = 'Write your application request logs in CouchDB'
+    gemspec.summary     = 'Write your application request logs on CouchDB or MongoDB'
     gemspec.description = 'This software implements a Rails 3 Middleware and ' \
-                          'an EventMachine reactor to store into CouchDB the ' \
+                          'an EventMachine reactor to store into a database the ' \
                           'results of HTTP request processing'
 
-    gemspec.authors     = ['Marcello Barnaba', 'Christian Wörner']
+    gemspec.authors     = ['Marcello Barnaba', 'Christian Wörner', 'Fabrizio Regini']
     gemspec.homepage    = 'http://github.com/Panmind/usage_tracker'
-    gemspec.email       = 'vjt@openssl.it'
+    gemspec.email       = 'info@panmind.org'
 
     gemspec.add_dependency('rails', '~> 3.0')
     gemspec.add_dependency('eventmachine')
     gemspec.add_dependency('couchrest')
+    gemspec.add_dependency('mongo')
+    gemspec.add_dependency('bson')
+    gemspec.add_dependency('bson_ext')
   end
 rescue LoadError
   puts 'Jeweler not available. Install it with: gem install jeweler'
@@ -30,7 +33,7 @@ Rake::RDocTask.new do |rdoc|
   rdoc.rdoc_files.add %w( README.md lib/**/*.rb )
 
   rdoc.main  = 'README.md'
-  rdoc.title = 'Rails Application Usage Tracker on CouchDB'
+  rdoc.title = 'Rails Application Usage Tracker'
 end
 
 desc 'Will someone help write tests?'
@@ -39,4 +42,3 @@ task :default do
   puts 'Can you help in writing tests? Please do :-)'
   puts
 end
-

data/VERSION

@@ -1 +1 @@
-0.4.0
+1.0.1

data/bin/usage_tracker

@@ -1,4 +1,4 @@
 #!/usr/bin/env ruby
 $LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
 
-require 'usage_tracker/reactor'
+require 'usage_tracker/runner'

data/config/usage_tracker.yml.sample

@@ -1,6 +1,20 @@
 defaults: &defaults
-  couchdb: "http://admin:suxsux@127.0.0.1:5984/usage_tracker"
+  # Available adapters: 'couchdb', 'mongodb'
+  # adapter: 'couchdb'
+
+  # CouchDB connect settings
+  # database: "http://127.0.0.1:5984/usage_tracker"
+
+  #  MongoDB connect settings
+  adapter: 'mongodb' 
+  database:
+    name: "usage_tracker"
+    host: "127.0.0.1"
+    port: 27017
+    collection: "data"
+
   listen:  "127.0.0.1:5985"
+  log_level: 'debug'
 
 development:
   <<: *defaults
@@ -10,4 +24,7 @@ production:
 
 test:
   <<: *defaults
-  couchdb: "http://admin:suxsux@127.0.0.1:5984/usage_tracker_test"
+  
+spec:
+  <<: *defaults
+

data/lib/usage_tracker.rb

@@ -6,6 +6,7 @@ require 'ostruct'
 require 'couchrest'
 require 'active_support/core_ext/object/blank'
 require 'usage_tracker/log'
+require 'usage_tracker/adapter'
 
 module UsageTracker
   class << self
@@ -14,11 +15,6 @@ module UsageTracker
       @env ||= ENV['RAILS_ENV'] || ARGV[0] || 'development'
     end
 
-    @@defaults = {
-      'couchdb' => 'http://localhost:5984/usage_tracker',
-      'listen'  => '127.0.0.1:5985'
-    }
-
     # Memoizes settings from the ./config/usage_tracker.yml file,
     # relative from __FILE__ and searches for the "usage_tracker"
     # configuration block. Raises RuntimeError if it cannot find
@@ -32,10 +28,9 @@ module UsageTracker
         settings = YAML.load(rc_file.read)[env] if rc_file.exist?
 
         if settings.blank?
-          settings = @@defaults
-          log "#{env} configuration block not found in #{rc_file}, using defaults"
-        elsif settings.values_at(*%w(couchdb listen)).any?(&:blank?)
-          raise "Incomplete configuration: please set the 'couchdb' and 'listen' keys"
+          raise "Configuration missing for #{env}"
+        elsif settings.values_at(*%w(adapter database listen)).any?(&:blank?)
+          raise "Incomplete configuration: please set the 'adapter', 'database' and 'listen' keys"
         end
 
         host, port = settings.delete('listen').split(':')
@@ -46,6 +41,9 @@ module UsageTracker
 
         settings['host'], settings['port'] = host, port.to_i
 
+        settings['log_level'] ||= :warn
+        log.level = settings['log_level']
+
         OpenStruct.new settings
       end
     end
@@ -54,6 +52,10 @@ module UsageTracker
       @database or raise "Not connected to the database"
     end
 
+    def adapter
+      @adapter or raise "Not connected to the database adapter"
+    end
+
     # Connects to the configured CouchDB and memoizes the
     # CouchRest::Database connection into an instance variable
     # and calls +load_views!+
@@ -61,17 +63,23 @@ module UsageTracker
     # Raises RuntimeError if the connection could not be established
     #
     def connect!
-      @database =
-        CouchRest.database!(settings.couchdb).tap do |db|
-          db.info
-          log "Connected to database #{settings.couchdb}"
-        end
-
-      load_views!
-    rescue Errno::ECONNREFUSED, RestClient::Exception => e
-      raise "Unable to connect to database #{settings.couchdb}: #{e.message}"
+      @adapter = Adapter::new settings
+      @database = @adapter.database
     end
 
+    # Code to run inside EventMachine
+    def run! 
+      host, port = UsageTracker.settings.host, UsageTracker.settings.port
+
+      unless (1024..65535).include? port.to_i
+        raise "Please set a listening port between 1024 and 65535"
+      end
+
+      EventMachine.open_datagram_socket host, port, Reactor
+      log "Listening on #{host}:#{port} UDP"
+      write_pid!
+    end 
+
     def write_pid!(pid = $$)
       dir = Pathname.new('.').join('tmp', 'pids')
       dir = Pathname.new(Dir.tmpdir) unless dir.directory?
@@ -87,29 +95,6 @@ module UsageTracker
       log.error message
       Kernel.raise Error, message
     end
-
-    private
-      # Loads CouchDB views from views.yml and verifies that
-      # they are loaded in the current instance, upgrading
-      # them if necessary.
-      def load_views!
-        new = YAML.load ERB.new(
-          Pathname.new(__FILE__).dirname.join('..', 'config', 'views.yml').read
-        ).result
-
-        id  = new['_id']
-        old = database.get id
-
-        if old['version'].to_i < new['version'].to_i
-          log "Upgrading Design Document #{id} to v#{new['version']}"
-          database.delete_doc old
-          database.save_doc new
-        end
-
-      rescue RestClient::ResourceNotFound
-        log "Creating Design Document #{id} v#{new['version']}"
-        database.save_doc new
-      end
   end
 
   class Error < StandardError; end

data/lib/usage_tracker/adapter.rb

@@ -0,0 +1,19 @@
+require 'usage_tracker/adapters/couchdb'
+require 'usage_tracker/adapters/mongodb'
+
+module UsageTracker
+  class Adapter
+    def self::new(settings)
+      klass =
+        case settings.adapter
+          when 'couchdb'
+            Adapters::Couchdb
+          when 'redis'
+            Adapters::Redis
+          when 'mongodb'
+            Adapters::Mongodb
+        end
+      klass::new(settings)
+    end
+  end
+end

data/lib/usage_tracker/adapters/couchdb.rb

@@ -0,0 +1,35 @@
+require 'pathname'
+require 'couchrest'
+require 'yaml'
+
+module UsageTracker
+  module Adapters
+    class Couchdb
+      attr_accessor :database
+      def initialize (settings)
+        @database =
+          CouchRest.database!(settings.database).tap do |db|
+            db.info
+          end
+      rescue Errno::ECONNREFUSED, RestClient::Exception => e
+        raise "Unable to connect to database #{settings.database}: #{e.message}"
+      end
+
+      def save_doc (doc)
+        doc['_id'] = make_id if doc['_id'].nil?
+        @database.save_doc(doc)
+      end
+
+      # Timestamp as _id has the advantage that documents
+      # are sorted automatically by CouchDB.
+      #
+      # Eventual duplication (multiple servers) is (possibly)
+      # avoided by adding a random digit at the end.
+      #
+      def make_id
+        Time.now.to_f.to_s.ljust(16, '0') + rand(10).to_s
+      end
+
+    end
+  end
+end

data/lib/usage_tracker/adapters/mongodb.rb

@@ -0,0 +1,23 @@
+require 'pathname'
+require 'mongo'
+require 'yaml'
+
+module UsageTracker
+  module Adapters
+    class Mongodb
+      attr_accessor :database
+      def initialize (settings)
+        @database =
+          db = Mongo::Connection.new(settings.database['host'], settings.database['port']).db(settings.database['name'])
+          @collection = db[settings.database['collection']]
+          db
+      rescue Errno::ECONNREFUSED, Mongo::ConnectionError => e
+        raise "Unable to connect to database #{settings.database['name']} with #{settings.adapter} adapter: #{e.message}"
+      end
+
+      def save_doc(doc)
+        @collection.insert(doc)
+      end
+    end
+  end
+end

data/lib/usage_tracker/log.rb

@@ -2,6 +2,15 @@ require 'logger'
 
 module UsageTracker
   class Log
+
+    Levels = {
+        :fatal => Logger::FATAL, 
+        :erorr => Logger::ERROR, 
+        :warn  => Logger::WARN, 
+        :info  => Logger::INFO, 
+        :debug => Logger::DEBUG 
+    }
+
     attr_reader :path
 
     [:debug, :info, :warn, :error, :fatal].each do |severity|
@@ -41,5 +50,11 @@ module UsageTracker
       close
       open
     end
+
+    def level=(level)
+      level = level.to_sym 
+      raise "Invalid log level" unless Levels.keys.include?(level)
+      @logger.level = Levels[level]
+    end
   end
 end

data/lib/usage_tracker/middleware.rb

@@ -1,15 +1,22 @@
 require 'timeout'
-require 'usage_tracker'
+require 'usage_tracker/log' 
 require 'usage_tracker/context'
 require 'usage_tracker/railtie' if defined?(Rails)
 
 # This middleware extracts some data from the incoming request
 # and sends it to the reactor, that parses and stores it.
 #
+
 module UsageTracker
   class Middleware
+
+    @@host    = 'localhost'
+    @@port    = 5985
+    @@backend = `hostname`.strip
+    @@logger  = UsageTracker::Log.new 
+
     @@headers = [
-      "REMOTE_ADDR",
+      # "REMOTE_ADDR",
       "REQUEST_METHOD",
       "PATH_INFO",
       "REQUEST_URI",
@@ -33,14 +40,11 @@ module UsageTracker
       "QUERY_STRING"
     ].freeze
 
-    @@backend, @@host, @@port = [
-      `hostname`.strip,
-      UsageTracker.settings.host,
-      UsageTracker.settings.port
-    ].each(&:freeze)
-
-    def initialize(app)
-      @app = app
+    def initialize(app, options={})
+      @@host    = options[:host]     if options.keys.include?(:host) 
+      @@port    = options[:port]     if options.keys.include?(:port) 
+      @@backend = options[:backend]  if options.keys.include?(:backend)
+      @app      = app
     end
 
     def call(env)
@@ -50,13 +54,14 @@ module UsageTracker
 
       begin
         data = {
-          :user_id  => env['rack.session'][:user_id],
-          :duration => ((req_end - req_start) * 1000).to_i,
-          :backend  => @@backend,
-          :xhr      => env['HTTP_X_REQUESTED_WITH'] == 'XMLHttpRequest',
-          :context  => env[Context.key],
-          :env      => {},
-          :status   => response[0] # response contains [status, headers, body]
+          :user_id   => env['rack.session'][:user_id],
+          :remote_ip => env['action_dispatch.remote_ip'].to_s, 
+          :duration  => ((req_end - req_start) * 1000).to_i,
+          :backend   => @@backend,
+          :xhr       => env['HTTP_X_REQUESTED_WITH'] == 'XMLHttpRequest',
+          :context   => env[Context.key],
+          :env       => {},
+          :status    => response[0] # response contains [status, headers, body]
         }
 
         @@headers.each {|key| data[:env][key.downcase] = env[key] unless env[key].blank?}
@@ -65,21 +70,30 @@ module UsageTracker
 
       rescue
         raise unless response # Error in the application, raise it up
-
         # Error in usage tracker itself
-        UsageTracker.log($!.message)
-        UsageTracker.log($!.backtrace.join("\n"))
+        @@logger.error($!.message)
+        @@logger.error($!.backtrace.join("\n"))
+        
       end
 
       return response
     end
 
     class << self
+
+      def development? 
+        defined?(Rails) && Rails.env.development? 
+      end
+
       # Writes the given `data` to the reactor, using the UDP protocol.
       # Times out after 1 second. If a write error occurs, data is lost.
       #
+
       def track(data)
         Timeout.timeout(1) do
+
+          @@logger.debug("Sending to #{@@host}:#{@@port} : #{data.to_json}") if development? 
+
           UDPSocket.open do |sock|
             sock.connect(@@host, @@port.to_i)
             sock.write_nonblock(data << "\n")
@@ -87,7 +101,7 @@ module UsageTracker
         end
 
       rescue Timeout::Error, Errno::EWOULDBLOCK, Errno::EAGAIN, Errno::EINTR
-        UsageTracker.log "Cannot track data: #{$!.message}"
+        @@logger.error "Cannot track data: #{$!.message}"
       end
     end
   end