memory_tracker 1.1.1 → 1.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ab14d34d93892036e9f137fd33242cdac5bdcf1c
+  data.tar.gz: 8c82efd9ff6680f4565778436c8305ea340ff259
+SHA512:
+  metadata.gz: dde284cd9242dc1a59394e5db587831478cf0936f8f570a55fe431176fa1634e80eeccefb030ee3c0da90ac3b09449e541de8a15ae75f2dfcc134964b2514404
+  data.tar.gz: 6299e4aa58d5bd4c30a028a0d113e3cc78efc6480f7dbd275ec776fcc1b008604645731ca8613e41cb3dce8467d5545838633f79447ac420e6b79f08c16673cf

data/Gemfile.lock

@@ -1,14 +1,15 @@
 PATH
   remote: .
   specs:
-    memory_tracker (1.0.6)
+    memory_tracker (1.2.0)
       activesupport
+      logfile_interval (~> 1.1.0)
       sys-proctable
 
 GEM
   remote: https://rubygems.org/
   specs:
-    activesupport (4.0.1)
+    activesupport (4.0.2)
       i18n (~> 0.6, >= 0.6.4)
       minitest (~> 4.2)
       multi_json (~> 1.3)
@@ -16,15 +17,16 @@ GEM
       tzinfo (~> 0.3.37)
     atomic (1.1.14)
     columnize (0.3.6)
-    debugger (1.6.2)
+    debugger (1.6.3)
       columnize (>= 0.3.1)
       debugger-linecache (~> 1.2.0)
-      debugger-ruby_core_source (~> 1.2.3)
+      debugger-ruby_core_source (~> 1.2.4)
     debugger-linecache (1.2.0)
-    debugger-ruby_core_source (1.2.3)
-    diff-lcs (1.2.4)
-    i18n (0.6.5)
+    debugger-ruby_core_source (1.2.4)
+    diff-lcs (1.2.5)
+    i18n (0.6.9)
     json (1.8.1)
+    logfile_interval (1.1.0)
     minitest (4.7.5)
     multi_json (1.8.2)
     rdoc (3.12.2)
@@ -34,7 +36,7 @@ GEM
       rspec-expectations (~> 2.14.0)
       rspec-mocks (~> 2.14.0)
     rspec-core (2.14.7)
-    rspec-expectations (2.14.3)
+    rspec-expectations (2.14.4)
       diff-lcs (>= 1.1.3, < 2.0)
     rspec-mocks (2.14.4)
     sys-proctable (0.9.3)

data/README.md

@@ -1,10 +1,10 @@
-# MemoryTracker
+# MemoryTracker [![Build Status](https://travis-ci.org/plerohellec/memory_tracker.png?branch=master)](https://travis-ci.org/plerohellec/memory_tracker)
 
-MemoryTracker collects, analyzes and reports memory usage data of each controller action of a Rails application. The data will identify the parts of the application that are the most memory hungry, so that they can be fixed and the overall memory usage of the application be reduced.
+MemoryTracker finds where the memory bloat comes from in a Rails application.
 
 # Features
 * Collect stats on each controller action memory consumption.
-* Display stats on every controller action of the application in a engine webpage embedded in the gem.
+* Display stats on every controller action of the application in a web page (the dashboard) embedded in the gem.
 
 # Dashboard
 ## Screenshot
@@ -19,9 +19,18 @@ That's where the collected statistics are displayed.
 * Controller actions that have a "total_allocated_object" value N times higher than the average.
 * Controller actions that trigger the garbage collector frequently. That the "count" value in the dashboard.
 
+# Enabling MemoryTracker
+Add it to your Gemfile:
+```ruby
+gem 'memory_tracker'
+```
+To get access to the MemoryTracker UI and see the stats, the engine must be mounted at config/routes.rb
+```ruby
+mount MemoryTracker::Engine => "/memtracker"
+```
 
-# How?
-MemoryTracker uses system memory data and Ruby garbage collector statistics to find the memory currently used by The Rails processes, how many objects were allocated in Ruby heaps, how many heaps were created and how many times the garbage collector ran. The MemoryTracker middleware captures the data before and after each HHTP request and saves the deltas per controller/action in a variety of stores.
+# How does it work?
+MemoryTracker uses system memory data and Ruby garbage collector statistics to find the memory currently used by The Rails processes, how many objects were allocated in Ruby heaps, how many heaps were created and how many times the garbage collector ran. The MemoryTracker middleware captures the data before and after each HTTP request and saves the deltas per controller/action in one or more stores.
 
 MemoryTracker is implemented as a Rails engine and reports its statistics in a web page served by the application where it's embedded. The report currently includes data from the InMemoryStore.
 
@@ -59,22 +68,11 @@ test:
   <<: *defaults
 ```
 
-# Enabling MemoryTracker
-Add it to your Gemfile:
-```ruby
-gem 'memory_tracker'
-```
-To get access to the MemoryTracker UI and see the stats, the engine must be mounted at config/routes.rb
-```ruby
-mount MemoryTracker::Engine => "/memtracker"
-```
-Once the app is running, go to /memtracker to review the list of controller actions.
-
 # Requirements
 The gem has been tested and is compatible with:
 * Ruby 2.0
 * Ruby 1.9 but 1.9 GC.stat does not include the total_allocated_object number so its value remains at 0 in MemoryTracker.
-* Rails 3.2
+* Rails 3.2, Rails 4.0
 
 # Tips
 * Ruby processes rarely leak memory. Instead they allocate memory whenever they need it and never release it. This memory is not lost, it's still available to the process that allocated it but it's unavailable for all other processes in the system.
@@ -87,7 +85,6 @@ The gem has been tested and is compatible with:
 # Coming soon
 * Redis store.
 * Store and report the most expensive URLs.
-* Support for Rails 4 (it might already work, it's just not been tested yet).
 
 # Contributing to memory_tracker
  
@@ -99,8 +96,10 @@ The gem has been tested and is compatible with:
 * Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
 * Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
 
+# References
+* [Demystifying the Ruby GC](http://samsaffron.com/archive/2013/11/22/demystifying-the-ruby-gc)
+
 # Copyright
 
-Copyright (c) 2013 Philippe Le Rohellec. See LICENSE.txt for
-further details.
+See LICENSE.txt.
 

data/lib/memory_tracker/env.rb

@@ -5,10 +5,15 @@ module MemoryTracker
     def initialize(env)
       @path = env['PATH_INFO']
 
-      routes_env = { :method => env['REQUEST_METHOD'] }
-      request = Rails.application.routes.recognize_path(env['REQUEST_URI'], routes_env)
-      @controller = request[:controller]
-      @action     = request[:action]
+      begin
+        routes_env = { :method => env['REQUEST_METHOD'] }
+        request = Rails.application.routes.recognize_path(env['REQUEST_PATH'], routes_env)
+        @controller = request[:controller]
+        @action     = request[:action]
+      rescue ActionController::RoutingError
+        @controller = 'unknown'
+        @action = 'unknown'
+      end
     end
   end
 end

data/lib/memory_tracker/gc_stat.rb

@@ -10,8 +10,8 @@ module MemoryTracker
       @stats.keys.sort
     end
 
-    def ordered_values
-      ordered_keys.inject([]) do |vals, key|
+    def ordered_values(ordered_columns = ordered_keys)
+      ordered_columns.inject([]) do |vals, key|
         vals << @stats[key]
         vals
       end

data/lib/memory_tracker/stores/gcstat_logfile_parser.rb

@@ -0,0 +1,31 @@
+require 'logfile_interval'
+
+module MemoryTracker
+  module Stores
+    class GcstatLogfileParser < LogfileInterval::LineParser::Base
+      # Line format:
+      # timestamp,action,count,heap_final_num,heap_free_num,heap_increment,heap_length,heap_live_num,heap_used,rss,total_allocated_object,total_freed_object,vsize
+
+
+      set_regex /^(\d+),(\w+#\w+),(\d+),(\d+),(\d+),(\d+),(\d+),(\d+),(\d+),([\d\.]+),(\d+),(\d+),([\d\.]+)$/
+
+      add_column :name => :timestamp,              :pos => 1,  :aggregator => :timestamp
+      add_column :name => :action,                 :pos => 2,  :aggregator => :count, :group_by => :action
+      add_column :name => :count,                  :pos => 3,  :aggregator => :average,   :conversion => :integer
+#       add_column :name => :heap_final_num,         :pos => 4,  :aggregator => :average,   :conversion => :integer
+#       add_column :name => :heap_free_num,          :pos => 5,  :aggregator => :average,   :conversion => :integer
+#       add_column :name => :heap_increment,         :pos => 6,  :aggregator => :average,   :conversion => :integer
+#       add_column :name => :heap_length,            :pos => 7,  :aggregator => :average,   :conversion => :integer
+#       add_column :name => :heap_live_num,          :pos => 8,  :aggregator => :average,   :conversion => :integer
+      add_column :name => :heap_used,              :pos => 9,  :aggregator => :average,   :conversion => :integer
+      add_column :name => :rss,                    :pos => 10,  :aggregator => :delta,   :conversion => :float, :group_by => :action
+      add_column :name => :total_allocated_object, :pos => 11,  :aggregator => :average,   :conversion => :integer
+#       add_column :name => :total_freed_object,     :pos => 12, :aggregator => :average,   :conversion => :integer
+      add_column :name => :vsize,                  :pos => 13, :aggregator => :delta,   :conversion => :float, :group_by => :action
+
+      def time
+        Time.at(self.timestamp.to_i)
+      end
+    end
+  end
+end

data/lib/memory_tracker/stores/gcstat_logfile_store.rb

@@ -3,8 +3,10 @@ module MemoryTracker
     class GcstatLogfileStore < Base
       register_store :gcstat_logfile
 
+      COLUMNS = [ :count, :heap_final_num, :heap_free_num, :heap_increment, :heap_length, :heap_live_num, :heap_used, :rss, :total_allocated_object, :total_freed_object, :vsize ]
+
       def initialize(opts)
-        logger_class = opts.fetch(:logger_class, 'ActiveSupport::BufferedLogger')
+        logger_class = opts.fetch(:logger_class, 'Logger')
         filename     = opts.fetch(:filename, "#{Rails.root}/log/memtracker_gcstat.log")
 
         @logger = logger_class.constantize.new(filename)
@@ -25,15 +27,16 @@ module MemoryTracker
       private
 
       def write_header
-        @logger.info "##{@request.end_gcstat.ordered_keys.join(',')}"
+        @logger.info "##{COLUMNS.join(',')}"
       end
 
       def write_request_log
-        @logger.info logline
+        @logger.info "#{Time.now.to_i},#{logline}"
       end
 
       def logline
-        @request.end_gcstat.ordered_values.join ','
+        logline = "#{@request.controller}##{@request.action},"
+        logline << @request.end_gcstat.ordered_values(COLUMNS).join(',')
       end
     end
   end

data/lib/memory_tracker/stores/url_logfile_store.rb

@@ -4,7 +4,7 @@ module MemoryTracker
       register_store :url_logfile
 
       def initialize(opts)
-        logger_class = opts.fetch(:logger_class, 'ActiveSupport::BufferedLogger')
+        logger_class = opts.fetch(:logger_class, 'Logger')
         filename     = opts.fetch(:filename, "#{Rails.root}/log/memtracker_urls.log")
 
         @logger = logger_class.constantize.new(filename)

data/lib/memory_tracker/version.rb

@@ -1,3 +1,3 @@
 module MemoryTracker
-  VERSION = "1.1.1"
+  VERSION = "1.2.0"
 end

data/memory_tracker.gemspec

@@ -38,6 +38,7 @@ Gem::Specification.new do |s|
     "lib/memory_tracker/memory_tracker.rb",
     "lib/memory_tracker/middleware.rb",
     "lib/memory_tracker/request.rb",
+    "lib/memory_tracker/stores/gcstat_logfile_parser.rb",
     "lib/memory_tracker/stores/gcstat_logfile_store.rb",
     "lib/memory_tracker/stores/in_memory_store.rb",
     "lib/memory_tracker/stores/url_logfile_store.rb",
@@ -50,32 +51,11 @@ Gem::Specification.new do |s|
   s.licenses = ["MIT"]
   s.rubygems_version = "2.0.3"
 
-  if s.respond_to? :specification_version then
-    s.specification_version = 4
-
-    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
-      s.add_runtime_dependency(%q<sys-proctable>, [">= 0"])
-      s.add_development_dependency(%q<debugger>, [">= 0"])
-      s.add_development_dependency(%q<rspec>, ["~> 2.14.0"])
-      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
-      s.add_development_dependency(%q<bundler>, ["~> 1.0"])
-      s.add_dependency('activesupport')
-
-    else
-      s.add_dependency(%q<sys-proctable>, [">= 0"])
-      s.add_dependency(%q<debugger>, [">= 0"])
-      s.add_dependency(%q<rspec>, ["~> 2.14.0"])
-      s.add_dependency(%q<rdoc>, ["~> 3.12"])
-      s.add_dependency(%q<bundler>, ["~> 1.0"])
-      s.add_dependency('activesupport')
-
-    end
-  else
-    s.add_dependency(%q<sys-proctable>, [">= 0"])
-    s.add_dependency(%q<debugger>, [">= 0"])
-    s.add_dependency(%q<rspec>, ["~> 2.14.0"])
-    s.add_dependency(%q<rdoc>, ["~> 3.12"])
-    s.add_dependency(%q<bundler>, ["~> 1.0"])
-    s.add_dependency('activesupport')
-  end
+  s.add_runtime_dependency(%q<sys-proctable>, [">= 0"])
+  s.add_runtime_dependency(%q<logfile_interval>, ["~>1.1.0"])
+  s.add_runtime_dependency(%q<activesupport>)
+  s.add_development_dependency(%q<debugger>, [">= 0"])
+  s.add_development_dependency(%q<rspec>, ["~> 2.14.0"])
+  s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
+  s.add_development_dependency(%q<bundler>, ["~> 1.0"])
 end

data/spec/lib/stores/gcstat_logfile_parser_spec.rb

@@ -0,0 +1,50 @@
+require 'spec_helper'
+
+GCSTAT_LOGFILE_DATA = <<LINES
+1386125780,posts#index,49,0,63777,571,1431,349791,860,88.055808,1686477,1336686,269.484032
+1386125782,posts#show,50,0,108615,517,1431,325083,914,90.218496,1770275,1445192,271.48287999999997
+1386125784,posts#show,51,0,108361,513,1431,272900,918,90.759168,1826445,1553545,272.15871999999996
+1386125785,posts#index,52,0,103229,513,1431,287439,918,90.759168,1917742,1630303,272.15871999999996
+1386125787,posts#index,52,0,103386,513,1431,372228,918,91.29983999999999,2002688,1630460,272.78950399999997
+1386125789,posts#index,54,0,101731,495,1431,289801,936,92.577792,2109945,1820144,273.77664
+1386125792,posts#show,54,0,101731,495,1431,375282,936,92.577792,2195426,1820144,273.77664
+1386125794,posts#show,55,0,123765,441,1431,324209,990,94.949376,2267943,1943734,276.271104
+LINES
+
+module MemoryTracker
+  module Stores
+    describe GcstatLogfileParser do
+      before :each do
+        @lines = GCSTAT_LOGFILE_DATA.split("\n")
+        @parser = GcstatLogfileParser
+
+        expectation = LogfileInterval::Logfile.any_instance.should_receive(:each_line)
+        @lines.reverse.each { |line| expectation.and_yield(line) }
+      end
+
+      it 'iterates over each line' do
+        logfile_iterator = LogfileInterval::Logfile.new('foo', @parser)
+        result_lines = []
+        logfile_iterator.each_line do |line|
+          result_lines << line
+        end
+
+        result_lines.count.should == 8
+        result_lines.first.should == @lines.last
+        result_lines.last.should  == @lines.first
+      end
+
+      it 'iterates over each parsed line' do
+        logfile_iterator = LogfileInterval::Logfile.new('foo', @parser)
+        records = []
+        logfile_iterator.each_parsed_line do |record|
+          records << record
+        end
+
+        records.count.should == 8
+        records.first.count.should == @lines.last.split(',')[2].to_i
+        records.last.count.should  == @lines.first.split(',')[2].to_i
+      end
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -14,6 +14,7 @@ require 'memory_tracker/stores/base'
 require 'memory_tracker/stores/loader'
 require 'memory_tracker/stores/in_memory_store'
 require 'memory_tracker/stores/gcstat_logfile_store'
+require 'memory_tracker/stores/gcstat_logfile_parser'
 require 'memory_tracker/stores/url_logfile_store'
 
 # This file is copied to spec/ when you run 'rails generate rspec:install'

metadata

@@ -1,8 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: memory_tracker
 version: !ruby/object:Gem::Version
-  version: 1.1.1
-  prerelease: 
+  version: 1.2.0
 platform: ruby
 authors:
 - Philippe Le Rohellec
@@ -14,39 +13,62 @@ dependencies:
 - !ruby/object:Gem::Dependency
   name: sys-proctable
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: logfile_interval
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.1.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.1.0
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: debugger
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -54,7 +76,6 @@ dependencies:
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -62,7 +83,6 @@ dependencies:
 - !ruby/object:Gem::Dependency
   name: rdoc
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -70,7 +90,6 @@ dependencies:
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -78,7 +97,6 @@ dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -86,27 +104,10 @@ dependencies:
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.0'
-- !ruby/object:Gem::Dependency
-  name: activesupport
-  requirement: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ! '>='
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ! '>='
-      - !ruby/object:Gem::Version
-        version: '0'
 description: Collect and analyze memory usage data for each individual Rails action
   controller.
 email: philippe@lerohellec.com
@@ -135,6 +136,7 @@ files:
 - lib/memory_tracker/memory_tracker.rb
 - lib/memory_tracker/middleware.rb
 - lib/memory_tracker/request.rb
+- lib/memory_tracker/stores/gcstat_logfile_parser.rb
 - lib/memory_tracker/stores/gcstat_logfile_store.rb
 - lib/memory_tracker/stores/in_memory_store.rb
 - lib/memory_tracker/stores/url_logfile_store.rb
@@ -146,30 +148,30 @@ files:
 - spec/lib/stores/in_memory_store_spec.rb
 - spec/lib/stores/loader_spec.rb
 - spec/lib/stores/url_logfile_store_spec.rb
+- spec/lib/stores/gcstat_logfile_parser_spec.rb
 - spec/lib/request_spec.rb
 - spec/lib/memory_tracker_spec.rb
 homepage: http://github.com/plerohellec/memory_tracker
 licenses:
 - MIT
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.25
+rubygems_version: 2.0.3
 signing_key: 
 specification_version: 4
 summary: Rails memory allocations tracker
@@ -179,5 +181,6 @@ test_files:
 - spec/lib/stores/in_memory_store_spec.rb
 - spec/lib/stores/loader_spec.rb
 - spec/lib/stores/url_logfile_store_spec.rb
+- spec/lib/stores/gcstat_logfile_parser_spec.rb
 - spec/lib/request_spec.rb
 - spec/lib/memory_tracker_spec.rb