geoptima 0.1.6 → 0.1.7

data/README.rdoc

@@ -1,6 +1,8 @@
 == Welcome to geoptima.rb
 
-Geoptima is a suite of applications for measuring and locating mobile/cellular subscriber experience on GPS enabled smartphones. It is produced by AmanziTel AB in Helsingborg, Sweden, and supports many phone manufacturers, with free downloads from the various app stores, markets or marketplaces. This Ruby library is only capable of reading the JSON format files priduced by these phones and reformating them as CSV for further analysis in Excel. This is a simple and independent way of analysing the data, when compared to the full-featured analysis applications and servers available from AmanziTel. If you want to analyse a limited amount of data in excel, or with Ruby, then this GEM might be for you. If you want to analyse large amounts of data, from many subscribers, or over long periods of time then rather consider the NetView and Customer IQ applications from AmanziTel at www.amanzitel.com.
+Geoptima is a suite of applications for measuring and locating mobile/cellular subscriber experience on GPS enabled smartphones. It is produced by AmanziTel AB in Helsingborg, Sweden, and supports many phone manufacturers, with free downloads from the various app stores, markets or marketplaces. This Ruby library is capable of reading the JSON format files produced by these phones and reformating them as CSV for further analysis in Excel. This is a simple and independent way of analysing the data, when compared to the full-featured analysis applications and servers available from AmanziTel. If you want to analyse a limited amount of data in excel, or with Ruby, then this GEM might be for you. If you want to analyse large amounts of data, from many subscribers, or over long periods of time then rather consider the NetView and Customer IQ applications from AmanziTel at www.amanzitel.com.
+
+More recent versions of this script, since about 0.1.0, have been able to perform statistics and generate charts. See below for the csv_chart script usage.
 
 For more information on Geoptima refer to:
 * {AmanziTel}[http://www.amanzitel.com] - for commercial applications and platforms supporting Geoptima
@@ -18,29 +20,34 @@ This script imports the JSON files on the command-line and then:
 * Optionally prints out all events on the console for visual inspection (using the -p option)
 * Optionally prints out all events to files in CSV format with merging of event types and fields into a single header for further processing with Excel or OpenOffice (using the -x option)
 * The CSV export mode has many further options for exporting to multiple files (-s), merging IMEI's into single dataset (-a), long headers with repeated information (-l), etc.
+* Configuration of the export format (static columns and header names) is possible with the -l and -M options
 
 Which event types to include and various other options are available using the command-line. Run it with the -h option to get a full list of options. The current version should support:
 
-  Usage: show_geoptima <-dpvxomlsah> <-L limit> <-E types> <-T min,max> file <files>
+  Usage: show_geoptima <-dwvpxomlsafh> <-P export_prefix> <-L limit> <-E types> <-T min,max> <-M mapfile> file <files>
   -d  debug mode (output more context during processing) (false)
+  -w  verbose mode (output extra information to console) 
+  -v  print geoptima library version 0.1.7
   -p  print mode (print out final results to console) 
-  -v  verbose mode (output extra information to console) 
   -x  export IMEI specific CSV files for further processing 
   -o  export field statistis 
   -m  map headers to classic NetView compatible version 
   -l  longer header list (phone and operator fields) 
   -s  seperate the export files by event type 
   -a  combine all IMEI's into a single dataset 
+  -f  flush stdout 
   -h  show this help
+  -P  prefix for exported files (default: ''; current: )
   -E  comma-seperated list of event types to show and export (default: all; current: )
   -T  time range to limit results to (default: all; current: )
   -L  limit verbose output to specific number of lines (10000)
+  -M  mapfile of normal->altered header names: 
 
 Currently the script also locates events that are close enough in time to GPS events. We hope to improve this with interpolation in the near future to be more compatible with the results from the commercial solutions. This time-window is also used for some of the extended header information, like LAC and CI, and effectively duplicates those fields from their own events to others. Take this into account when doing statistics on the results. It is better to use the original values, not the duplicates, if you want reliable statistics.
 
 ==== csv_stats
 
-This script reads any CSV file and outputs histograms of the values for each column. These can be manually imported into a spreadsheet or charting program for further analysis or charting.
+This script reads any CSV file and outputs histograms of the values for each column. These can be manually imported into a spreadsheet or charting program for further analysis or charting. This script is to a large extent superceded by the csv_chart script which both calculates stats as well as generates charts of the results.
 
 ==== csv_charts
 
@@ -120,9 +127,11 @@ https://github.com/craigtaverner/geoptima.rb/blob/master/images/Chart_All_Call%2
 === Installation
 
 Two options:
-* As a RubyGem: jruby -S gem install geoptima
+* As a RubyGem: gem install geoptima json
 * From source: git clone git@github.com:craigtaverner/geoptima.rb.git
 
+If you install without the json gem you will get a working library but not well optimized for JSON parsing. To install the json gem you need the minimal build environment for installing C code gems. On Linux and Mac this is usually already the case (or fix with 'apt-get install build-essential'). On windows you need to install the DevKit. See below for windows specific installation instructions.
+
 When installing the ruby gem you get the commands like 'show_geoptima' and 'csv_charts' in the path. However, if you install from source, you instead get them as examples/show_geoptima.rb.
 
   git clone git@github.com:craigtaverner/geoptima.rb.git
@@ -131,13 +140,23 @@ When installing the ruby gem you get the commands like 'show_geoptima' and 'csv_
 
 This should produce a file called 357841036600753.csv containing a CSV version of the Geoptima events for importing into Excel or OpenOffice.Calc for further processing. The number is the IMEI of the phone used, and if you process many JSON files in one go, you will get as many output files as there are phones represented by the data.
 
-The examples directory also contains a number of sample *.spec files for the csv_chart command.
+The examples directory also contains a number of sample *.spec files for the csv_chart command. In order for csv_chart to be able to generate charts, you need to also install a supported chart library. We have planned support for various charting libraries, but at the moment only support GRUFF. For installation of GRUFF, see below.
 
 ==== Installing on Windows
 
 For windows use the RubyInstaller at http://rubyinstaller.org/
 
-To get chart support working you will probably need to install the DevKit as described on https://github.com/oneclick/rubyinstaller/wiki/Development-Kit
+To get the optimized json gem installed, or to get chart support working you will need to install the DevKit as described on https://github.com/oneclick/rubyinstaller/wiki/Development-Kit. Please follow those instructions carefully as they are not entirely trivial.
+
+==== Running on Windows
+
+The best way to run on windows is to edit your system path (usually go to 'my computer->properties->advanced->environment), and add the path to your ruby bin directory (usually C:\Ruby\bin) to the end of the system PATH. After that, open a command-prompt (start->cmd.exe) and type 'ruby -v' to see that it worked. Then you can execute the 'gem install' commands and the 'show_geoptima' commands from this console.
+
+==== Installing Gruff for Charting
+
+In order to support charting, you need to install Gruff (see http://nubyonrails.com/pages/gruff). This depends on image-magick, so you need to install that first. Please refer to the installation instructions specific to your OS. On Ubuntu the following should  work:
+  apt-get install imagemagick libmagickcore-dev
+  gem install rmagick
 
 === Contributing
 

data/bin/csv_chart

@@ -10,7 +10,7 @@ require 'geoptima/options'
 require 'fileutils'
 require 'geoptima/daterange'
 
-Geoptima::assert_version("0.1.6")
+Geoptima::assert_version(">=0.1.6")
 Geoptima::Chart.available? || puts("No charting libraries available") || exit(-1)
 
 $export_dir = '.'

data/bin/csv_merge

@@ -9,7 +9,7 @@ require 'geoptima/options'
 require 'fileutils'
 require 'geoptima/daterange'
 
-Geoptima::assert_version("0.1.6")
+Geoptima::assert_version(">=0.1.6")
 
 $export_dir = '.'
 $export_name = 'merged.csv'

data/bin/geoptima_file_time

@@ -1,10 +1,14 @@
 #!/usr/bin/env ruby
 
-require 'date'
+# useful if being run inside a source code checkout
+$: << 'lib'
+$: << '../lib'
 
-$root=DateTime.parse("1970-01-01 00:00:00")
-ARGV.each do |arg|
-  base,seconds=arg.split(/_/)
-  date = ($root + seconds.to_f/(60*60*24))
-  puts "#{date}\t#{arg}"
+require 'geoptima/version'
+require 'geoptima/file_time'
+
+Geoptima::assert_version(">=0.1.7")
+
+ARGV.each do |filename|
+  puts "#{(filename.to_s+" "*40)[0..40]} -->   #{Geoptima::FileTime.from_file filename}"
 end

data/bin/show_geoptima

@@ -8,7 +8,7 @@ require 'date'
 require 'geoptima'
 require 'geoptima/options'
 
-Geoptima::assert_version("0.1.6")
+Geoptima::assert_version(">=0.1.6")
 
 $debug=false
 
@@ -18,7 +18,6 @@ $print_limit = 10000
 
 $files = Geoptima::Options.process_args do |option|
   option.p {$print = true}
-  option.v {$verbose = true}
   option.x {$export = true}
   option.s {$seperate = true}
   option.o {$export_stats = true}
@@ -104,13 +103,16 @@ def show_header_maps
   end
 end
 
+exit 0 if($print_version && !$verbose)
+
 $help = true if($files.length < 1)
 if $help
   puts <<EOHELP
-Usage: show_geoptima <-dpvxomlsah> <-L limit> <-E types> <-T min,max> <-M mapfile> file <files>
+Usage: show_geoptima <-dwvpxomlsafh> <-P export_prefix> <-L limit> <-E types> <-T min,max> <-M mapfile> file <files>
   -d  debug mode (output more context during processing) #{cw $debug}
+  -w  verbose mode (output extra information to console) #{cw $verbose}
+  -v  print geoptima library version #{Geoptima::VERSION}
   -p  print mode (print out final results to console) #{cw $print}
-  -v  verbose mode (output extra information to console) #{cw $verbose}
   -x  export IMEI specific CSV files for further processing #{cw $export}
   -o  export field statistis #{cw $export_stats}
   -m  map headers to classic NetView compatible version #{cw $map_headers}

data/examples/csv_chart.rb

@@ -10,7 +10,7 @@ require 'geoptima/options'
 require 'fileutils'
 require 'geoptima/daterange'
 
-Geoptima::assert_version("0.1.6")
+Geoptima::assert_version(">=0.1.6")
 Geoptima::Chart.available? || puts("No charting libraries available") || exit(-1)
 
 $export_dir = '.'

data/examples/csv_merge.rb

@@ -9,7 +9,7 @@ require 'geoptima/options'
 require 'fileutils'
 require 'geoptima/daterange'
 
-Geoptima::assert_version("0.1.6")
+Geoptima::assert_version(">=0.1.6")
 
 $export_dir = '.'
 $export_name = 'merged.csv'

data/examples/geoptima_file_time.rb

@@ -1,10 +1,14 @@
 #!/usr/bin/env ruby
 
-require 'date'
+# useful if being run inside a source code checkout
+$: << 'lib'
+$: << '../lib'
 
-$root=DateTime.parse("1970-01-01 00:00:00")
-ARGV.each do |arg|
-  base,seconds=arg.split(/_/)
-  date = ($root + seconds.to_f/(60*60*24))
-  puts "#{date}\t#{arg}"
+require 'geoptima/version'
+require 'geoptima/file_time'
+
+Geoptima::assert_version(">=0.1.7")
+
+ARGV.each do |filename|
+  puts "#{(filename.to_s+" "*40)[0..40]} -->   #{Geoptima::FileTime.from_file filename}"
 end

data/examples/show_geoptima.rb

@@ -8,7 +8,7 @@ require 'date'
 require 'geoptima'
 require 'geoptima/options'
 
-Geoptima::assert_version("0.1.6")
+Geoptima::assert_version(">=0.1.6")
 
 $debug=false
 
@@ -18,7 +18,6 @@ $print_limit = 10000
 
 $files = Geoptima::Options.process_args do |option|
   option.p {$print = true}
-  option.v {$verbose = true}
   option.x {$export = true}
   option.s {$seperate = true}
   option.o {$export_stats = true}
@@ -104,13 +103,16 @@ def show_header_maps
   end
 end
 
+exit 0 if($print_version && !$verbose)
+
 $help = true if($files.length < 1)
 if $help
   puts <<EOHELP
-Usage: show_geoptima <-dpvxomlsah> <-L limit> <-E types> <-T min,max> <-M mapfile> file <files>
+Usage: show_geoptima <-dwvpxomlsafh> <-P export_prefix> <-L limit> <-E types> <-T min,max> <-M mapfile> file <files>
   -d  debug mode (output more context during processing) #{cw $debug}
+  -w  verbose mode (output extra information to console) #{cw $verbose}
+  -v  print geoptima library version #{Geoptima::VERSION}
   -p  print mode (print out final results to console) #{cw $print}
-  -v  verbose mode (output extra information to console) #{cw $verbose}
   -x  export IMEI specific CSV files for further processing #{cw $export}
   -o  export field statistis #{cw $export_stats}
   -m  map headers to classic NetView compatible version #{cw $map_headers}

data/lib/geoptima/data.rb

@@ -12,7 +12,9 @@ module Geoptima
   SPERDAY = 60*60*24
   MSPERDAY = 1000*60*60*24
   SHORT = 256*256
-  MIN_DATETIME = DateTime.parse("2000-01-01")
+  MIN_VALID_DATETIME = DateTime.parse("1970-01-01")
+  MAX_VALID_DATETIME = DateTime.parse("2040-01-01")
+  MIN_DATETIME = DateTime.parse("2008-01-01")
   MAX_DATETIME = DateTime.parse("2040-01-01")
 
   class Config
@@ -141,10 +143,10 @@ module Geoptima
       start && start >= (Data.min_start-1) && start < Data.max_start
     end
     def self.min_start
-      @@min_start ||= MIN_DATETIME
+      @@min_start ||= MIN_VALID_DATETIME
     end
     def self.max_start
-      @@max_start ||= MAX_DATETIME
+      @@max_start ||= MAX_VALID_DATETIME
     end
     def events
       @events ||= make_events

data/lib/geoptima/daterange.rb

@@ -25,12 +25,12 @@ module Geoptima
       @range = Range.new(@min,@max)
     end
     if ENV['RUBY_VERSION'] =~ /1\.8/
-      puts "Defining Range.include? to wrap for 1.8"
+      puts "Defining Range.include? to wrap for 1.8" if($verbose)
       def include?(time)
         @range.include?(time)
       end
     else
-      puts "Defining Range.include? to perform inequality tests for 1.9"
+      puts "Defining Range.include? to perform inequality tests for 1.9" if($verbose)
       def include?(time)
         (time >= min) && (time <= @max)
       end

data/lib/geoptima/file_time.rb

@@ -0,0 +1,30 @@
+require 'date'
+
+module Geoptima
+  module FileTime
+    @@root=DateTime.parse("1970-01-01 00:00:00")
+    DAY_SECONDS = 24 * 60 * 60
+    DAY_MILLIS = DAY_SECONDS * 1000
+    HUNDRED_YEARS_SECONDS = 100 * 365 * DAY_SECONDS
+    HUNDRED_YEARS_MILLIS = HUNDRED_YEARS_SECONDS * 1000
+    def self.from_file(arg)
+      base,time=arg.to_s.split(/_/)
+      self.from(time)
+    end
+    def self.from(time)
+      time = time.to_f
+      ms = time > HUNDRED_YEARS_SECONDS
+      (@@root + time.to_f/(ms ? DAY_MILLIS : DAY_SECONDS))
+    end
+  end
+end
+
+if $PROGRAM_NAME =~ /\/file_time.rb$/
+  puts "Running test cases"
+  [123456789,1337089446603].each do |time|
+    puts "#{(time.to_s+" "*40)[0..40]} -->   #{Geoptima::FileTime.from time}"
+  end
+  ["356409048945284_1334764343.json","353491048201465_1337160798.json","3859F91B6B2C_1337097736981.json","\"724044021273460_1337093857605.txt\""].each do |filename|
+    puts "#{(filename.to_s+" "*40)[0..40]} -->   #{Geoptima::FileTime.from_file filename}"
+  end
+end

data/lib/geoptima/options.rb

@@ -47,6 +47,7 @@ module Geoptima
       options = Options.new(debug)
       options.add('f') {$flush_stdout = true}
       options.add('v') {$print_version = true}
+      options.add('w') {$verbose = true}
       options.add('d') {$debug = true}
       options.add('h') {$help = true}
       puts "Processing options: #{options}" if(debug)

data/lib/geoptima/version.rb

@@ -1,29 +1,88 @@
 module Geoptima
 
-  VERSION = "0.1.6"
+  VERSION = "0.1.7"
 
-  def self.version_as_int(ver)
-    base = 1
-    ver.split(/\./).reverse.inject(0) do |acc,v|
-      acc += base * v.to_i
-      base *= 100
-      acc
+  class Version
+    attr_reader :comparator, :version, :major, :minor, :patch
+    def initialize(text)
+      @comparator,@version = clean_version(text)
+      @major,@minor,@patch=@version.split(/\./)
     end
+    def clean_version(version)
+      comparator,cleaned = "==", version
+      if version =~ /^([\>\<\=]+)(\d+\.\d+\.\d+)/
+        comparator = $1[0..2]
+        cleaned = $2
+      end
+      [comparator,cleaned]
+    end
+    def to_i
+      unless @version_int
+        base = 1
+        @version_int = version.split(/\./).reverse.inject(0) do |acc,v|
+          acc += base * v.to_i
+          base *= 100
+          acc
+        end
+      end
+      @version_int
+    end
+    def to_s
+      version
+    end
+    def as_geoptima_version
+      self
+    end
+    def compare(other)
+      ogv = other.as_geoptima_version
+      self.to_i.send(ogv.comparator, ogv.to_i)
+    end
+    def diff(other)
+      other.as_geoptima_version.to_i - self.to_i
+    end
+  end
+
+  def self.version
+    @@version ||= VERSION.as_geoptima_version
   end
 
-  def self.compare_version(expected_ver)
-    version_as_int(expected_ver) - version_as_int(VERSION)
+  def self.version=(test_version)
+    @@version = test_version.as_geoptima_version
   end
 
-  def self.assert_version(expected_ver)
-    if expected_ver.to_s != VERSION
-      diff = compare_version(expected_ver)
-      if(diff != 0)
-        msg = diff > 0 ? "against and older library" : "an older script"
-        puts "Geoptima library version mismatch. Expected #{expected_ver}, found #{VERSION}. Are you running #{msg}?"
+  def self.assert_version(expected_ver, test_mode = false)
+    unless version.compare(expected_ver)
+      diff = version.diff(expected_ver)
+      if(!test_mode)
+        puts "Geoptima library version mismatch. Expected #{expected_ver}, found #{version}."
         exit -1
       end
+      return diff
     end
+    true
   end
 
 end
+
+class String
+  def as_geoptima_version
+    Geoptima::Version.new(self)
+  end
+end
+
+if $PROGRAM_NAME =~ /version.rb$/
+  test_lib_versions = ["0.1.5","0.1.6","0.1.7"]
+  test_comparators = ["","==","<","<=",">",">="]
+  test_lib_versions.each do |lib_ver|
+    Geoptima.version = lib_ver
+    puts "Testing with library version: #{lib_ver}"
+    puts (["Version"]+test_comparators.map{|v| "#{v}    "[0..3]}).join(" | ")
+    test_lib_versions.map do |test_ver|
+      puts "#{test_ver}   | "+(test_comparators.map do |comp|
+        expected_ver = "#{comp}#{test_ver}"
+        Geoptima.assert_version expected_ver, true
+      end.map{|v| "    #{v}"[-4..-1]}.join(" | "))
+    end
+  end
+end
+

data/lib/geoptima.rb

@@ -1,4 +1,5 @@
 require 'geoptima/version.rb'
 require 'geoptima/data.rb'
 require 'geoptima/options.rb'
+require 'geoptima/file_time.rb'
 

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: geoptima
 version: !ruby/object:Gem::Version 
-  hash: 23
+  hash: 21
   prerelease: 
   segments: 
   - 0
   - 1
-  - 6
-  version: 0.1.6
+  - 7
+  version: 0.1.7
 platform: ruby
 authors: 
 - Craig Taverner
@@ -15,8 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2012-04-23 00:00:00 +02:00
-default_executable: 
+date: 2012-05-18 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: multi_json
@@ -80,22 +79,22 @@ files:
 - lib/geoptima/version.rb
 - lib/geoptima/data.rb
 - lib/geoptima/chart.rb
-- lib/geoptima/options.rb
 - lib/geoptima/daterange.rb
+- lib/geoptima/options.rb
+- lib/geoptima/file_time.rb
 - lib/geoptima.rb
 - examples/show_geoptima_sos.rb
 - examples/show_geoptima.rb
 - examples/csv_chart.rb
+- examples/csv_merge.rb
 - examples/geoptima_file_time.rb
 - examples/csv_stats.rb
-- examples/csv_merge.rb
 - examples/sample_geoptima.json
 - README.rdoc
 - CHANGELOG
 - CONTRIBUTORS
 - Gemfile
 - geoptima.gemspec
-has_rdoc: true
 homepage: http://github.com/craigtaverner/geoptima.rb
 licenses: []
 
@@ -133,7 +132,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 
 rubyforge_project: geoptima
-rubygems_version: 1.6.2
+rubygems_version: 1.8.15
 signing_key: 
 specification_version: 3
 summary: Ruby access to Geoptima JSON files