geoptima 0.1.18 → 0.1.19

data/CONTRIBUTORS

@@ -2,4 +2,4 @@ Maintainer:
   Craig Taverner <craig at amanzi dot com>
 
 Contributors:
-* 
+* ... none yet ...

data/Gemfile

@@ -1,5 +1,3 @@
-source :gemcutter
-
 gemspec
 
 group 'test' do
@@ -10,6 +8,10 @@ group 'test' do
   gem "test-unit"
 end
 
+gem "multi_json", ">= 1.1.0"
+gem "json_pure", ">= 1.6.5"
+gem "json", ">= 1.6.5"
+
 #gem 'ruby-debug-base19' if RUBY_VERSION.include? "1.9"
 #gem 'ruby-debug-base' if RUBY_VERSION.include? "1.8"
 #gem "ruby-debug-ide"

data/README.rdoc

@@ -24,10 +24,11 @@ This script imports the JSON files on the command-line and then:
 
 Which event types to include and various other options are available using the command-line. Run it with the -h option to get a full list of options. The current version should support:
 
-  Usage: show_geoptima <-dwvpxomlsafh> <-P export_prefix> <-L limit> <-E types> <-T min,max> <-M mapfile> file <files>
+  Usage: show_geoptima <-dwvpxomlsafegh> <-P export_prefix> <-L limit>
+                     <-E types> <-T min,max> <-M mapfile> file <files>
   -d  debug mode (output more context during processing) (false)
   -w  verbose mode (output extra information to console) 
-  -v  print geoptima library version 0.1.7
+  -v  print geoptima library version 0.1.19
   -p  print mode (print out final results to console) 
   -x  export IMEI specific CSV files for further processing 
   -o  export field statistis 
@@ -36,12 +37,67 @@ Which event types to include and various other options are available using the c
   -s  seperate the export files by event type 
   -a  combine all IMEI's into a single dataset 
   -f  flush stdout 
+  -e  show error statistics 
+  -g  export GPX traces 
+  -t  split time colum to multiple columns 
   -h  show this help
   -P  prefix for exported files (default: ''; current: )
   -E  comma-seperated list of event types to show and export (default: all; current: )
+  -A  application category map file (default: app names)
   -T  time range to limit results to (default: all; current: )
+  -B  location limited to specified bounds in one of these formats:
+        minlat,minlon,maxlat,maxlon
+        minlat..maxlat,minlon..maxlon
+        DIST(distance_in_km,lat,lon)
+        RANGE[minlat,minlon,maxlat,maxlon]
+        RANGE[minlat..maxlat,minlon..maxlon]
+      (default: all; current: )
   -L  limit verbose output to specific number of lines (10000)
   -M  mapfile of normal->altered header names: 
+  -G  GPX export options as ';' separated list of key:value pairs
+      Current GPX options: {"scale"=>190, "padding"=>5, "limit"=>2,
+      "png_limit"=>10, "points"=>true, "point_size"=>2, "point_color"=>"auto"}
+  -X  Geolocation options as ';' separated list of key:value pairs
+      Current geolocation options: {"algorithm"=>"window", "window"=>60}
+  
+  The GPX and Geolocation options require futher explanation:
+  
+  Known supported GPX options (might be more, see data.rb code):
+  limit:2     	    Limit GPX output to traces with at least this number of events
+  png_limit:10	    Limit PNG output to traces with at least this number of events
+  merge:     	    Merge all traces into a single trace
+  only_merge:	    Do not export unmerged traces
+  scale:190	    Size of print area in PNG output
+  padding:5	    Space around print area
+  points:true	    Turn on/off points
+  point_size:2	    Set point size
+  point_color:auto  Set point color: RRGGBBAA in hex (else 'auto')
+  format:     	    Export format: 'gpx', 'csv', 'png', default 'all'
+  waypoints:	    Export waypoints for events: <event_type>, default 'all'
+  
+  PNG images will be 'scale + 2 * padding' big (200 for current settings).
+  The scale will be used for the widest dimension, and the other will be reduced
+  to fit the actual size of the trace. No projection is used, with the points
+  simply mapped to their GPS locations. This will cause visual distortions far
+  from the equator where dlat!=dlon.
+  
+  Known supported geolocation options (might be more, see data.rb code):
+  algorithm:window  Which geolocation algorithm to use
+  window:60	    Time window in seconds, has slightly different
+                    meanings for different algorithms
+  
+  Currently supported algorithms:
+  window:      select GPS point within window seconds of event,
+               GPS points after the event take priority.
+               (this is the default for geoptima GEM version >= 0.1.19)
+  +win:        select only GPS points after event (within time window)
+  -win:        select only GPS points before event (within time window)
+               (this is the default for geoptima GEM version < 0.1.19)
+  closest:     select closest GPS point within window seconds of event
+               (similar to window option, but chooses closest)
+  interpolate: Linear interpolation between two closest points
+               (experimental, do not use yet)
+               (read redmine wiki page for explanation of algorithm)
 
 Currently the script also locates events that are close enough in time to GPS events. We hope to improve this with interpolation in the near future to be more compatible with the results from the commercial solutions. This time-window is also used for some of the extended header information, like LAC and CI, and effectively duplicates those fields from their own events to others. Take this into account when doing statistics on the results. It is better to use the original values, not the duplicates, if you want reliable statistics.
 
@@ -150,7 +206,7 @@ To get the optimized json gem installed, or to get chart support working you wil
 
 ==== Running on Windows
 
-The best way to run on windows is to edit your system path (usually go to 'my computer->properties->advanced->environment), and add the path to your ruby bin directory (usually C:\Ruby\bin) to the end of the system PATH. After that, open a command-prompt (start->cmd.exe) and type 'ruby -v' to see that it worked. Then you can execute the 'gem install' commands and the 'show_geoptima' commands from this console.
+The best way to run on windows is to edit your system path (usually go to 'my computer->properties->advanced->environment), and add the path to your ruby bin directory (usually C:\\Ruby\\bin) to the end of the system PATH. After that, open a command-prompt (start->cmd.exe) and type 'ruby -v' to see that it worked. Then you can execute the 'gem install' commands and the 'show_geoptima' commands from this console.
 
 ==== Installing Gruff for Charting
 

data/bin/show_geoptima

@@ -8,7 +8,7 @@ require 'date'
 require 'geoptima'
 require 'geoptima/options'
 
-Geoptima::assert_version(">=0.1.18")
+Geoptima::assert_version(">=0.1.19")
 
 $debug=false
 
@@ -21,6 +21,18 @@ $gpx_options = {
   'points' => true, 'point_size' => 2, 'point_color' => 'auto'
 }
 
+$geolocation_options = {
+  'algorithm' => 'window', 'window' => 60
+}
+
+def make_hash_options(arg)
+  arg.split(/[\,\;]+/).inject({}) do |a,v|
+    k=v.split(/[\:\=]+/)
+    a[k[0]]=k[1]||true
+    a
+  end
+end
+
 $files = Geoptima::Options.process_args do |option|
   option.p {$print = true}
   option.x {$export = true}
@@ -38,8 +50,9 @@ $files = Geoptima::Options.process_args do |option|
   option.B {$location_range = Geoptima::LocationRange.from ARGV.shift}
   option.L {$print_limit = [1,ARGV.shift.to_i].max}
   option.M {$mapfile = ARGV.shift}
-  option.G {$gpx_options.merge! ARGV.shift.split(/[\,\;]+/).inject({}){|a,v| k=v.split(/[\:\=]+/);a[k[0]]=k[1]||true;a}}
+  option.G {$gpx_options.merge! make_hash_options(ARGV.shift)}
   option.A {$app_categories = Geoptima::AppCategories.new(ARGV.shift)}
+  option.X {$geolocation_options.merge! make_hash_options(ARGV.shift)}
 end.map do |file|
   File.exist?(file) ? file : puts("No such file: #{file}")
 end.compact
@@ -116,10 +129,33 @@ end
 
 exit 0 if($print_version && !$verbose)
 
+class String
+  def wrappad(max=80,pad=8)
+    max = [max,pad+1].max
+    wrapped = [self.to_s]
+    while wrapped[-1].length > max
+      a,b = wrapped[-1][0..(max-1)],wrapped[-1][max..-1]
+      if (si = a.rindex(/\s+/)) && (a[0..(si-1)] =~ /\S/)
+        b = [a[si..-1],b].join.gsub(/^\s+/,'')
+        a = a[0..(si-1)]
+      else
+        a+='-'
+      end
+      a = a.gsub(/\s+$/,'')
+      b = b.gsub(/^\s+/,'')
+      wrapped[-1] = a
+      wrapped << [' '*pad,b].join
+      puts "WRAP[#{max}:#{pad}]: #{wrapped.inspect}"
+    end
+    wrapped
+  end
+end
+
 $help = true if($files.length < 1)
 if $help
   puts <<EOHELP
-Usage: show_geoptima <-dwvpxomlsafegh> <-P export_prefix> <-L limit> <-E types> <-T min,max> <-M mapfile> file <files>
+Usage: show_geoptima <-dwvpxomlsafegh> <-P export_prefix> <-L limit>
+                     <-E types> <-T min,max> <-M mapfile> file <files>
   -d  debug mode (output more context during processing) #{cw $debug}
   -w  verbose mode (output extra information to console) #{cw $verbose}
   -v  print geoptima library version #{Geoptima::VERSION}
@@ -149,20 +185,47 @@ Usage: show_geoptima <-dwvpxomlsafegh> <-P export_prefix> <-L limit> <-E types>
   -L  limit verbose output to specific number of lines #{cw $print_limit}
   -M  mapfile of normal->altered header names: #{$mapfile}
   -G  GPX export options as ';' separated list of key:value pairs
-      Current GPX options: #{$gpx_options.inspect}
-      Known supported GPX options (might be more, see data.rb code):
-        limit:#{$gpx_options['limit']}\t\tLimit GPX output to traces with at least this number of events
-        png_limit:#{$gpx_options['png_limit']}\t\tLimit PNG output to traces with at least this number of events
-        merge:#{$gpx_options['merge']}\t\tMerge all traces into a single trace
-        only_merge:#{$gpx_options['merge']}\t\tDo not export unmerged traces
-        scale:#{$gpx_options['scale']}\t\tSize of print area in PNG output
-        padding:#{$gpx_options['padding']}\t\tSpace around print area
-        points:#{$gpx_options['points']}\t\tTurn on/off points
-        point_size:#{$gpx_options['point_size']}\t\tSet point size
-        point_color:#{$gpx_options['point_color']}\tSet point color: RRGGBBAA in hex (else 'auto')
-        format:#{$gpx_options['format']}\t\tExport format: 'gpx', 'csv', 'png', default 'all'
-        waypoints:#{$gpx_options['waypoints']}\t\tExport waypoints for events: <event_type>, default 'all'
-      PNG images will be 'scale + 2 * padding' big (#{$gpx_options['scale'].to_i+2*$gpx_options['padding'].to_i} for current settings). The scale will be used for the widest dimension, and the other will be reduced to fit the actual size of the trace. The projection used is non, with the points simply mapped to their GPS locations. This will cause visual distortions far from the equator where dlat!=dlon.
+#{('      Current GPX options: '+$gpx_options.inspect).wrappad(80,6).join("\n")}
+  -X  Geolocation options as ';' separated list of key:value pairs
+#{('      Current geolocation options: '+$geolocation_options.inspect).wrappad(80,6).join("\n")}
+
+The GPX and Geolocation options require futher explanation:
+
+Known supported GPX options (might be more, see data.rb code):
+  limit:#{$gpx_options['limit']}     \t    Limit GPX output to traces with at least this number of events
+  png_limit:#{$gpx_options['png_limit']}\t    Limit PNG output to traces with at least this number of events
+  merge:#{$gpx_options['merge']}     \t    Merge all traces into a single trace
+  only_merge:#{$gpx_options['merge']}\t    Do not export unmerged traces
+  scale:#{$gpx_options['scale']}\t    Size of print area in PNG output
+  padding:#{$gpx_options['padding']}\t    Space around print area
+  points:#{$gpx_options['points']}\t    Turn on/off points
+  point_size:#{$gpx_options['point_size']}\t    Set point size
+  point_color:#{$gpx_options['point_color']}  Set point color: RRGGBBAA in hex (else 'auto')
+  format:#{$gpx_options['format']}     \t    Export format: 'gpx', 'csv', 'png', default 'all'
+  waypoints:#{$gpx_options['waypoints']}\t    Export waypoints for events: <event_type>, default 'all'
+PNG images will be 'scale + 2 * padding' big (#{$gpx_options['scale'].to_i+2*$gpx_options['padding'].to_i} for current settings).
+The scale will be used for the widest dimension, and the other will be reduced
+to fit the actual size of the trace. No projection is used, with the points
+simply mapped to their GPS locations. This will cause visual distortions far
+from the equator where dlat!=dlon.
+
+Known supported geolocation options (might be more, see data.rb code):
+algorithm:#{$geolocation_options['algorithm']}    Which geolocation algorithm to use
+window:#{$geolocation_options['window']}\t    Time window in seconds, has slightly different
+                    meanings for different algorithms
+
+Currently supported algorithms:
+  window:      select GPS point within window seconds of event,
+               GPS points after the event take priority.
+               (this is the default for geoptima GEM version >= 0.1.19)
+  +win:        select only GPS points after event (within time window)
+  -win:        select only GPS points before event (within time window)
+               (this is the default for geoptima GEM version < 0.1.19)
+  closest:     select closest GPS point within window seconds of event
+               (similar to window option, but chooses closest)
+  interpolate: Linear interpolation between two closest points
+               (experimental, do not use yet)
+               (read redmine wiki page for explanation of algorithm)
 EOHELP
   show_header_maps
   exit 0
@@ -171,7 +234,13 @@ end
 $verbose = $verbose || $debug
 show_header_maps if($verbose)
 
-$datasets = Geoptima::Dataset.make_datasets($files, :locate => true, :time_range => $time_range, :location_range => $location_range, :combine_all => $combine_all)
+$datasets = Geoptima::Dataset.make_datasets($files,
+  :locate => true,
+  :time_range => $time_range,
+  :location_range => $location_range,
+  :geolocation_options => $geolocation_options,
+  :combine_all => $combine_all
+)
 
 class Export
   attr_reader :files, :imei, :names, :headers

data/examples/show_geoptima.rb

@@ -8,7 +8,7 @@ require 'date'
 require 'geoptima'
 require 'geoptima/options'
 
-Geoptima::assert_version(">=0.1.18")
+Geoptima::assert_version(">=0.1.19")
 
 $debug=false
 
@@ -21,6 +21,18 @@ $gpx_options = {
   'points' => true, 'point_size' => 2, 'point_color' => 'auto'
 }
 
+$geolocation_options = {
+  'algorithm' => 'window', 'window' => 60
+}
+
+def make_hash_options(arg)
+  arg.split(/[\,\;]+/).inject({}) do |a,v|
+    k=v.split(/[\:\=]+/)
+    a[k[0]]=k[1]||true
+    a
+  end
+end
+
 $files = Geoptima::Options.process_args do |option|
   option.p {$print = true}
   option.x {$export = true}
@@ -38,8 +50,9 @@ $files = Geoptima::Options.process_args do |option|
   option.B {$location_range = Geoptima::LocationRange.from ARGV.shift}
   option.L {$print_limit = [1,ARGV.shift.to_i].max}
   option.M {$mapfile = ARGV.shift}
-  option.G {$gpx_options.merge! ARGV.shift.split(/[\,\;]+/).inject({}){|a,v| k=v.split(/[\:\=]+/);a[k[0]]=k[1]||true;a}}
+  option.G {$gpx_options.merge! make_hash_options(ARGV.shift)}
   option.A {$app_categories = Geoptima::AppCategories.new(ARGV.shift)}
+  option.X {$geolocation_options.merge! make_hash_options(ARGV.shift)}
 end.map do |file|
   File.exist?(file) ? file : puts("No such file: #{file}")
 end.compact
@@ -116,10 +129,33 @@ end
 
 exit 0 if($print_version && !$verbose)
 
+class String
+  def wrappad(max=80,pad=8)
+    max = [max,pad+1].max
+    wrapped = [self.to_s]
+    while wrapped[-1].length > max
+      a,b = wrapped[-1][0..(max-1)],wrapped[-1][max..-1]
+      if (si = a.rindex(/\s+/)) && (a[0..(si-1)] =~ /\S/)
+        b = [a[si..-1],b].join.gsub(/^\s+/,'')
+        a = a[0..(si-1)]
+      else
+        a+='-'
+      end
+      a = a.gsub(/\s+$/,'')
+      b = b.gsub(/^\s+/,'')
+      wrapped[-1] = a
+      wrapped << [' '*pad,b].join
+      puts "WRAP[#{max}:#{pad}]: #{wrapped.inspect}"
+    end
+    wrapped
+  end
+end
+
 $help = true if($files.length < 1)
 if $help
   puts <<EOHELP
-Usage: show_geoptima <-dwvpxomlsafegh> <-P export_prefix> <-L limit> <-E types> <-T min,max> <-M mapfile> file <files>
+Usage: show_geoptima <-dwvpxomlsafegh> <-P export_prefix> <-L limit>
+                     <-E types> <-T min,max> <-M mapfile> file <files>
   -d  debug mode (output more context during processing) #{cw $debug}
   -w  verbose mode (output extra information to console) #{cw $verbose}
   -v  print geoptima library version #{Geoptima::VERSION}
@@ -149,20 +185,47 @@ Usage: show_geoptima <-dwvpxomlsafegh> <-P export_prefix> <-L limit> <-E types>
   -L  limit verbose output to specific number of lines #{cw $print_limit}
   -M  mapfile of normal->altered header names: #{$mapfile}
   -G  GPX export options as ';' separated list of key:value pairs
-      Current GPX options: #{$gpx_options.inspect}
-      Known supported GPX options (might be more, see data.rb code):
-        limit:#{$gpx_options['limit']}\t\tLimit GPX output to traces with at least this number of events
-        png_limit:#{$gpx_options['png_limit']}\t\tLimit PNG output to traces with at least this number of events
-        merge:#{$gpx_options['merge']}\t\tMerge all traces into a single trace
-        only_merge:#{$gpx_options['merge']}\t\tDo not export unmerged traces
-        scale:#{$gpx_options['scale']}\t\tSize of print area in PNG output
-        padding:#{$gpx_options['padding']}\t\tSpace around print area
-        points:#{$gpx_options['points']}\t\tTurn on/off points
-        point_size:#{$gpx_options['point_size']}\t\tSet point size
-        point_color:#{$gpx_options['point_color']}\tSet point color: RRGGBBAA in hex (else 'auto')
-        format:#{$gpx_options['format']}\t\tExport format: 'gpx', 'csv', 'png', default 'all'
-        waypoints:#{$gpx_options['waypoints']}\t\tExport waypoints for events: <event_type>, default 'all'
-      PNG images will be 'scale + 2 * padding' big (#{$gpx_options['scale'].to_i+2*$gpx_options['padding'].to_i} for current settings). The scale will be used for the widest dimension, and the other will be reduced to fit the actual size of the trace. The projection used is non, with the points simply mapped to their GPS locations. This will cause visual distortions far from the equator where dlat!=dlon.
+#{('      Current GPX options: '+$gpx_options.inspect).wrappad(80,6).join("\n")}
+  -X  Geolocation options as ';' separated list of key:value pairs
+#{('      Current geolocation options: '+$geolocation_options.inspect).wrappad(80,6).join("\n")}
+
+The GPX and Geolocation options require futher explanation:
+
+Known supported GPX options (might be more, see data.rb code):
+  limit:#{$gpx_options['limit']}     \t    Limit GPX output to traces with at least this number of events
+  png_limit:#{$gpx_options['png_limit']}\t    Limit PNG output to traces with at least this number of events
+  merge:#{$gpx_options['merge']}     \t    Merge all traces into a single trace
+  only_merge:#{$gpx_options['merge']}\t    Do not export unmerged traces
+  scale:#{$gpx_options['scale']}\t    Size of print area in PNG output
+  padding:#{$gpx_options['padding']}\t    Space around print area
+  points:#{$gpx_options['points']}\t    Turn on/off points
+  point_size:#{$gpx_options['point_size']}\t    Set point size
+  point_color:#{$gpx_options['point_color']}  Set point color: RRGGBBAA in hex (else 'auto')
+  format:#{$gpx_options['format']}     \t    Export format: 'gpx', 'csv', 'png', default 'all'
+  waypoints:#{$gpx_options['waypoints']}\t    Export waypoints for events: <event_type>, default 'all'
+PNG images will be 'scale + 2 * padding' big (#{$gpx_options['scale'].to_i+2*$gpx_options['padding'].to_i} for current settings).
+The scale will be used for the widest dimension, and the other will be reduced
+to fit the actual size of the trace. No projection is used, with the points
+simply mapped to their GPS locations. This will cause visual distortions far
+from the equator where dlat!=dlon.
+
+Known supported geolocation options (might be more, see data.rb code):
+algorithm:#{$geolocation_options['algorithm']}    Which geolocation algorithm to use
+window:#{$geolocation_options['window']}\t    Time window in seconds, has slightly different
+                    meanings for different algorithms
+
+Currently supported algorithms:
+  window:      select GPS point within window seconds of event,
+               GPS points after the event take priority.
+               (this is the default for geoptima GEM version >= 0.1.19)
+  +win:        select only GPS points after event (within time window)
+  -win:        select only GPS points before event (within time window)
+               (this is the default for geoptima GEM version < 0.1.19)
+  closest:     select closest GPS point within window seconds of event
+               (similar to window option, but chooses closest)
+  interpolate: Linear interpolation between two closest points
+               (experimental, do not use yet)
+               (read redmine wiki page for explanation of algorithm)
 EOHELP
   show_header_maps
   exit 0
@@ -171,7 +234,13 @@ end
 $verbose = $verbose || $debug
 show_header_maps if($verbose)
 
-$datasets = Geoptima::Dataset.make_datasets($files, :locate => true, :time_range => $time_range, :location_range => $location_range, :combine_all => $combine_all)
+$datasets = Geoptima::Dataset.make_datasets($files,
+  :locate => true,
+  :time_range => $time_range,
+  :location_range => $location_range,
+  :geolocation_options => $geolocation_options,
+  :combine_all => $combine_all
+)
 
 class Export
   attr_reader :files, :imei, :names, :headers

data/geoptima.gemspec

@@ -12,13 +12,39 @@ Gem::Specification.new do |s|
   s.rubyforge_project = 'geoptima'
   s.summary = "Ruby access to Geoptima JSON files"
   s.description = <<-EOF
-Geoptima is a suite of applications for measuring and locating mobile/cellular subscriber experience on GPS enabled smartphones.
-It is produced by AmanziTel AB in Helsingborg, Sweden, and supports many phone manufacturers, with free downloads from the
-various app stores, markets or marketplaces. This Ruby library is only capable of reading the JSON format files priduced by these phones
-and reformating them as CSV for further analysis in Excel. This is a simple and independent way of analysing the data, when
-compared to the full-featured analysis applications and servers available from AmanziTel. If you want to analyse a limited amount
-of data in excel, or with Ruby, then this GEM might be for you. If you want to analyse large amounts of data, from many subscribers, or over long periods of time
-then rather consider the NetView and Customer IQ applications from AmanziTel at www.amanzitel.com.
+
+Geoptima is a suite of applications for measuring and locating
+mobile/cellular subscriber experience on GPS enabled smartphones.  It is
+produced by AmanziTel AB in Helsingborg, Sweden, and supports many phone
+manufacturers, with free downloads from the various app stores, markets or
+marketplaces.  This Ruby library is capable of reading the JSON format files
+produced by these phones and reformating them as CSV, GPX and PNG for
+further analysis in Excel.  This is a simple and independent way of
+analysing the data, when compared to the full-featured analysis applications
+and servers available from AmanziTel.  If you want to analyse a limited
+amount of data in excel, or with Ruby, then this GEM might be for you.  If
+you want to analyse large amounts of data, from many subscribers, or over
+long periods of time then rather consider the NetView and Customer IQ
+applications from AmanziTel at www.amanzitel.com.
+
+Current features available in the library and the show_geoptima command:
+* Import one or many JSON files
+* Organize data by device id (IMEI) into datasets
+* Split by event type
+* Time ordering and time correlation (associate data from one event to another):
+** Add GPS locations to other events (time window and interpolation algorithms)
+** Add signal strenth, battery level, etc. to other events
+* Export event tables to CSV format for further processing in excel
+* Make and export GPS traces in GPX and PNG format for simple map reports
+
+The amount of data possible to process is limited by memory, since all data
+is imported in ruby data structures for procssing.  If you need to process
+larger amounts of data, you will need a database-driven approach, like that
+provided by AmanziTel's NetView and Customer IQ solutions.  This Ruby gem is
+actually used by parts of the data pre-processing chain of 'Customer IQ',
+but it not used by the main database and statistics engine that generates
+the reports.
+
 EOF
 
   s.require_path = 'lib'

data/lib/geoptima/after.ports

@@ -0,0 +1,17 @@
+tcp        0      0 127.0.0.1:53748         0.0.0.0:*               LISTEN     
+tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN     
+tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN     
+tcp        0      0 127.0.0.1:34678         0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
+tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:17500           0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:60648           0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN     
+tcp6       0      0 127.0.0.1:7473          :::*                    LISTEN     
+tcp6       0      0 127.0.0.1:7474          :::*                    LISTEN     
+tcp6       0      0 :::22                   :::*                    LISTEN     
+tcp6       0      0 ::1:631                 :::*                    LISTEN     
+tcp6       0      0 :::3000                 :::*                    LISTEN     
+tcp6       0      0 :::1337                 :::*                    LISTEN     
+tcp6       0      0 :::34495                :::*                    LISTEN     

data/lib/geoptima/before.ports

@@ -0,0 +1,16 @@
+tcp        0      0 127.0.0.1:53748         0.0.0.0:*               LISTEN     
+tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN     
+tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN     
+tcp        0      0 127.0.0.1:34678         0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
+tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:17500           0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:60648           0.0.0.0:*               LISTEN     
+tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN     
+tcp6       0      0 127.0.0.1:7473          :::*                    LISTEN     
+tcp6       0      0 127.0.0.1:7474          :::*                    LISTEN     
+tcp6       0      0 :::22                   :::*                    LISTEN     
+tcp6       0      0 ::1:631                 :::*                    LISTEN     
+tcp6       0      0 :::1337                 :::*                    LISTEN     
+tcp6       0      0 :::34495                :::*                    LISTEN     

data/lib/geoptima/data.rb

@@ -4,6 +4,7 @@ require 'rubygems'
 require 'multi_json'
 require 'geoptima/daterange'
 require 'geoptima/locationrange'
+require 'geoptima/locator'
 require 'geoptima/timer'
 begin
   require 'png'
@@ -16,8 +17,6 @@ end
 #
 module Geoptima
 
-  SPERDAY = 60*60*24
-  MSPERDAY = 1000*60*60*24
   SHORT = 256*256
   MIN_VALID_DATETIME = DateTime.parse("1970-01-01")
   MAX_VALID_DATETIME = DateTime.parse("2040-01-01")
@@ -484,7 +483,8 @@ module Geoptima
     }
 
     include ErrorCounter
-    attr_reader :file, :header, :name, :data, :fields, :time, :latitude, :longitude, :timeoffset
+    include Locatable
+    attr_reader :file, :gps, :header, :name, :data, :fields, :time, :latitude, :longitude, :timeoffset
     def initialize(file,start,name,header,data,previous=nil)
       @file = file
       @name = name
@@ -553,14 +553,12 @@ module Geoptima
     def location
       @location ||= latitude && Point.new(latitude,longitude)
     end
-    def locate(gps)
+    def set_location(gps)
       incr_error "GPS String Data" if(gps['latitude'].is_a? String)
       @latitude = gps['latitude'].to_f
       @longitude = gps['longitude'].to_f
       @location = nil
-    end
-    def locate_if_closer_than(gps,seconds=60)
-      locate(gps) if(closer_than(gps,seconds))
+      @gps = gps
     end
     def puts line
       Kernel.puts "#{name}[#{time}]: #{line}"
@@ -868,6 +866,7 @@ module Geoptima
       @options = options
       @time_range = options[:time_range] || DateRange.new(Config[:min_datetime],Config[:max_datetime])
       @location_range = options[:location_range] || LocationRange.everywhere
+      @geolocation_options = options[:geolocation_options] || {}
       @fields = {}
     end
 
@@ -1098,20 +1097,14 @@ module Geoptima
     end
 
     def locate_events
-      prev_gps = nil
-      count = 0
       puts "Locating #{sorted.length} events" if(true||$debug)
-      sorted.each do |event|
-        timer('locate.each').start
-        if event.name === 'gps'
-          event.locate(event)
-          prev_gps = event
-        elsif prev_gps
-          count += 1 if(event.locate_if_closer_than(prev_gps,60))
-        end
-        timer('locate.each').stop
+      locator = Geoptima::Locator.new self.sorted, @geolocation_options
+      timer("locate.all").start
+      locator.locate
+      timer("locate.all").stop
+      if (true||$debug)
+        puts "Located #{locator.located.length} / #{sorted.length} events (timed: #{timer("locate.all")}"
       end
-      puts "Located #{count} / #{sorted.length} events" if($debug)
     end
 
     def to_s

data/lib/geoptima/locator.rb

@@ -0,0 +1,172 @@
+#!/usr/bin/env ruby
+
+require 'geoptima/locationrange'
+
+module Geoptima
+
+  SPERDAY = 60*60*24
+  MSPERDAY = 1000*SPERDAY
+
+  class LocatorAlgorithm
+    def locate(locatable)
+      if locatable.next_point
+        locatable.location = locatable.next_point
+      elsif locatable.previous_point
+        locatable.location = locatable.previous_point
+      else
+        locatable.location = nil
+      end
+    end
+  end
+  class BeforeLocatorAlgorithm < LocatorAlgorithm
+    def locate(locatable)
+      if locatable.previous_point
+        locatable.location = locatable.previous_point
+      else
+        locatable.location = nil
+      end
+    end
+  end
+  class AfterLocatorAlgorithm < LocatorAlgorithm
+    def locate(locatable)
+      if locatable.next_point
+        locatable.location = locatable.next_point
+      else
+        locatable.location = nil
+      end
+    end
+  end
+  class ClosestLocatorAlgorithm < LocatorAlgorithm
+    def locate(locatable)
+      results = [
+        [locatable.next_point_gap,locatable.next_point],
+        [locatable.previous_point_gap,locatable.previous_point]
+      ].reject do |x|
+        x[0].nil? && x[1].nil?
+      end.sort do |a,b|
+        a[0] <=> b[0]
+      end[0]
+      locatable.location = results && results[1]
+    end
+  end
+  class InterpolationLocatorAlgorithm < ClosestLocatorAlgorithm
+    def locate(locatable)
+      if locatable.previous_poing && locatable.next_point && locatable.next_point.prev_point
+        correlateEvent2Point(point,wavg(point.prev,point.next,point),'interpolated');
+      elsif closest = super.locate(locatable)
+        correlateEvent2Point(point,closest,'correlated');
+      else
+        puts "No correlation possible for point: "+point
+      end
+    end
+  end
+  module Locatable
+    attr_accessor :previous_point, :next_point, :previous_point_gap, :next_point_gap, :location
+    def closer_than(gps,window=0.0)
+      if $debug && gps && window > 0
+        puts "Comparing times:"
+        puts "\tGPS: #{gps.time}"
+        puts "\tEvent: #{self.time}"
+        puts "\tTDiff: #{(self - gps).abs.to_f}"
+        puts "\tWindow: #{window}"
+      end
+      gps && (window <= 0.0 || (self - gps).abs < window)
+    end
+    def set_next_if(gps,time_window=0.0)
+      if closer_than(gps,time_window)
+        self.next_point = gps.location
+        self.next_point_gap = (self - gps).abs
+      end
+    end
+    def set_previous_if(gps,time_window=0.0)
+      if closer_than(gps,time_window)
+        self.previous_point = gps.location
+        self.previous_point_gap = (self - gps).abs
+      end
+    end
+  end
+  class LocatableImpl
+    attr_reader :name, :attributes
+    attr_accessor :time
+    include Locatable
+    def initialize(attributes)
+      @attributes = Hash[*attributes.map{|k,v| [k.to_s,v]}.flatten]
+      @name = @attributes['name']
+      @time = @attributes['time']
+    end
+    def [](key)
+      @attributes[key.to_s] || @attributes[key.to_s.gsub(/#{name}\./,'')]
+    end
+    def []=(key,value)
+      @attributes[key.to_s] = value
+    end
+    def -(other)
+      (self.time - other.time) * SPERDAY
+    end
+    def to_s
+      @attributes.inspect
+    end
+  end
+  class Locator
+    attr_reader :sorted, :located, :failed, :options
+    def initialize(sorted, options={})
+      @sorted = sorted
+      @options = Hash[*options.map{|k,v| [k.to_s.intern,v]}.flatten]
+      @options[:algorithm] ||= 'window'
+      @options[:window] ||= 60
+      puts "Initialized geo-location on #{@sorted.length} events with options: #{@options.inspect}" if($debug)
+    end
+    def start
+      @start ||= sorted[0] && sorted[0][:time] || DateTime.now
+    end
+    def algorithm
+      @algorithm ||= case @options[:algorithm].to_s
+      when /^\-win/
+        BeforeLocatorAlgorithm.new
+      when /^\+win/
+        AfterLocatorAlgorithm.new
+      when /closest/
+        ClosestLocatorAlgorithm.new
+      when /inter/
+        InterpolationLocatorAlgorithm.new
+      else
+        LocatorAlgorithm.new
+      end
+    end
+    def locate
+      gps = nil
+      @located = []
+      @failed = []
+      locatables = []
+      time_window = @options[:window].to_i
+      time_window = 60 if(time_window<1)
+      puts "Locating within window[#{time_window}] using algorithm:#{algorithm.class}" if($debug)
+      sorted.each do |event|
+        event.time ||= start + event[:timestamp].to_f / Geoptima::MSPERDAY
+        if event.name === 'gps'
+          gps = event
+          puts "Setting GPS location point: #{gps.inspect}" if($debug)
+          gps.location = Point.new(gps['latitude'].to_f, gps['longitude'].to_f)
+          locatables.each do |event|
+            event.set_next_if(gps,time_window)
+          end
+          locatables = []
+        else
+          event.set_previous_if(gps,time_window)
+          locatables << event
+          @located << event
+        end
+      end
+      @located = @located.map do |l|
+        if self.algorithm.locate(l)
+          l
+        else
+          @failed << l
+          nil
+        end
+      end.compact
+    end
+  end
+end
+
+

data/lib/geoptima/version.rb

@@ -1,6 +1,6 @@
 module Geoptima
 
-  VERSION = "0.1.18"
+  VERSION = "0.1.19"
 
   class Version
     attr_reader :comparator, :version, :major, :minor, :patch

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: geoptima
 version: !ruby/object:Gem::Version
-  version: 0.1.18
+  version: 0.1.19
   prerelease: 
 platform: ruby
 authors:
@@ -9,14 +9,14 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-01-07 00:00:00.000000000 Z
+date: 2013-05-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: multi_json
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: 1.1.0
   type: :runtime
@@ -24,7 +24,7 @@ dependencies:
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: 1.1.0
 - !ruby/object:Gem::Dependency
@@ -32,7 +32,7 @@ dependencies:
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: 1.6.5
   type: :runtime
@@ -40,31 +40,43 @@ dependencies:
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: 1.6.5
-description: ! 'Geoptima is a suite of applications for measuring and locating mobile/cellular
-  subscriber experience on GPS enabled smartphones.
+description: |2+
 
-  It is produced by AmanziTel AB in Helsingborg, Sweden, and supports many phone manufacturers,
-  with free downloads from the
+  Geoptima is a suite of applications for measuring and locating
+  mobile/cellular subscriber experience on GPS enabled smartphones.  It is
+  produced by AmanziTel AB in Helsingborg, Sweden, and supports many phone
+  manufacturers, with free downloads from the various app stores, markets or
+  marketplaces.  This Ruby library is capable of reading the JSON format files
+  produced by these phones and reformating them as CSV, GPX and PNG for
+  further analysis in Excel.  This is a simple and independent way of
+  analysing the data, when compared to the full-featured analysis applications
+  and servers available from AmanziTel.  If you want to analyse a limited
+  amount of data in excel, or with Ruby, then this GEM might be for you.  If
+  you want to analyse large amounts of data, from many subscribers, or over
+  long periods of time then rather consider the NetView and Customer IQ
+  applications from AmanziTel at www.amanzitel.com.
 
-  various app stores, markets or marketplaces. This Ruby library is only capable of
-  reading the JSON format files priduced by these phones
+  Current features available in the library and the show_geoptima command:
+  * Import one or many JSON files
+  * Organize data by device id (IMEI) into datasets
+  * Split by event type
+  * Time ordering and time correlation (associate data from one event to another):
+  ** Add GPS locations to other events (time window and interpolation algorithms)
+  ** Add signal strenth, battery level, etc. to other events
+  * Export event tables to CSV format for further processing in excel
+  * Make and export GPS traces in GPX and PNG format for simple map reports
 
-  and reformating them as CSV for further analysis in Excel. This is a simple and
-  independent way of analysing the data, when
+  The amount of data possible to process is limited by memory, since all data
+  is imported in ruby data structures for procssing.  If you need to process
+  larger amounts of data, you will need a database-driven approach, like that
+  provided by AmanziTel's NetView and Customer IQ solutions.  This Ruby gem is
+  actually used by parts of the data pre-processing chain of 'Customer IQ',
+  but it not used by the main database and statistics engine that generates
+  the reports.
 
-  compared to the full-featured analysis applications and servers available from AmanziTel.
-  If you want to analyse a limited amount
-
-  of data in excel, or with Ruby, then this GEM might be for you. If you want to analyse
-  large amounts of data, from many subscribers, or over long periods of time
-
-  then rather consider the NetView and Customer IQ applications from AmanziTel at
-  www.amanzitel.com.
-
-'
 email: craig@amanzi.com
 executables:
 - show_geoptima
@@ -90,8 +102,11 @@ files:
 - lib/geoptima/daterange.rb
 - lib/geoptima/options.rb
 - lib/geoptima/timer.rb
+- lib/geoptima/before.ports
 - lib/geoptima/file_time.rb
 - lib/geoptima/locationrange.rb
+- lib/geoptima/after.ports
+- lib/geoptima/locator.rb
 - lib/geoptima.rb
 - examples/show_geoptima_sos.rb
 - examples/show_geoptima.rb
@@ -111,30 +126,33 @@ homepage: http://github.com/craigtaverner/geoptima.rb
 licenses: []
 post_install_message: 
 rdoc_options:
-- --quiet
-- --title
+- "--quiet"
+- "--title"
 - Geoptima.rb
-- --line-numbers
-- --main
+- "--line-numbers"
+- "--main"
 - README.rdoc
-- --inline-source
+- "--inline-source"
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: 1.8.6
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: 909654323
 requirements: []
 rubyforge_project: geoptima
-rubygems_version: 1.8.24
+rubygems_version: 1.8.25
 signing_key: 
 specification_version: 3
 summary: Ruby access to Geoptima JSON files