oozou-fusion_tables 0.2.3.dev.20110408163600

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/CHANGELOG

@@ -0,0 +1,6 @@
+== 0.2.2
+
+* 1.9.2 compatibility through gdata_19 gem
+* examples updated for 1.9.2
+* datetime columns now include time
+

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010 Tom Verbeure, Simon Tokumine
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.textile

@@ -0,0 +1,148 @@
+h1. fusion-tables
+
+This gem lets you easily interact with Google Fusion Tables from your Ruby application. Here is a "live visualisation of london bike hire availability":http://tables.googlelabs.com/DataSource?snapid=78314 and some "example maps and charts":http://www.tokumine.com/2010/08/10/fusion-tables-gem/.
+
+h2. Gem Dependencies
+
+* gdata_19 >= 1.1.2
+
+h2. Installation
+
+bc. gem install fusion_tables
+
+h2. Rubies
+
+Tested on:
+
+* 1.8.7
+* 1.9.2-p0
+
+h2. To Use
+
+bc. require 'fusion_tables'
+
+or in Rails 2.3.x
+
+bc. config.gem 'fusion_tables'
+
+h2. API examples
+ 
+"Twitter example":http://github.com/tokumine/fusion-tables/blob/master/examples/compare_tweets.rb
+"Boris bike example":http://github.com/tokumine/fusion-tables/blob/master/examples/boris_bikes.rb 
+"Tests":http://github.com/tokumine/fusion-tables/tree/master/test/
+
+Here is a brief rundown:
+
+<pre><code># Connect to service	
+@ft = GData::Client::FusionTables.new      
+@ft.clientlogin(username, password)
+
+# Browse existing tables
+@ft.show_tables
+ # => [table_1, table_2] 
+
+# Getting table id suitable for using with google maps (see more below)
+table_1.id #=> 42342 (the table's google id)
+
+# Count data
+table_1.count #=> 1
+
+# Select data
+table_1.select 
+ #=> data
+
+# Select data with conditions
+table_1.select "name", "WHERE x=n"
+ #=> data
+
+# Select ROWIDs
+row_ids = table_1.select "ROWID"
+
+# Drop tables
+@ft.drop table_1.id                    # table id
+@ft.drop [table_1.id, table_2.id]     # arrays of table ids
+@ft.drop /yacht/                      # regex on table name
+
+# Creating a table
+cols = [{:name => "friend name",    :type => 'string' },
+        {:name => "age",            :type => 'number' },
+        {:name => "meeting time",   :type => 'datetime' },
+        {:name => "where",          :type => 'location' }]
+
+new_table = @ft.create_table "My upcoming meetings", cols
+
+# Inserting rows (auto chunks every 500)
+data = [{"friend name" 	=> "Eric Wimp", 
+         "age"          => 25, 
+         "meeting time" => Time.utc(2010,"aug",10,20,15,1),
+         "where"        => "29 Acacia Road, Nuttytown"}]
+new_table.insert data
+
+# Delete row
+new_table.delete row_id
+</code></pre>
+
+h2. Fusion Tables secret Geospatial Sauce
+
+*"Geolocated Tweets example":http://tables.googlelabs.com/DataSource?snapid=73106*
+
+Fusion Tables is a labs product from Google. You can "read more here":http://tables.googlelabs.com/, but the key thing is that it gives you *access to the google tile mill for fast generation of google map layers across large datasets* 
+
+Fusion Tables supports the following geometry types:
+
+* lat/long
+* addresses (automatically geocodes them for you)
+* KML (point, polyline, polygon, multipolygon)
+
+h2. Integrate with google maps v3
+
+Adding a fusion tables datalayer with many points/polygons to your v3 map is as simple as:
+
+bc. layer = new google.maps.FusionTablesLayer(139529);
+
+That's it
+
+You can also refine the tiles by SQL, and can even do so dynamically:
+
+<pre><code>
+layer = new google.maps.FusionTablesLayer(198945, {
+  query: "SELECT address FROM 198945 WHERE ridership > 5000"}
+);
+</code></pre>
+
+Finally, fusion tables also lets you make Heatmaps
+
+<pre><code>
+layer = new google.maps.FusionTablesLayer(136705, {
+  heatmap: true
+});
+</code></pre>
+
+You can also export your data (filtered and geocoded) to KML. As an example, here are "all the Gasoline filling stations in the UK":http://tables.googlelabs.com/exporttable?query=select+col0%2Ccol1%2Ccol2%2Ccol3%2Ccol4%2Ccol5%2Ccol6%2Ccol12%2Ccol13%2Ccol14%2Ccol15%2Ccol16%2Ccol17%2Ccol18%2Ccol19%2Ccol20%2Ccol21+from+214045+&o=kmllink&g=col0
+
+read "more here":http://code.google.com/apis/maps/documentation/javascript/overlays.html#FusionTables
+
+h2. Known Issues
+
+# The gem uses the Google gdata_19 gem which conflicts with the GData2 gem. Uninstall gdata2 to regain sanity.
+# Currently you have to make a table public before you can display it on a map, unfortunately, this can only be done on the web interface. A suggested workaround is to put all your data in 1 big public table, and then query for the data you want to display based off a key/flag column
+
+h2. Note on Patches/Pull Requests
+ 
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+h2. Copyright
+
+Largely based on Tom Verbeure's work for MTBGuru: http://code.google.com/p/mtbguru-fusiontables/
+
+Copyright (c) 2010 Tom Verbeure, Simon Tokumine. See LICENSE for details.
+
+
+
+

data/Rakefile

@@ -0,0 +1,54 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "oozou-fusion_tables"
+    gem.summary = %Q{Google Fusion Tables API wrapper}
+    gem.description = %Q{A simple Google Fusion Tables API wrapper. Supports bulk inserts and most API functions}
+    gem.email = "simon@tinypla.net"
+    gem.homepage = "http://github.com/oozou/fusion-tables"
+    gem.authors = ["Simon Tokumine", "Tom Verbeure"]
+    gem.add_development_dependency "thoughtbot-shoulda", ">= 0"
+    gem.add_dependency "gdata_19", ">= 1.1.2"
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/test_*.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+task :test => :check_dependencies
+
+task :default => :test
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "fusion_tables #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/TODO

@@ -0,0 +1,11 @@
+Areas of improvement:
+
+* response format json/xml
+* "create table" should return name
+* describe table should include name
+* multiple batch statements for all actions (only insert now). Need on all other commands. Better SQL.
+* docs need updating, esp on response codes
+* response codes need to be more standardised
+* irritating can't include ROWID with * in selects
+* add columns
+* better heatmaps - scale is wonky

data/VERSION

@@ -0,0 +1 @@
+0.2.3.dev.20110408163600

data/examples/boris_bikes.rb

@@ -0,0 +1,159 @@
+# This library creates a FT and posts data from the Boris Bikes API to it every minute
+#
+# Add photos to infowindow
+# Add fusion table graphs to the inside of the infowindows too
+
+
+require 'net/http'
+require 'uri'
+require 'rubygems'
+require 'geo_ruby'
+require 'fusion_tables'
+require 'time'
+require 'json'
+require 'yaml'
+require 'ap'
+include GeoRuby
+include SimpleFeatures
+
+class Object
+  def try(method, *args, &block)
+    send(method, *args, &block)
+  end
+end
+
+def color max, number_to_color, min=0, opacity=80
+  color = ["FFFFB2", "FFFFB2", "FEB24C", "FD8D3C", "F03B20", "BD0026"]
+  #color = %w(FEE0D2 FCBBA1 FC9272 FB6A4A EF3B2C CB181D A50F15 67000D)
+  color.reverse!
+  #color = ["FFFFCC", "D9F0A3", "ADDD8E", "78C679", "31A354", "31A354"] #<- greens
+  chunk = (max-min)/color.size
+  index = (number_to_color/chunk).floor
+  "#{color[index]}#{opacity}"
+end
+
+def to_google(x,y)
+  a = `echo "#{x} #{y}" | cs2cs + +init=epsg:4326 +to +init=epsg:3785 -f "%.12f"`
+  a = a.split(" ")
+  {:x => a[0], :y => a[1]}
+end
+
+def from_google(x,y)
+  a = `echo "#{x} #{y}" | cs2cs + +init=epsg:3785 +to +init=epsg:4326 -f "%.12f"`
+  a = a.split(" ")
+  {:x => a[0], :y => a[1]}
+end
+
+
+def buffer(center_x, center_y, radius, quality = 4, precision = 12)
+  points = []
+  radians = Math::PI / 180
+  
+  coords = to_google(center_x, center_y)
+  center_x = coords[:x].to_f
+  center_y = coords[:y].to_f
+  
+  0.step(360, quality) do |i|
+    x = center_x + (radius * Math.cos(i * radians))
+    y = center_y + (radius * Math.sin(i * radians))
+    coords = from_google(x,y)
+    points << Point.from_x_y(round(coords[:x].to_f, precision), round(coords[:y].to_f, precision))
+  end
+  points
+end
+
+
+def round number, precision = 12
+  (number * 10**precision).round.to_f / 10**precision
+end  
+
+# Configure settings
+config = YAML::load_file(File.join(File.dirname(__FILE__), 'credentials.yml'))
+DEFAULT_SRID = 4328
+
+
+# Configure fusion tables
+ft = GData::Client::FusionTables.new
+ft.clientlogin(config["google_username"], config["google_password"])
+table_name = "Boris Bikes"
+cols = [
+  {:name => 'name',         :type => 'string'},  
+  {:name => 'created_at',   :type => 'datetime'},
+  {:name => 'updated_at',   :type => 'datetime'},
+  {:name => 'boris_id',     :type => 'number'},
+  {:name => 'temporary',    :type => 'number'},
+  {:name => 'installed',    :type => 'number'},
+  {:name => 'locked',       :type => 'number'},
+  {:name => 'nb_empty_docs',:type => 'number'},
+  {:name => 'nb_bikes',     :type => 'number'},
+  {:name => 'nb_docs',      :type => 'number'},
+  {:name => 'image',        :type => 'string'},
+  {:name => 'geom',         :type => 'location'},  
+  {:name => 'geom_fill',    :type => 'string'},  
+  {:name => 'geom_border',  :type => 'string'},  
+]
+
+# Create FT if it doesn't exist
+tables = ft.show_tables
+table  = tables.select{|t| t.name == table_name}.first
+table  = ft.create_table(table_name, cols) if !table
+
+while true do
+  bikes = JSON.parse(Net::HTTP.get(URI.parse('http://borisapi.heroku.com/stations.json')))  
+  
+  # get largest bike rack to calibrate buffer
+  max = 0
+  bikes.each do |b|
+    slots = b["nb_empty_docks"] + b["nb_bikes"]
+    max = slots if slots > max
+  end  
+  
+  # loop through data constructing fusion table date
+  data = []
+  max_radius = 150.0 #in meters
+  buffer_chunk = max_radius / max
+  
+  bikes.each do |b|
+    if b["lat"].to_f > 50 #ignore non geographic ones
+      docs = (b["nb_bikes"] + b["nb_empty_docks"])    
+      geom = Polygon.from_points [buffer(b["long"].to_f, b["lat"].to_f, docs*buffer_chunk)]    
+      #geom = Point.from_x_y b["long"].to_f, b["lat"].to_f
+      
+      data << {
+        "name"          => b["name"],
+        "created_at"    => Time::parse(b["created_at"]),
+        "updated_at"    => Time::parse(b["updated_at"]),
+        "boris_id"      => b["id"],
+        "temporary"     => (b["temporary"] ? 1 : 0), 
+        "installed"     => (b["installed"] ? 1 : 0),
+        "locked"        => (b["locked"]    ? 1 : 0),
+        "nb_empty_docs" => b["nb_empty_docks"],
+        "nb_bikes"      => b["nb_bikes"],
+        "nb_docs"       => docs,
+        "image"         => "",
+        "geom"          => geom.as_kml,
+        "geom_fill"     => color(max,b["nb_bikes"]),
+        "geom_border"   => color(max,b["nb_bikes"],0,"FF"),
+      }
+      puts "packing data for #{b["name"]}"
+    end  
+  end       
+  
+  # get current number of rows ready to delete
+  row_ids = table.select "ROWID"
+  
+  # put new data up
+  puts "sending bikes to fusion tables..."
+  table.insert data
+  
+  # remove old data
+  puts "deleting old rows"
+  row_ids.each do |id|
+    table.delete id[:rowid]
+  end
+    
+  # Be nice and wait
+  puts "...done! sleeping..."
+  sleep 500
+end
+

data/examples/compare_tweets.rb

@@ -0,0 +1,143 @@
+# encoding: UTF-8
+#
+# Twitter Fusion Tables Mashup
+# S.Tokumine 2010
+#
+# looks for tweets in the live stream around certain 
+# cities and posts them to fusion tables with KML attached
+#
+# Gem dependencies:
+#
+# tweetstream
+# GeoRuby
+# fusion_tables
+#
+# Output from running this for an evening
+# http://tables.googlelabs.com/DataSource?snapid=72509
+#
+require 'rubygems'
+require 'tweetstream'
+require 'geo_ruby'
+include GeoRuby
+include SimpleFeatures
+require 'fusion_tables'
+require 'time'
+require 'yaml'
+
+class Object
+  def try(method, *args, &block)
+    send(method, *args, &block)
+  end
+end
+
+# Configure settings
+config = YAML::load_file(File.join(File.dirname(__FILE__), 'credentials.yml'))
+DEFAULT_SRID = 4328
+
+# Twitter places
+places = {
+  :san_francisco => [-122.75,36.8,-121.75,37.8],
+  :new_york      => [-74,40,-73,41],
+  :tokyo         => [139.3,35,140.3,36],
+  :london        => [-0.54,51.2,0.46,52.2],
+  :madrid        => [-4.2,40,-3.2,41],
+  :paris         => [1.75,48.5,2.75, 49.5],
+  :beijing       => [115.9,39,116.9,40],
+  :mumbai        => [72.75,18.88,73.75,19.88],
+}
+
+# Configure fusion tables
+ft = GData::Client::FusionTables.new
+ft.clientlogin(config["google_username"], config["google_password"])
+table_name = "TwitterFusion"
+cols = [
+  {:name => 'screen_name',  :type => 'string'},  
+  {:name => 'avatar',       :type => 'string'},
+  {:name => 'text',         :type => 'string'},
+  {:name => 'created',      :type => 'datetime'},
+  {:name => 'url',          :type => 'string'},
+  {:name => 'location',     :type => 'location'},
+  {:name => 'iso',          :type => 'location'},
+  {:name => 'country_name', :type => 'location'},  
+  {:name => 'city',         :type => 'string'}
+]
+
+# Create FT if it doesn't exist
+tables = ft.show_tables
+table  = tables.select{|t| t.name == table_name}.first
+table  = ft.create_table(table_name, cols) if !table
+
+# Configure Twitter stream client
+data = []
+tw = TweetStream::Client.new(config["twitter_username"],config["twitter_password"])
+
+# configure friendly rate limit handling
+tw.on_limit do |skip_count|
+  sleep 5
+end 
+
+# start searching twitter stream and posting to FT
+tw.filter(:locations => places.values.join(",")) do |tweet|  
+  begin    
+
+    country = "unknown"
+    iso     = "unknown"    
+    begin
+      country = tweet.try(:[],:place).try(:[], :country)
+      iso = tweet.try(:[],:place).try(:[], :country_code)
+    rescue
+    end
+        
+    # Divine the tweets geometry
+    #
+    # overly complex due to 
+    # * some US tweets have their lat/longs flipped (but not all...)
+    # * some geo tweets are made using a "place" envelope rather than exact lat/kng      
+    if tweet[:geo]
+      if iso == 'US' && tweet[:geo][:coordinates][1] > 0
+        p = Point.from_x_y(tweet[:geo][:coordinates][0],tweet[:geo][:coordinates][1]) 
+      else
+        p = Point.from_x_y(tweet[:geo][:coordinates][1],tweet[:geo][:coordinates][0]) 
+      end  
+    else
+      p = Polygon.from_coordinates(tweet[:place][:bounding_box][:coordinates]).envelope.center
+    end
+  
+    # work out which city the tweet is from by testing with an extended bounding box
+    # BBox extention needed as twitter returns things outside our defined bboxes...
+    city = "unknown"
+    places.each do |key, value|
+      if !(p.x < value[0]-1 || p.x > value[2]+1 || p.y < value[1]-1 || p.y > value[3]+1)         
+        city = key.to_s.gsub("_"," ")    
+        break
+      end
+    end
+            
+    # pack data    
+    data << {
+      "screen_name"  => tweet[:user][:screen_name],
+      "avatar"       => tweet[:user][:profile_image_url],
+      "text"         => tweet[:text],
+      "created"      => Time.parse(tweet[:created_at]),
+      "url"          => "http://twitter.com/#{tweet[:user][:screen_name]}/status/#{tweet[:id]}",
+      "location"     => p.as_kml,
+      "iso"          => iso,
+      "country_name" => country,
+      "city"         => city
+    } 
+  rescue => e
+    puts "ERROR: #{e.inspect}, #{e.backtrace}"
+    #let sleeping dogs lie...
+  end  
+  
+  # let us know how we're doing
+  puts "#{50-data.size}: #{city}, #{tweet.text}"
+  
+  # Post to fusion tables  
+  if data.size == 50
+    puts "sending data to fusion tables..."
+    ft_data = data
+    data = []
+    table.insert ft_data
+  end      
+end