googleplay_dev_scraper 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b6aefe94f964073e9d527321688dc59f193ee6de
+  data.tar.gz: 816f696b633e74030b30d283db45e972cc064cf0
+SHA512:
+  metadata.gz: 5450bde67ec8fe15a7574e7432f6153963da771f3ae6fdea3958709a488456313a35d8c42b4a8fee1c5ca92e54c98f9dda9c1e9097012926d33c858a35051ede
+  data.tar.gz: 6a82afc021d2a482db58286b4204d0234671512cf8a88d5e743eb425c2331dcd65cac04f9f0b830652453d88446cda2fe85cf70ec46e418cdd9287e834297fb7

data/.gitignore

@@ -0,0 +1,20 @@
+*~
+.idea/
+
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/ChangeLog

@@ -0,0 +1,40 @@
+2013/7/18 : Takuya Murakami
+
+	* ver 1.0.0
+	* gem name changed to 'googleplay_dev_scraper'
+	* use DateTime class for some api
+	* remove google checkout specific apis (payouts, order_detail, auto_deliver)
+
+2013/6/21 : Takuya Murakami
+
+	* add 'wallet_orders' command
+
+2013/3/30 : Takuya Murakami
+
+	* ver 0.2.2
+	* Fixed URL for sales report for v2 API.
+	* Refactoring
+
+2013/3/30 : Philipp.Sandhaus@cewecolor.de
+
+	* Fixed URL for app statistic download
+
+2012/12/13 : Takuya Murakami
+
+	* ver 0.2.1
+	* Use YAML for configuration file format.
+	* Refactoring
+
+2012/12/13 : Takuya Murakami
+
+	* ver 0.1.3
+	* Change project name : 'Google Play Scraper'
+	* Support gem
+
+2012/12/13 : Takuya Murakami
+
+	* Use Bundler to install mechanize
+
+2012/07/10 : Takuya Murakami
+
+	* Add support to download application statistics

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/README.ja.md

@@ -0,0 +1,172 @@
+Android 開発者向け Google Play / Google Wallet Scraper
+======================================================
+
+はじめに
+========
+
+このツールは、Google Play デベロッパーコンソール、
+および Google Wallet で提供される販売者向けの売上
+レポートなどの CSV ファイルを自動でダウンロードするための
+ツールです。
+
+Google Play デベロッパーコンソールからは以下のものをダウンロードできます。
+
+* 販売レポート
+* 予想販売レポート
+* アプリ統計情報
+
+Google Wallet Merchant Center からは以下のものをダウンロードできます。
+
+* オーダー一覧
+
+売上の集計をするなり、経理システムにぶち込むなり、お好きにどうぞ。
+
+
+必要システム
+============
+
+以下のものが必要です。
+
+* Ruby 1.9.3以上
+* RubyGems
+
+以下のようにしてインストールします。
+
+    $ gem install googleplay_dev_scraper
+
+
+設定
+====
+
+設定ファイルを ~/.googleplay_dev_scraper に YAML フォーマットで作成してください。
+以下にサンプルを示します。
+
+Google Play メールアドレスとパスワード、デベロッパIDを設定してください。
+(素のパスワードを設定するのでアクセス権には注意)
+
+デベロッパID は、developer console にログインした後の URL 末尾の
+dev_acc=... の数字です。
+
+```
+# GooglePlay dev scraper config file sample (YAML format)
+#
+# Place this content to your ~/.googleplay_dev_scraper or
+# ./.googleplay_dev_scraper.
+#
+# WARNING: This file contains password, be careful
+# of file permission.
+
+# Your E-mail address to login google play
+email: foo@example.com
+
+# Your password to login google play
+password: "Your Password"
+
+# Developer account ID
+# You can find your developer account ID in the URL 
+# after 'dev_acc=...' when login the developer console.
+dev_acc: "12345678901234567890"
+
+# Proxy host and port number (if needed) 
+#proxy_host: proxy.example.com
+#proxy_port: 8080
+```
+
+なお、設定値はコマンドラインで与えることもできます。詳細は
+--help オプションで確認してください。
+
+
+使い方
+======
+
+売上レポート取得
+----------------
+
+2011年10月の売上を取得する場合は以下のようにします。
+結果は標準出力に出力されます。
+
+    $ googleplay_dev_scraper sales 2011 10
+
+また推定売上レポートもダウンロードできます。
+
+    $ googleplay_dev_scraper estimated 2011 10
+
+
+オーダー一覧取得
+----------------
+
+オーダーの一覧を取得します。
+開始日と終了日を指定します。時刻は日本時間で指定。
+
+    $ googleplay_dev_scraper orders "2011-08-01 00:00:00" "2011-09-30 23:59:59"
+
+
+アプリケーション統計情報取得
+----------------------------
+
+Developer Console の統計情報 CSV エクスポートと同じものを得ます。
+対象となるアプリのパッケージ名と、開始日/終了日を指定してください。
+
+    $ googleplay_dev_scraper appstats your.package.name 20120101 20120630 > stat.zip
+
+ZIP ファイルが標準出力に出力されるので、リダイレクトでファイルに
+落としてください。
+
+
+API の利用
+==========
+
+例:
+
+```
+require 'googleplay_dev_scraper'
+
+scraper = GooglePlayDevScraper::Scraper.new
+
+# set config (Note: config file is not read via API access)
+scraper.config.email = "foo@example.com"
+scraper.config.password = "YOUR_PASSWORD"
+scraper.config.dev_acc = "1234567890"
+
+# get sales report / estimated sales report
+puts scraper.get_sales_report(2012, 11)
+puts scraper.get_estimated_sales_report(2012, 12)
+
+# get orders
+puts scraper.get_order_list(DateTime.parse("2012-11-01"), DateTime.parse("2012-11-30"))
+```
+
+内部動作とか
+============
+
+Mechanize を使って Web サイトに自動アクセスし、フォームを叩いて
+CSV を入手するだけです。
+
+本体は scraper.rb です。ソース見れば何やってるかはわかると思います。
+Rails アプリの中で使うとか、お好きにどうぞ。
+
+
+ライセンス
+==========
+
+Public domain 扱いとします。
+
+
+免責事項
+========
+
+* 無保証です。
+* Google 側のサイトの作りが変わったら当然動作しなくなります。
+* Google から怒られても責任は取りません。
+* 動かなくても文句言わない。自分で直すように。
+* 直したら修正を送るなり pull request するなりしてくれると嬉しい。
+
+
+ひとりごと
+==========
+
+* Google さん、Android 向けの Google Wallet API (オーダー一覧とか)解放してくれるとすごく嬉しいのですが、、、
+
+---
+'13/7/18
+Takuya Murakami, E-mail: tmurakam at tmurakam.org

data/README.md

@@ -0,0 +1,135 @@
+Google Play / Google Wallet Scraper for Android Developers
+==========================================================
+
+Introduction
+============
+
+This tool is designed to download CSV report files from
+Google Play developer console and google wallet.
+
+It can download following CSV files from Google Play
+developer console
+
+* Sales report (monthly report)
+* Estimateds sales report
+* Application statistics
+
+It can download following CSV files from Google Wallet.
+
+* Order list (almost realtime)
+
+You don't need any merchant key, because this tool scrapes
+google play / wallet website.
+
+Requirements/Installation
+=========================
+
+* Ruby >=1.9.3
+* RubyGems
+
+To install:
+
+    $ gem install googleplay_dev_scraper
+
+Configuration
+=============
+
+Create configuration file at ~/.googleplay_dev_scraper,
+or ./.googleplay_dev_scraper in YAML format.
+
+```
+# GooglePlay scraper config file sample (YAML format)
+#
+# Place this content to your ~/.googleplay_dev_scraper or
+# ./.googleplay_dev_scraper.
+#
+# WARNING: This file contains password, be careful
+# of file permission.
+
+# Your E-mail address to login google play
+email: foo@example.com
+
+# Your password to login google play
+password: "Your Password"
+
+# Developer account ID
+# You can find your developer account ID in the URL 
+# after 'dev_acc=...' when login the developer console.
+dev_acc: "12345678901234567890"
+
+# Proxy host and port number (if needed) 
+#proxy_host: proxy.example.com
+#proxy_port: 8080
+```
+
+You can specify configuration parameters with command line
+options. See details with --help option.
+
+How to use
+==========
+
+Get sales report
+----------------
+
+To download sales report for October 2011:
+
+    $ googleplay_dev_scraper sales 2011 10
+
+Or you can download estimated report too:
+
+    $ googleplay_dev_scraper estimated 2011 10
+
+Get order report
+----------------
+
+To download order report, specify start and end time as:
+
+    $ googleplay_dev_scraper orders "2011-08-01 00:00:00" "2011-09-30 23:59:59"
+
+Get application statistics
+--------------------------
+
+Export application statistics in CSV format.
+Specify application package name and start/end date.
+
+    $ googleplay_dev_scraper appstats your.package.name 20120101 20120630 > stat.zip
+
+Note: You must redirect output to zip file!
+
+API usage
+=========
+
+Example:
+
+```
+require 'googleplay_dev_scraper'
+
+scraper = GooglePlayDevScraper::Scraper.new
+
+# set config (Note: config file is not read via API access)
+scraper.config.email = "foo@example.com"
+scraper.config.password = "YOUR_PASSWORD"
+scraper.config.dev_acc = "1234567890"
+
+# get sales report / estimated sales report
+puts scraper.get_sales_report(2012, 11)
+puts scraper.get_estimated_sales_report(2012, 12)
+
+# get orders
+puts scraper.get_order_list(DateTime.parse("2012-11-01 00:00:00", DateTime.parse("2012-11-30T23:59:59"))
+```
+
+License
+=======
+
+Public domain
+
+
+Disclaimer
+==========
+
+* NO WARRANTY
+
+---
+'13/7/18
+Takuya Murakami, E-mail: tmurakam at tmurakam.org

data/Rakefile

@@ -0,0 +1,7 @@
+require "bundler/gem_tasks"
+
+task :default => :spec
+
+task :spec do
+  system "bundle exec rspec -c"
+end

data/bin/googleplay_dev_scraper

@@ -0,0 +1,122 @@
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+# design time...
+lib = File.expand_path('../lib', File.dirname(__FILE__))
+$LOAD_PATH.unshift(lib)
+
+require 'optparse'
+require 'googleplay_dev_scraper'
+
+Version = GooglePlayDevScraper::VERSION
+
+module GooglePlayDevScraper
+  class Tool
+    def initialize
+      @scraper = Scraper.new
+
+      @show_details = false
+      @auto_archive = false
+    end
+
+    def main
+      @scraper.config.load_config
+      parse_arguments
+
+      usage if ARGV.size < 1
+
+      command = ARGV.shift
+      case command
+      when "sales"
+        sales
+      when "estimated"
+        estimated
+      when "orders"
+        orders
+      when "appstats"
+        appstats
+      when "wallet_orders"
+        wallet_orders
+      else
+        usage
+      end
+    end
+
+    def usage
+      STDERR.puts "Usage: #{$0} [options] command [arguments...]"
+      STDERR.puts "       #{$0} --help"
+      STDERR.puts "  Commands:"
+      STDERR.puts "    sales <year> <month>             Get monthly sales report"
+      STDERR.puts "    estimated <year> <month>         Get estimated sales report"
+      STDERR.puts "    orders <start_date> <end_date>   Get order list"
+      STDERR.puts "    appstats <package_name> <startDay> <endDay>   Get app stats (in zip file)"
+      exit 1
+    end
+
+    def parse_arguments
+      config = @scraper.config
+
+      opts = OptionParser.new do |opt|
+        opt.on('-u email', 'set email address') {|v| config.email = v}
+        opt.on('-p password', 'set password') {|v| config.password = v}
+        opt.on('-a dev_acc', 'set dev_acc') {|v| config.dev_acc = v}
+
+        opt.on('-P host:port', 'Set HTTP proxy/port') {|v|
+          a = v.split(/:/)
+          config.proxy_host = a[0]
+          config.proxy_port = a[1]
+        }
+        opt.on('-v', '--version', 'Show version') {
+          puts Version
+          exit 0
+        }
+        opt.parse!(ARGV)
+      end
+    end
+
+    def sales
+      usage if (ARGV.size < 2)
+
+      year = ARGV[0]
+      month = ARGV[1]
+
+      puts @scraper.get_sales_report(year, month)
+    end
+
+    def estimated
+      usage if (ARGV.size < 2)
+
+      year = ARGV[0]
+      month = ARGV[1]
+
+      puts @scraper.get_estimated_sales_report(year, month)
+    end
+
+    # get daily orders
+    def orders
+      usage if (ARGV.size < 2)
+
+      startdate = DateTime.parse(ARGV[0])
+      enddate = DateTime.parse(ARGV[1])
+
+      puts @scraper.get_order_list(startdate, enddate)
+    end
+
+    def wallet_orders
+      puts @scraper.get_wallet_orders
+    end
+
+    def appstats
+      usage if ARGV.size < 3
+      package = ARGV[0]
+      startDay = ARGV[1]
+      endDay = ARGV[2]
+
+      puts @scraper.get_appstats(package, startDay, endDay)
+    end
+  end
+end
+
+tool = GooglePlayDevScraper::Tool.new
+tool.main
+

data/dot.googleplay_dev_scraper

@@ -0,0 +1,22 @@
+# GooglePlay scraper config file sample (YAML format)
+#
+# Place this content to your ~/.googleplay_dev_scraper or
+# ./.googleplay_dev_scraper.
+#
+# WARNING: This file contains password, be careful
+# of file permission.
+
+# Your E-mail address to login google play
+email: foo@example.com
+
+# Your password to login google play
+password: "Your Password"
+
+# Developer account ID
+# You can find your developer account ID in the URL 
+# after 'dev_acc=...' when login the developer console.
+dev_acc: "12345678901234567890"
+
+# Proxy host and port number (if needed) 
+#proxy_host: proxy.example.com
+#proxy_port: 8080

data/googleplay_dev_scraper.gemspec

@@ -0,0 +1,25 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'googleplay_dev_scraper/version'
+
+Gem::Specification.new do |gem|
+  gem.name          = "googleplay_dev_scraper"
+  gem.version       = GooglePlayDevScraper::VERSION
+  gem.authors       = ["Takuya Murakami"]
+  gem.email         = ["tmurakam@tmurakam.org"]
+  gem.description   = %q{Scraping and download CSV data from Google Play developer console and Google Wallet.}
+  gem.summary       = %q{Scraper for Google Play developer console and Google wallet}
+  gem.homepage      = "https://github.com/tmurakam/googleplay_dev_scraper"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+
+  gem.add_dependency('mechanize', '>= 2.5.0')
+
+  gem.add_development_dependency 'rspec'
+  gem.add_development_dependency 'rake'
+  gem.add_development_dependency 'rdoc'
+end

data/lib/googleplay_dev_scraper.rb

@@ -0,0 +1,4 @@
+require "googleplay_dev_scraper/version"
+require "googleplay_dev_scraper/scraper_base"
+require "googleplay_dev_scraper/scraper_config"
+require "googleplay_dev_scraper/scraper"

data/lib/googleplay_dev_scraper/scraper.rb

@@ -0,0 +1,172 @@
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+# = GooglePlay Scraper
+# Author:: Takuya Murakami
+# License:: Public domain
+
+require 'mechanize'
+require 'csv'
+require 'yaml'
+require 'date'
+
+module GooglePlayDevScraper
+  #
+  # Google Play and google checkout scraper
+  #
+  class Scraper < ScraperBase
+
+    def initialize
+      super
+    end
+
+    def body_string
+      @agent.page.body.force_encoding("UTF-8")
+    end
+
+    # Get sales report (report_type = payout_report)
+    # [year]
+    #   Year (ex. 2012)
+    # [month]
+    #   Month (1 - 12)
+    # [Return]
+    #   CSV string
+    #
+    def get_sales_report(year, month)
+      #url = sprintf('https://play.google.com/apps/publish/salesreport/download?report_date=%04d_%02d&report_type=payout_report&dev_acc=%s', year, month, @config.dev_acc)
+      url = sprintf('https://play.google.com/apps/publish/v2/salesreport/download?report_date=%04d_%02d&report_type=payout_report&dev_acc=%s', year, month, @config.dev_acc)
+      try_get(url)
+
+      body_string
+    end
+
+    # Get estimated sales report (report_type = sales_report)
+    #
+    # [year]
+    #   Year (ex. 2012)
+    # [month]
+    #   Month (1 - 12)
+    # [Return]
+    #   CSV string
+    #
+    def get_estimated_sales_report(year, month)
+      #https://play.google.com/apps/publish/v2/salesreport/download?report_date=2013_03&report_type=sales_report&dev_acc=09924472108471074593
+      url = sprintf('https://play.google.com/apps/publish/v2/salesreport/download?report_date=%04d_%02d&report_type=sales_report&dev_acc=%s', year, month, @config.dev_acc)
+      try_get(url)
+
+      body_string
+    end
+
+    # Get order list
+    #
+    # [start_date]
+    #   start time (DateTime)
+    # [end_date]
+    #   end time (DateTime)
+    # [Return]
+    #   CSV string
+    def get_order_list(start_time, end_time)
+      # unix time in ms
+      start_ut = start_time.to_time.to_i * 1000
+      end_ut = end_time.to_time.to_i * 1000
+
+      try_get("https://wallet.google.com/merchant/pages/")
+      if @agent.page.uri.path =~ /(bcid-[^\/]+)\/(oid-[^\/]+)\/(cid-[^\/]+)\//
+        bcid = $1
+        oid = $2
+        cid = $3
+
+        # You can check the URL with your browser.
+        # (download csv file, and check download history with chrome/firefox)
+        try_get("https://wallet.google.com/merchant/pages/" +
+                bcid + "/" + oid + "/" + cid +
+                "/purchaseorderdownload?startTime=#{start_ut}" + 
+                "&endTime=#{end_ut}")
+        body_string
+      end
+    end
+
+    # Get application statistics CSV in zip
+    #
+    # [package]
+    #   package name
+    # [start_day]
+    #   start date (yyyyMMdd)
+    # [end_day]
+    #   end date (yyyyMMdd)
+    # [Return]
+    #   application statics zip data
+    #
+    def get_appstats(package, start_day, end_day)
+      dim = "overall,country,language,os_version,device,app_version,carrier&met=active_device_installs,daily_device_installs,daily_device_uninstalls,daily_device_upgrades,active_user_installs,total_user_installs,daily_user_installs,daily_user_uninstalls,daily_avg_rating,total_avg_rating"
+      url = "https://play.google.com/apps/publish/v2/statistics/download"
+      url += "?package=#{package}"
+      url += "&sd=#{start_day}&ed=#{end_day}"
+      url += "&dim=#{dim}"
+      #url += "&dev_acc=#{@config.dev_acc}"
+
+      STDERR.puts "URL = #{url}"
+      try_get(url)
+      @agent.page.body
+    end
+
+    # dump CSV (util)
+    def dump_csv(csv_string)
+      headers = nil
+      CSV.parse(csv_string) do |row|
+        unless headers
+          headers = row
+          next
+        end
+
+        i = 0
+        row.each do |column|
+          puts "#{headers[i]} : #{column}"
+          i = i + 1
+        end
+        puts
+      end
+    end
+
+    #
+    # Get order list from wallet html page
+    #
+    def get_wallet_orders
+      try_get("https://wallet.google.com/merchant/pages/")
+      html = body_string
+
+      doc = Nokogiri::HTML(html)
+
+      #doc.xpath("//table[@id='purchaseOrderListTable']")
+
+      result = ""
+
+      doc.xpath("//tr[@class='orderRow']").each do |e|
+        order_id = e['id']
+
+        date = nil
+        desc = nil
+        total = nil
+        status = nil
+
+        e.children.each do |e2|
+          case e2['class']
+          when /wallet-date-column/
+            date = e2.content
+          when /wallet-description-column/
+            desc = e2.content
+          when /wallet-total-column/
+            total = e2.content
+          when /wallet-status-column/
+            e3 = e2.children.first
+            status = e3['title'] unless e3.nil?
+          end
+        end
+
+        result += [order_id, date, desc, status, total].join(",") + "\n"
+      end
+
+      result
+    end
+  end
+end

data/lib/googleplay_dev_scraper/scraper_base.rb

@@ -0,0 +1,66 @@
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+# = GooglePlay Scraper
+# Author:: Takuya Murakami
+# License:: Public domain
+
+require 'mechanize'
+require 'csv'
+require 'yaml'
+
+module GooglePlayDevScraper
+  class ScraperBase
+    attr_accessor :agent
+    attr_accessor :config
+
+    def initialize
+      @agent = nil
+      @config = ScraperConfig.new
+    end
+
+    def setup
+      #Mechanize.log = Logger.new("mechanize.log")
+      #Mechanize.log.level = Logger::INFO
+
+      unless @agent
+        @agent = Mechanize.new
+      end
+      if @config.proxy_host && @config.proxy_host.length >= 1
+        @agent.set_proxy(@config.proxy_host, @config.proxy_port)
+      end
+    end
+
+    def try_get(url)
+      unless @agent
+        setup
+      end
+
+      # try to get
+      @agent.get(url)
+
+      # login needed?
+      if @agent.page.uri.host != "accounts.google.com" || @agent.page.uri.path != "/ServiceLogin"
+        # already login-ed
+        return
+      end
+
+      # do login
+      form = @agent.page.forms.find {|f| f.form_node['id'] == "gaia_loginform"}
+      unless form
+        raise 'No login form'
+      end
+      form.field_with(:name => "Email").value = @config.email
+      form.field_with(:name => "Passwd").value = @config.password
+      form.click_button
+
+      if @agent.page.uri.host == "accounts.google.com"
+        STDERR.puts "login failed? : uri = " + @agent.page.uri.to_s
+        raise 'Google login failed'
+      end
+
+      # retry get
+      @agent.get(url)
+    end
+  end
+end

data/lib/googleplay_dev_scraper/scraper_config.rb

@@ -0,0 +1,71 @@
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+# = GooglePlay dev Scraper
+# Author:: Takuya Murakami
+# License:: Public domain
+
+require 'mechanize'
+require 'csv'
+require 'yaml'
+
+module GooglePlayDevScraper
+  #
+  # Configurations
+  #
+  class ScraperConfig
+    # Google account
+    attr_accessor :email
+
+    # Password to login google account
+    attr_accessor :password
+
+    # developer account ID
+    attr_accessor :dev_acc
+
+    # HTTP proxy host
+    attr_accessor :proxy_host
+
+    # HTTP proxy port
+    attr_accessor :proxy_port
+
+    def initialize
+      @dev_acc = nil
+    end
+
+    def load_config(path = nil)
+      config_files = [ path, ".googleplay_dev_scraper", "#{ENV['HOME']}/.googleplay_dev_scraper" ]
+
+      config_files.each do |file|
+        load_config_file(file)
+      end
+    end
+
+    def load_config_file(file)
+      if file && File.exists?(file)
+        open(file) do |f|
+          begin
+            read_config(f.read)
+          rescue Psych::SyntaxError => e
+            STDERR.puts "Error: configuration file syntax: #{file}"
+            exit 1
+          rescue
+            STDERR.puts "Error: load configuration file: #{file}"
+            exit 1
+          end
+        end
+      end
+    end
+
+    def read_config(data)
+      h = YAML.load(data)
+      if h
+        @email      ||= h['email']
+        @password   ||= h['password']
+        @dev_acc    ||= h['dev_acc']
+        @proxy_host ||= h['proxy_host']
+        @proxy_port ||= h['proxy_port']
+      end
+    end
+  end
+end

data/lib/googleplay_dev_scraper/version.rb

@@ -0,0 +1,3 @@
+module GooglePlayDevScraper
+  VERSION = "1.0.0"
+end

data/spec/scraper_config_spec.rb

@@ -0,0 +1,66 @@
+# -*- coding: utf-8 -*-
+require 'spec_helper'
+
+describe GooglePlayDevScraper::ScraperConfig do
+  before do
+    @config = GooglePlayDevScraper::ScraperConfig.new
+  end
+
+  context "read_config" do
+    it "read empty config" do
+      yaml = <<EOF
+# no data
+EOF
+      @config.read_config(yaml)
+
+      @config.email.should be_nil
+      @config.password.should be_nil
+      @config.dev_acc.should be_nil
+      @config.proxy_host.should be_nil
+      @config.proxy_port.should be_nil
+    end
+
+    it "read normal config" do
+      yaml = <<EOF
+email: EMAIL
+password: PASSWORD
+dev_acc: DEV_ACC
+proxy_host: PROXY_HOST
+proxy_port: PROXY_PORT
+EOF
+      @config.read_config(yaml)
+
+      @config.email.should == "EMAIL"
+      @config.password.should == "PASSWORD"
+      @config.dev_acc.should == "DEV_ACC"
+      @config.proxy_host.should == "PROXY_HOST"
+      @config.proxy_port.should == "PROXY_PORT"
+    end
+  end
+
+  context "load_config" do
+    before do 
+      # make mock
+      def @config.load_config_file(file)
+        @config_files ||= Array.new
+        @config_files.push(file)
+      end
+
+      def @config.config_files
+        @config_files
+      end
+    end
+
+    it "without path" do
+      @config.load_config
+      a = @config.config_files.should ==
+        [ nil, ".googleplay_dev_scraper", ENV['HOME'] + "/.googleplay_dev_scraper"]
+    end
+
+    it "with path" do
+      @config.load_config("/some/path")
+      a = @config.config_files.should ==
+        [ "/some/path", ".googleplay_dev_scraper", ENV['HOME'] + "/.googleplay_dev_scraper"]
+    end
+  end
+end

data/spec/scraper_spec.rb

@@ -0,0 +1,39 @@
+# -*- coding: utf-8 -*-
+require 'spec_helper'
+
+describe GooglePlayDevScraper::Scraper do
+  before do
+    @scraper = ScraperMock.new
+
+    @dev_acc = "1234567890"
+    @scraper.config.dev_acc = @dev_acc
+  end
+
+  context "Setup" do
+    it "setup without proxy" do
+      @scraper.setup
+
+      @scraper.agent.proxy_addr.should be_nil
+      @scraper.agent.proxy_port.should be_nil
+    end
+
+    it "setup with proxy" do
+      @scraper.config.proxy_host = "proxy.example.com"
+      @scraper.config.proxy_port = 12345
+
+      @scraper.setup
+
+      @scraper.agent.proxy_addr.should == "proxy.example.com"
+      @scraper.agent.proxy_port.should == 12345
+    end
+  end
+
+  context "get sales report" do
+    it "normal access" do
+      @scraper.get_sales_report(2012, 11)
+      @scraper.accessed_url.should == "https://play.google.com/apps/publish/v2/salesreport/download?report_date=2012_11&report_type=payout_report&dev_acc=#{@dev_acc}"
+    end
+  end
+  
+end
+

data/spec/spec_helper.rb

@@ -0,0 +1,34 @@
+# -*- encoding: utf-8 -*-
+require 'rubygems'
+require 'googleplay_dev_scraper'
+
+class ScraperMock < GooglePlayDevScraper::Scraper
+  attr_reader :accessed_url
+
+  def initialize
+    super
+    @agent = MechanizeMock.new
+  end
+
+  def try_get(url)
+    setup
+    @accessed_url = url
+  end
+end
+
+class MechanizeMock < Mechanize
+  def initialize
+    super
+    @page_mock = PageMock.new
+  end
+
+  def page
+    @page_mock
+  end
+end
+
+class PageMock
+  def body
+    return "BODY"
+  end
+end

metadata

@@ -0,0 +1,121 @@
+--- !ruby/object:Gem::Specification
+name: googleplay_dev_scraper
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Takuya Murakami
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-07-18 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: mechanize
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.5.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.5.0
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rdoc
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Scraping and download CSV data from Google Play developer console and
+  Google Wallet.
+email:
+- tmurakam@tmurakam.org
+executables:
+- googleplay_dev_scraper
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- ChangeLog
+- Gemfile
+- README.ja.md
+- README.md
+- Rakefile
+- bin/googleplay_dev_scraper
+- dot.googleplay_dev_scraper
+- googleplay_dev_scraper.gemspec
+- lib/googleplay_dev_scraper.rb
+- lib/googleplay_dev_scraper/scraper.rb
+- lib/googleplay_dev_scraper/scraper_base.rb
+- lib/googleplay_dev_scraper/scraper_config.rb
+- lib/googleplay_dev_scraper/version.rb
+- spec/scraper_config_spec.rb
+- spec/scraper_spec.rb
+- spec/spec_helper.rb
+homepage: https://github.com/tmurakam/googleplay_dev_scraper
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.5
+signing_key: 
+specification_version: 4
+summary: Scraper for Google Play developer console and Google wallet
+test_files:
+- spec/scraper_config_spec.rb
+- spec/scraper_spec.rb
+- spec/spec_helper.rb