market_bot 0.6.3 → 0.6.4
Sign up to get free protection for your applications and to get access to all the features.
- data/README.markdown +6 -21
- data/VERSION +1 -1
- data/lib/market_bot/android/app.rb +3 -0
- data/lib/market_bot/android/leaderboard.rb +3 -0
- data/market_bot.gemspec +1 -1
- metadata +2 -2
data/README.markdown
CHANGED
@@ -1,6 +1,8 @@
|
|
1
|
-
# Market Bot -
|
1
|
+
# Market Bot - Ruby scraper for Google Play (Android Market)
|
2
2
|
|
3
3
|
Market Bot is a high performance Ruby scraper for Google's Android Market with a simple to use API.
|
4
|
+
Currently it supports scraping apps, leaderboards, and app searches.
|
5
|
+
Books, music, movies, etc aren't currently supported.
|
4
6
|
It is built on top of Nokogiri and Typhoeus.
|
5
7
|
Used in production to power [www.droidmeter.com](http://www.droidmeter.com/?t=github).
|
6
8
|
|
@@ -13,7 +15,7 @@ Used in production to power [www.droidmeter.com](http://www.droidmeter.com/?t=gi
|
|
13
15
|
|
14
16
|
gem install market_bot
|
15
17
|
|
16
|
-
##
|
18
|
+
## Simple API Examples
|
17
19
|
|
18
20
|
require 'rubygems'
|
19
21
|
require 'market_bot'
|
@@ -35,7 +37,7 @@ Used in production to power [www.droidmeter.com](http://www.droidmeter.com/?t=gi
|
|
35
37
|
sq.update
|
36
38
|
puts "Results found: #{sq.results.count}"
|
37
39
|
|
38
|
-
## Advanced Examples
|
40
|
+
## Advanced API Examples
|
39
41
|
|
40
42
|
require 'rubygems'
|
41
43
|
require 'market_bot'
|
@@ -61,23 +63,6 @@ Used in production to power [www.droidmeter.com](http://www.droidmeter.com/?t=gi
|
|
61
63
|
puts "First place app (#{first_app.title}) price: #{first_app.price}" unless first_app.error
|
62
64
|
puts "Last place app (#{last_app.title}) price: #{last_app.price}" unless last_app.error
|
63
65
|
|
64
|
-
## Benchmarks
|
65
|
-
|
66
|
-
Below are fairly typical results using 20 http workers (Typhoeus::Hydra.new(:max_concurrency => 20). The first benchmark downloads/parses a leaderboard containing 504 entries. The second benchmark takes those 504 entries, converts them to MarketBot::Android::App objects and then downloads/parses their details. Note that 20 workers are used to prevent DOSing the market servers. Higher values may work, but I don't recommend it.
|
67
|
-
|
68
|
-
$ rake benchmark:android
|
69
|
-
----------------------------------------------------
|
70
|
-
Benchmark Leaderboard: Top Selling Paid Apps
|
71
|
-
----------------------------------------------------
|
72
|
-
app count: 504
|
73
|
-
time: 3.16 seconds
|
74
|
-
|
75
|
-
----------------------------------------------------
|
76
|
-
Benchmark Apps: top Selling Paid Apps
|
77
|
-
----------------------------------------------------
|
78
|
-
app count: 504
|
79
|
-
time: 35.685 seconds
|
80
|
-
|
81
66
|
## Contributing to Market Bot
|
82
67
|
|
83
68
|
1. Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
|
@@ -90,6 +75,6 @@ Below are fairly typical results using 20 http workers (Typhoeus::Hydra.new(:max
|
|
90
75
|
|
91
76
|
## Copyright
|
92
77
|
|
93
|
-
Copyright (c) 2011 Chad Remesch. See LICENSE.txt for
|
78
|
+
Copyright (c) 2011 - 2012 Chad Remesch. See LICENSE.txt for
|
94
79
|
further details.
|
95
80
|
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.6.
|
1
|
+
0.6.4
|
@@ -145,6 +145,9 @@ module MarketBot
|
|
145
145
|
request = Typhoeus::Request.new(market_url)
|
146
146
|
|
147
147
|
request.on_complete do |response|
|
148
|
+
# HACK: Typhoeus <= 0.4.2 returns a response, 0.5.0pre returns the request.
|
149
|
+
response = response.response if response.is_a?(Typhoeus::Request)
|
150
|
+
|
148
151
|
result = nil
|
149
152
|
|
150
153
|
begin
|
@@ -131,6 +131,9 @@ module MarketBot
|
|
131
131
|
def process_page(url, page_num)
|
132
132
|
request = Typhoeus::Request.new(url)
|
133
133
|
request.on_complete do |response|
|
134
|
+
# HACK: Typhoeus <= 0.4.2 returns a response, 0.5.0pre returns the request.
|
135
|
+
response = response.response if response.is_a?(Typhoeus::Request)
|
136
|
+
|
134
137
|
result = Leaderboard.parse(response.body)
|
135
138
|
update_callback(result, page_num)
|
136
139
|
end
|
data/market_bot.gemspec
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: market_bot
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.6.
|
4
|
+
version: 0.6.4
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -240,7 +240,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
240
240
|
version: '0'
|
241
241
|
segments:
|
242
242
|
- 0
|
243
|
-
hash:
|
243
|
+
hash: 895662135504160235
|
244
244
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
245
245
|
none: false
|
246
246
|
requirements:
|