colinramsay-robotstxt 1.0.3 → 1.0.4
Sign up to get free protection for your applications and to get access to all the features.
- data/README +2 -1
- data/lib/robotstxtparser.rb +1 -1
- metadata +1 -1
data/README
CHANGED
@@ -16,6 +16,7 @@ Use it like this:
|
|
16
16
|
require 'robotstxtparser'
|
17
17
|
|
18
18
|
# Also accepts a local file
|
19
|
-
rp = RobotsTxtParser.new(
|
19
|
+
rp = RobotsTxtParser.new()
|
20
|
+
rp.read("http://something.com/robots.txt")
|
20
21
|
|
21
22
|
rp.user_agents('Google') # returns ["/logs", "/admin"]
|
data/lib/robotstxtparser.rb
CHANGED