wayback 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.yardopts +10 -0
- data/CHANGELOG.md +3 -0
- data/CONTRIBUTING.md +52 -0
- data/LICENSE.md +20 -0
- data/README.md +81 -0
- data/Rakefile +11 -0
- data/lib/wayback.rb +32 -0
- data/lib/wayback/api/archive.rb +42 -0
- data/lib/wayback/api/utils.rb +27 -0
- data/lib/wayback/archive.rb +15 -0
- data/lib/wayback/base.rb +127 -0
- data/lib/wayback/client.rb +62 -0
- data/lib/wayback/configurable.rb +48 -0
- data/lib/wayback/default.rb +68 -0
- data/lib/wayback/error.rb +31 -0
- data/lib/wayback/error/already_favorited.rb +10 -0
- data/lib/wayback/error/already_retweeted.rb +10 -0
- data/lib/wayback/error/bad_gateway.rb +11 -0
- data/lib/wayback/error/bad_request.rb +10 -0
- data/lib/wayback/error/client_error.rb +28 -0
- data/lib/wayback/error/configuration_error.rb +8 -0
- data/lib/wayback/error/decode_error.rb +9 -0
- data/lib/wayback/error/forbidden.rb +10 -0
- data/lib/wayback/error/gateway_timeout.rb +11 -0
- data/lib/wayback/error/identity_map_key_error.rb +9 -0
- data/lib/wayback/error/internal_server_error.rb +11 -0
- data/lib/wayback/error/not_acceptable.rb +10 -0
- data/lib/wayback/error/not_found.rb +10 -0
- data/lib/wayback/error/server_error.rb +28 -0
- data/lib/wayback/error/service_unavailable.rb +11 -0
- data/lib/wayback/error/too_many_requests.rb +12 -0
- data/lib/wayback/error/unauthorized.rb +10 -0
- data/lib/wayback/error/unprocessable_entity.rb +10 -0
- data/lib/wayback/factory.rb +21 -0
- data/lib/wayback/identity.rb +50 -0
- data/lib/wayback/identity_map.rb +22 -0
- data/lib/wayback/page.rb +18 -0
- data/lib/wayback/response/parse_memento.rb +61 -0
- data/lib/wayback/response/parse_memento_page.rb +23 -0
- data/lib/wayback/response/raise_error.rb +31 -0
- data/lib/wayback/version.rb +18 -0
- data/spec/fixtures/list.timemap +9 -0
- data/spec/fixtures/page.html +225 -0
- data/spec/helper.rb +65 -0
- data/spec/wayback/api/archive_spec.rb +73 -0
- data/spec/wayback/archive_spec.rb +23 -0
- data/spec/wayback/base_spec.rb +117 -0
- data/spec/wayback/client_spec.rb +114 -0
- data/spec/wayback/error/client_error_spec.rb +23 -0
- data/spec/wayback/error/server_error_spec.rb +20 -0
- data/spec/wayback/error_spec.rb +20 -0
- data/spec/wayback/identifiable_spec.rb +50 -0
- data/spec/wayback/page_spec.rb +36 -0
- data/spec/wayback_spec.rb +47 -0
- data/wayback.gemspec +26 -0
- metadata +175 -0
data/.yardopts
ADDED
data/CHANGELOG.md
ADDED
data/CONTRIBUTING.md
ADDED
@@ -0,0 +1,52 @@
|
|
1
|
+
## Contributing
|
2
|
+
In the spirit of [free software][free-sw], **everyone** is encouraged to help
|
3
|
+
improve this project.
|
4
|
+
|
5
|
+
[free-sw]: http://www.fsf.org/licensing/essays/free-sw.html
|
6
|
+
|
7
|
+
Here are some ways *you* can contribute:
|
8
|
+
|
9
|
+
* by using alpha, beta, and prerelease versions
|
10
|
+
* by reporting bugs
|
11
|
+
* by suggesting new features
|
12
|
+
* by writing or editing documentation
|
13
|
+
* by writing specifications
|
14
|
+
* by writing code (**no patch is too small**: fix typos, add comments, clean up
|
15
|
+
inconsistent whitespace)
|
16
|
+
* by refactoring code
|
17
|
+
* by fixing [issues][]
|
18
|
+
* by reviewing patches
|
19
|
+
* [financially][pledgie]
|
20
|
+
|
21
|
+
[issues]: https://github.com/xolator/wayback/issues
|
22
|
+
|
23
|
+
## Submitting an Issue
|
24
|
+
We use the [GitHub issue tracker][issues] to track bugs and features. Before
|
25
|
+
submitting a bug report or feature request, check to make sure it hasn't
|
26
|
+
already been submitted. When submitting a bug report, please include a [Gist][]
|
27
|
+
that includes a stack trace and any details that may be necessary to reproduce
|
28
|
+
the bug, including your gem version, Ruby version, and operating system.
|
29
|
+
Ideally, a bug report should include a pull request with failing specs.
|
30
|
+
|
31
|
+
[gist]: https://gist.github.com/
|
32
|
+
|
33
|
+
## Submitting a Pull Request
|
34
|
+
1. [Fork the repository.][fork]
|
35
|
+
2. [Create a topic branch.][branch]
|
36
|
+
3. Add specs for your unimplemented feature or bug fix.
|
37
|
+
4. Run `bundle exec rake spec`. If your specs pass, return to step 3.
|
38
|
+
5. Implement your feature or bug fix.
|
39
|
+
6. Run `bundle exec rake spec`. If your specs fail, return to step 5.
|
40
|
+
7. Run `open coverage/index.html`. If your changes are not completely covered
|
41
|
+
by your tests, return to step 3.
|
42
|
+
8 Run `RUBYOPT=W2 bundle exec rake spec 2>&1 | grep wayback`. If your changes
|
43
|
+
produce any warnings, return to step 5.
|
44
|
+
9. Add documentation for your feature or bug fix.
|
45
|
+
10. Run `bundle exec rake yard`. If your changes are not 100% documented, go
|
46
|
+
back to step 9.
|
47
|
+
11. Commit and push your changes.
|
48
|
+
12. [Submit a pull request.][pr]
|
49
|
+
|
50
|
+
[fork]: http://help.github.com/fork-a-repo/
|
51
|
+
[branch]: http://learn.github.com/p/branching.html
|
52
|
+
[pr]: http://help.github.com/send-pull-requests/
|
data/LICENSE.md
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2013 XOlator.
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,81 @@
|
|
1
|
+
# Wayback Gem
|
2
|
+
|
3
|
+
![Build Status](https://secure.travis-ci.org/XOlator/wayback_gem.png?branch=master)
|
4
|
+
![Coverage Status](https://coveralls.io/repos/XOlator/wayback_gem/badge.png?branch=master)
|
5
|
+
|
6
|
+
A Ruby interface to Archive.org's Wayback Machine Memento API.
|
7
|
+
|
8
|
+
## Installation
|
9
|
+
gem install wayback
|
10
|
+
|
11
|
+
## Quick Start Guide
|
12
|
+
So you want to get up and tweeting as fast as possible?
|
13
|
+
|
14
|
+
|
15
|
+
## Documentation
|
16
|
+
COMING SOON! -- [documentation](http://rdoc.info/gems/wayback)
|
17
|
+
|
18
|
+
|
19
|
+
## Configuration
|
20
|
+
|
21
|
+
There is no real configuration necessary for accessing Archive.org's Wayback Machine Memento API, however you can change endpoint and other basic connection options.
|
22
|
+
|
23
|
+
The current defaults for this gem are:
|
24
|
+
|
25
|
+
```ruby
|
26
|
+
Wayback.configure do |c|
|
27
|
+
c.endpoint = 'http://api.wayback.archive.org'
|
28
|
+
c.connection_options = {
|
29
|
+
:headers => {:user_agent => "Wayback Ruby Gem #{Wayback::Version}"},
|
30
|
+
:request => {:open_timeout => 5, :timeout => 10},
|
31
|
+
:ssl => {:verify => false},
|
32
|
+
}
|
33
|
+
c.identiy_map = false
|
34
|
+
c.middleware = Faraday::Builder.new do |builder|
|
35
|
+
# Convert request params to "www-form-urlencoded"
|
36
|
+
builder.use Faraday::Request::UrlEncoded
|
37
|
+
# Follow redirects
|
38
|
+
builder.use FaradayMiddleware::FollowRedirects
|
39
|
+
# Handle 4xx server responses
|
40
|
+
builder.use Wayback::Response::RaiseError, Wayback::Error::ClientError
|
41
|
+
# Handle 5xx server responses
|
42
|
+
builder.use Wayback::Response::RaiseError, Wayback::Error::ServerError
|
43
|
+
# Parse memento page
|
44
|
+
builder.use Wayback::Response::ParseMementoPage
|
45
|
+
# Parse link-format with custom memento parser
|
46
|
+
builder.use Wayback::Response::ParseMemento
|
47
|
+
# Set Faraday's HTTP adapter
|
48
|
+
builder.adapter Faraday.default_adapter
|
49
|
+
end
|
50
|
+
```
|
51
|
+
|
52
|
+
|
53
|
+
## Usage Examples
|
54
|
+
|
55
|
+
**Fetch the timeline of archived pages**
|
56
|
+
|
57
|
+
```ruby
|
58
|
+
Wayback.list('http://www.xolator.com')
|
59
|
+
```
|
60
|
+
|
61
|
+
**Fetch a specific archived page**
|
62
|
+
|
63
|
+
```ruby
|
64
|
+
Wayback.page('http://www.xolator.com', 1363488758)
|
65
|
+
Wayback.page('http://www.xolator.com', :first)
|
66
|
+
Wayback.page('http://www.xolator.com', :last)
|
67
|
+
```
|
68
|
+
|
69
|
+
|
70
|
+
## Additional Notes
|
71
|
+
Based heavily on the [Twitter gem](https://www.github.com/sferik/twitter). (Xie xie!)
|
72
|
+
|
73
|
+
More information on Archive's Wayback Machine memento API can be found [here](http://mementoweb.org/depot/native/ia/).
|
74
|
+
|
75
|
+
|
76
|
+
|
77
|
+
## Copyright
|
78
|
+
Copyright (c) 2013 XOlator.
|
79
|
+
See [LICENSE][] for details.
|
80
|
+
|
81
|
+
[license]: LICENSE.md
|
data/Rakefile
ADDED
data/lib/wayback.rb
ADDED
@@ -0,0 +1,32 @@
|
|
1
|
+
require 'wayback/archive'
|
2
|
+
require 'wayback/client'
|
3
|
+
require 'wayback/configurable'
|
4
|
+
require 'wayback/default'
|
5
|
+
require 'wayback/page'
|
6
|
+
|
7
|
+
module Wayback
|
8
|
+
class << self
|
9
|
+
include Wayback::Configurable
|
10
|
+
|
11
|
+
# Delegate to a Wayback::Client
|
12
|
+
#
|
13
|
+
# @return [Wayback::Client]
|
14
|
+
def client
|
15
|
+
@client = Wayback::Client.new(options) unless defined?(@client) && @client.hash == options.hash
|
16
|
+
@client
|
17
|
+
end
|
18
|
+
|
19
|
+
def respond_to_missing?(method_name, include_private=false); client.respond_to?(method_name, include_private); end if RUBY_VERSION >= "1.9"
|
20
|
+
def respond_to?(method_name, include_private=false); client.respond_to?(method_name, include_private) || super; end if RUBY_VERSION < "1.9"
|
21
|
+
|
22
|
+
private
|
23
|
+
|
24
|
+
def method_missing(method_name, *args, &block)
|
25
|
+
return super unless client.respond_to?(method_name)
|
26
|
+
client.send(method_name, *args, &block)
|
27
|
+
end
|
28
|
+
|
29
|
+
end
|
30
|
+
end
|
31
|
+
|
32
|
+
Wayback.setup
|
@@ -0,0 +1,42 @@
|
|
1
|
+
require 'time'
|
2
|
+
require 'wayback/api/utils'
|
3
|
+
require 'wayback/error/not_found'
|
4
|
+
|
5
|
+
module Wayback
|
6
|
+
module API
|
7
|
+
module Archive
|
8
|
+
include Wayback::API::Utils
|
9
|
+
|
10
|
+
# Return a list of archived pages
|
11
|
+
#
|
12
|
+
# @return [Wayback::Archive]
|
13
|
+
# @param url [String] The page that of which was archived.
|
14
|
+
# @param options [Hash] A customizable set of options.
|
15
|
+
# @example Return the list of available archives for a web page.
|
16
|
+
# Wayback.list('http://gleu.ch')
|
17
|
+
def list(url, options={})
|
18
|
+
object_from_response(Wayback::Archive, :get, "/list/timemap/link/#{url}", options)
|
19
|
+
end
|
20
|
+
|
21
|
+
# Returns the HTML contents of an archive page, fetched by date
|
22
|
+
#
|
23
|
+
# @raise [Wayback::Error::Unauthorized] Error raised when supplied user credentials are not valid.
|
24
|
+
# @return [Wayback::Page]
|
25
|
+
# @param url [String] The page that of which was archived.
|
26
|
+
# @param options [Hash] A customizable set of options.
|
27
|
+
# @example Return the HTML archive for the page.
|
28
|
+
# Wayback.page('http://gleu.ch')
|
29
|
+
# Wayback.page('http://gleu.ch', 'Tue, 17 Jan 2012 07:33:06 GMT')
|
30
|
+
# Wayback.page('http://gleu.ch', '20130113125339')
|
31
|
+
# Wayback.page('http://gleu.ch', :first)
|
32
|
+
# Wayback.page('http://gleu.ch', :last)
|
33
|
+
def page(url, date=0, options={})
|
34
|
+
date = 0 if date == :first
|
35
|
+
date = Time.now if date == :last
|
36
|
+
date = Time.parse(date).to_i unless [Fixnum,Time,Integer].include?(date.class)
|
37
|
+
object_from_response(Wayback::Page, :get, "/memento/#{date.to_i}/#{url}", options)
|
38
|
+
end
|
39
|
+
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
module Wayback
|
2
|
+
module API
|
3
|
+
module Utils
|
4
|
+
|
5
|
+
private
|
6
|
+
|
7
|
+
# @param klass [Class]
|
8
|
+
# @param request_method [Symbol]
|
9
|
+
# @param path [String]
|
10
|
+
# @param options [Hash]
|
11
|
+
# @return [Object]
|
12
|
+
def object_from_response(klass, request_method, path, options={})
|
13
|
+
response = send(request_method.to_sym, path, options)
|
14
|
+
klass.from_response(response)
|
15
|
+
end
|
16
|
+
|
17
|
+
# def handle_forbidden_error(klass, error)
|
18
|
+
# if error.message == klass::MESSAGE
|
19
|
+
# raise klass.new
|
20
|
+
# else
|
21
|
+
# raise error
|
22
|
+
# end
|
23
|
+
# end
|
24
|
+
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
data/lib/wayback/base.rb
ADDED
@@ -0,0 +1,127 @@
|
|
1
|
+
require 'wayback/error/identity_map_key_error'
|
2
|
+
|
3
|
+
module Wayback
|
4
|
+
class Base
|
5
|
+
# Define methods that retrieve the value from an initialized instance variable Hash, using the attribute as a key
|
6
|
+
#
|
7
|
+
# @param attrs [Array, Set, Symbol]
|
8
|
+
def self.attr_reader(*attrs)
|
9
|
+
mod = Module.new do
|
10
|
+
attrs.each do |attribute|
|
11
|
+
define_method attribute do
|
12
|
+
@attrs[attribute.to_sym]
|
13
|
+
end
|
14
|
+
define_method "#{attribute}?" do
|
15
|
+
!!@attrs[attribute.to_sym]
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
const_set(:Attributes, mod)
|
20
|
+
include mod
|
21
|
+
end
|
22
|
+
|
23
|
+
# return [Wayback::IdentityMap]
|
24
|
+
def self.identity_map
|
25
|
+
return unless Wayback.identity_map
|
26
|
+
@identity_map = Wayback.identity_map.new unless defined?(@identity_map) && @identity_map.class == Wayback.identity_map
|
27
|
+
@identity_map
|
28
|
+
end
|
29
|
+
|
30
|
+
# Retrieves an object from the identity map.
|
31
|
+
#
|
32
|
+
# @param attrs [Hash]
|
33
|
+
# @return [Wayback::Base]
|
34
|
+
def self.fetch(attrs)
|
35
|
+
return unless identity_map
|
36
|
+
if object = identity_map.fetch(Marshal.dump(attrs))
|
37
|
+
return object
|
38
|
+
end
|
39
|
+
return yield if block_given?
|
40
|
+
raise Wayback::Error::IdentityMapKeyError, "key not found"
|
41
|
+
end
|
42
|
+
|
43
|
+
# Stores an object in the identity map.
|
44
|
+
#
|
45
|
+
# @param object [Object]
|
46
|
+
# @return [Wayback::Base]
|
47
|
+
def self.store(object)
|
48
|
+
return object unless identity_map
|
49
|
+
identity_map.store(Marshal.dump(object.attrs), object)
|
50
|
+
end
|
51
|
+
|
52
|
+
# Returns a new object based on the response hash
|
53
|
+
#
|
54
|
+
# @param response [Hash]
|
55
|
+
# @return [Wayback::Base]
|
56
|
+
def self.from_response(response={})
|
57
|
+
fetch_or_new(response[:body])
|
58
|
+
end
|
59
|
+
|
60
|
+
# Retrieves an object from the identity map, or stores it in the
|
61
|
+
# identity map if it doesn't already exist.
|
62
|
+
#
|
63
|
+
# @param attrs [Hash]
|
64
|
+
# @return [Wayback::Base]
|
65
|
+
def self.fetch_or_new(attrs={})
|
66
|
+
return unless attrs
|
67
|
+
return new(attrs) unless identity_map
|
68
|
+
|
69
|
+
fetch(attrs) do
|
70
|
+
object = new(attrs)
|
71
|
+
store(object)
|
72
|
+
end
|
73
|
+
end
|
74
|
+
|
75
|
+
# Initializes a new object
|
76
|
+
#
|
77
|
+
# @param attrs [Hash]
|
78
|
+
# @return [Wayback::Base]
|
79
|
+
def initialize(attrs={})
|
80
|
+
@attrs = attrs
|
81
|
+
end
|
82
|
+
|
83
|
+
# Fetches an attribute of an object using hash notation
|
84
|
+
#
|
85
|
+
# @param method [String, Symbol] Message to send to the object
|
86
|
+
def [](method)
|
87
|
+
send(method.to_sym)
|
88
|
+
rescue NoMethodError
|
89
|
+
nil
|
90
|
+
end
|
91
|
+
|
92
|
+
# Retrieve the attributes of an object
|
93
|
+
#
|
94
|
+
# @return [Hash]
|
95
|
+
def attrs
|
96
|
+
@attrs.inject({}) do |attrs, (key, value)|
|
97
|
+
attrs.merge!(key => respond_to?(key) ? send(key) : value)
|
98
|
+
end
|
99
|
+
end
|
100
|
+
alias to_hash attrs
|
101
|
+
|
102
|
+
# Update the attributes of an object
|
103
|
+
#
|
104
|
+
# @param attrs [Hash]
|
105
|
+
# @return [Wayback::Base]
|
106
|
+
def update(attrs)
|
107
|
+
@attrs.update(attrs)
|
108
|
+
self
|
109
|
+
end
|
110
|
+
|
111
|
+
protected
|
112
|
+
|
113
|
+
# @param attr [Symbol]
|
114
|
+
# @param other [Wayback::Base]
|
115
|
+
# @return [Boolean]
|
116
|
+
def attr_equal(attr, other)
|
117
|
+
self.class == other.class && !other.send(attr).nil? && send(attr) == other.send(attr)
|
118
|
+
end
|
119
|
+
|
120
|
+
# @param other [Wayback::Base]
|
121
|
+
# @return [Boolean]
|
122
|
+
def attrs_equal(other)
|
123
|
+
self.class == other.class && !other.attrs.empty? && attrs == other.attrs
|
124
|
+
end
|
125
|
+
|
126
|
+
end
|
127
|
+
end
|
@@ -0,0 +1,62 @@
|
|
1
|
+
require 'faraday'
|
2
|
+
require 'wayback/api/archive'
|
3
|
+
require 'wayback/configurable'
|
4
|
+
require 'wayback/error/client_error'
|
5
|
+
require 'wayback/error/decode_error'
|
6
|
+
require 'uri'
|
7
|
+
|
8
|
+
module Wayback
|
9
|
+
# Wrapper for the Wayback Memento/Timehop API
|
10
|
+
class Client
|
11
|
+
include Wayback::API::Archive
|
12
|
+
include Wayback::Configurable
|
13
|
+
|
14
|
+
# Initializes a new Client object
|
15
|
+
#
|
16
|
+
# @param options [Hash]
|
17
|
+
# @return [Wayback::Client]
|
18
|
+
def initialize(options={})
|
19
|
+
Wayback::Configurable.keys.each do |key|
|
20
|
+
instance_variable_set(:"@#{key}", options[key] || Wayback.instance_variable_get(:"@#{key}"))
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
# Perform an HTTP DELETE request
|
25
|
+
def delete(path, params={})
|
26
|
+
request(:delete, path, params)
|
27
|
+
end
|
28
|
+
|
29
|
+
# Perform an HTTP GET request
|
30
|
+
def get(path, params={})
|
31
|
+
request(:get, path, params)
|
32
|
+
end
|
33
|
+
|
34
|
+
# Perform an HTTP POST request
|
35
|
+
def post(path, params={})
|
36
|
+
signature_params = params.values.any?{|value| value.respond_to?(:to_io)} ? {} : params
|
37
|
+
request(:post, path, params, signature_params)
|
38
|
+
end
|
39
|
+
|
40
|
+
# Perform an HTTP PUT request
|
41
|
+
def put(path, params={})
|
42
|
+
request(:put, path, params)
|
43
|
+
end
|
44
|
+
|
45
|
+
|
46
|
+
private
|
47
|
+
|
48
|
+
def request(method, path, params={}, signature_params=params)
|
49
|
+
connection.send(method.to_sym, path, params).env
|
50
|
+
rescue Faraday::Error::ClientError
|
51
|
+
raise Wayback::Error::ClientError
|
52
|
+
end
|
53
|
+
|
54
|
+
# Returns a Faraday::Connection object
|
55
|
+
#
|
56
|
+
# @return [Faraday::Connection]
|
57
|
+
def connection
|
58
|
+
@connection ||= Faraday.new(@endpoint, @connection_options.merge(:builder => @middleware))
|
59
|
+
end
|
60
|
+
|
61
|
+
end
|
62
|
+
end
|