twingly-search 5.0.1 → 5.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +1 -2
- data/CHANGELOG.md +16 -0
- data/README.md +33 -34
- data/Rakefile +0 -6
- data/examples/find_all_posts_mentioning_github.rb +3 -3
- data/examples/hello_world.rb +2 -2
- data/examples/livefeed_loop.rb +24 -0
- data/lib/twingly/livefeed/client.rb +121 -0
- data/lib/twingly/livefeed/error.rb +28 -0
- data/lib/twingly/livefeed/parser.rb +96 -0
- data/lib/twingly/livefeed/post.rb +66 -0
- data/lib/twingly/livefeed/result.rb +39 -0
- data/lib/twingly/livefeed/version.rb +5 -0
- data/lib/twingly/livefeed.rb +6 -0
- data/lib/twingly/search/client.rb +3 -2
- data/lib/twingly/search/error.rb +6 -5
- data/lib/twingly/search/parser.rb +39 -13
- data/lib/twingly/search/post.rb +65 -21
- data/lib/twingly/search/query.rb +46 -16
- data/lib/twingly/search/result.rb +11 -0
- data/lib/twingly/search/version.rb +1 -1
- data/spec/client_spec.rb +2 -2
- data/spec/error_spec.rb +27 -7
- data/spec/fixtures/incomplete_result.xml +2 -0
- data/spec/fixtures/livefeed/empty_api_key_result.xml +3 -0
- data/spec/fixtures/livefeed/non_xml_result.xml +1 -0
- data/spec/fixtures/livefeed/not_found_result.xml +3 -0
- data/spec/fixtures/livefeed/service_unavailable_result.xml +3 -0
- data/spec/fixtures/livefeed/unauthorized_api_key_result.xml +3 -0
- data/spec/fixtures/livefeed/valid_empty_result.xml +2 -0
- data/spec/fixtures/livefeed/valid_result.xml +79 -0
- data/spec/fixtures/minimal_valid_result.xml +81 -52
- data/spec/fixtures/nonexistent_api_key_result.xml +3 -3
- data/spec/fixtures/service_unavailable_result.xml +3 -3
- data/spec/fixtures/unauthorized_api_key_result.xml +3 -3
- data/spec/fixtures/undefined_error_result.xml +3 -3
- data/spec/fixtures/valid_empty_result.xml +2 -2
- data/spec/fixtures/valid_links_result.xml +36 -0
- data/spec/fixtures/vcr_cassettes/livefeed_valid_request.yml +169 -0
- data/spec/fixtures/vcr_cassettes/search_for_spotify_on_sv_blogs.yml +578 -447
- data/spec/fixtures/vcr_cassettes/search_without_valid_api_key.yml +15 -14
- data/spec/livefeed/client_spec.rb +135 -0
- data/spec/livefeed/error_spec.rb +51 -0
- data/spec/livefeed/parser_spec.rb +351 -0
- data/spec/livefeed/post_spec.rb +26 -0
- data/spec/livefeed/result_spec.rb +18 -0
- data/spec/parser_spec.rb +191 -94
- data/spec/post_spec.rb +25 -6
- data/spec/query_spec.rb +41 -34
- data/spec/result_spec.rb +1 -0
- data/spec/spec_helper.rb +10 -0
- data/twingly-search-api-ruby.gemspec +2 -3
- metadata +44 -24
- data/spec/fixtures/valid_non_blog_result.xml +0 -26
- data/spec/fixtures/valid_result.xml +0 -22975
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3c9c84cb7c69272bfd499b2901f475082d4dc251
|
4
|
+
data.tar.gz: d51285306f3bc40452016e81bee6a6b22f783c59
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 407a46a73b5bff7aed5277240f45bf7be611cafc2b64b8b9a22de2b019be3f3cd49d1da236c4ecdd5a6a221e0a82e54482ba60c40ca4babae33f0e6e0f93b389
|
7
|
+
data.tar.gz: 7f64f9eec9386722f72f13701f473d687680016f2bf303be68007a30b116872b61e8790ed6a43c8ff3ba7702edc81baecf31f6b0c7cd4850bf07070da439796b
|
data/.travis.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,21 @@
|
|
1
1
|
# Change Log
|
2
2
|
|
3
|
+
## [v5.0.1](https://github.com/twingly/twingly-search-api-ruby/tree/v5.0.1) (2016-03-03)
|
4
|
+
[Full Changelog](https://github.com/twingly/twingly-search-api-ruby/compare/v5.0.0...v5.0.1)
|
5
|
+
|
6
|
+
**Fixed bugs:**
|
7
|
+
|
8
|
+
- Not possible to remove start\_time from query [\#53](https://github.com/twingly/twingly-search-api-ruby/issues/53)
|
9
|
+
|
10
|
+
**Closed issues:**
|
11
|
+
|
12
|
+
- New release [\#48](https://github.com/twingly/twingly-search-api-ruby/issues/48)
|
13
|
+
|
14
|
+
**Merged pull requests:**
|
15
|
+
|
16
|
+
- Be able to remove start/end time from Query [\#54](https://github.com/twingly/twingly-search-api-ruby/pull/54) ([dentarg](https://github.com/dentarg))
|
17
|
+
- Actually test what we intend to test [\#52](https://github.com/twingly/twingly-search-api-ruby/pull/52) ([dentarg](https://github.com/dentarg))
|
18
|
+
|
3
19
|
## [v5.0.0](https://github.com/twingly/twingly-search-api-ruby/tree/v5.0.0) (2016-02-17)
|
4
20
|
[Full Changelog](https://github.com/twingly/twingly-search-api-ruby/compare/v4.0.1...v5.0.0)
|
5
21
|
|
data/README.md
CHANGED
@@ -3,7 +3,7 @@
|
|
3
3
|
[![Build Status](https://travis-ci.org/twingly/twingly-search-api-ruby.png?branch=master)](https://travis-ci.org/twingly/twingly-search-api-ruby)
|
4
4
|
[![Code Climate](https://codeclimate.com/github/twingly/twingly-search-api-ruby/badges/gpa.svg)](https://codeclimate.com/github/twingly/twingly-search-api-ruby)
|
5
5
|
|
6
|
-
A Ruby gem for Twingly's Search API (previously known as Analytics API). Twingly is a blog search service that provides a searchable API known as [Twingly Search API]
|
6
|
+
A Ruby gem for Twingly's Blog Search API (previously known as Analytics API) and Blog LiveFeed API. Twingly is a blog search service that provides a searchable API known as [Twingly Blog Search API][Blog Search API documentation] and a blog data firehose called [Twingly Blog LiveFeed API][Blog LiveFeed API documentation].
|
7
7
|
|
8
8
|
## Installation
|
9
9
|
|
@@ -21,6 +21,18 @@ gem "twingly-search"
|
|
21
21
|
|
22
22
|
## Usage
|
23
23
|
|
24
|
+
The `twingly-search` gem talks to a commercial API and requires an API key. Best practice is to set the `TWINGLY_SEARCH_KEY` environment variable to the obtained key. `Twingly::Search::Client` and `Twingly::LiveFeed::Client` can be passed a key at initialization if your setup does not allow environment variables.
|
25
|
+
|
26
|
+
To learn more about the features of this gem, read the [gem documentation] or check out the example code that can be found in [examples/](examples/).
|
27
|
+
|
28
|
+
To learn more about the capabilities of Twingly's APIs, please read the [Blog Search API documentation] and [Blog LiveFeed API documentation].
|
29
|
+
|
30
|
+
[gem documentation]: http://www.rubydoc.info/github/twingly/twingly-search-api-ruby
|
31
|
+
[Blog Search API documentation]: https://developer.twingly.com/resources/search/
|
32
|
+
[Blog LiveFeed API documentation]: https://developer.twingly.com/resources/livefeed/
|
33
|
+
|
34
|
+
### Blog Search API
|
35
|
+
|
24
36
|
```ruby
|
25
37
|
require "twingly/search"
|
26
38
|
|
@@ -29,8 +41,8 @@ client = Twingly::Search::Client.new do |client|
|
|
29
41
|
end
|
30
42
|
|
31
43
|
query = client.query do |query|
|
32
|
-
query.
|
33
|
-
query.language
|
44
|
+
query.search_query = "github page-size:10"
|
45
|
+
query.language = "sv"
|
34
46
|
end
|
35
47
|
|
36
48
|
result = query.execute
|
@@ -38,20 +50,30 @@ result = query.execute
|
|
38
50
|
result.posts # will include all returned posts
|
39
51
|
```
|
40
52
|
|
41
|
-
|
53
|
+
### Blog LiveFeed API
|
42
54
|
|
43
|
-
|
55
|
+
```ruby
|
56
|
+
require "twingly/livefeed"
|
44
57
|
|
45
|
-
|
58
|
+
client = Twingly::LiveFeed::Client.new do |client|
|
59
|
+
client.user_agent = "MyCompany/1.0"
|
60
|
+
# Start getting posts indexed by Twingly at this timestamp
|
61
|
+
client.timestamp = Time.now - 3600 # 1 hour ago
|
62
|
+
client.max_posts = 1000 # Maximum number of posts returned per call
|
63
|
+
end
|
46
64
|
|
47
|
-
|
48
|
-
|
65
|
+
# get the next chunk of posts
|
66
|
+
result = client.next_result
|
67
|
+
=> #<Twingly::LiveFeed::Result:0x3fcd98215c14 @posts, @ts=2017-04-18 14:42:18 UTC, @from=2017-04-18 13:42:06 UTC, @number_of_posts=989, @max_number_of_posts=1000, @first_post=2017-04-18 13:42:19 UTC, @last_post=2017-04-18 14:42:13 UTC>
|
68
|
+
|
69
|
+
result.posts # will include all returned posts
|
70
|
+
```
|
49
71
|
|
50
72
|
## Requirements
|
51
73
|
|
52
|
-
* API key,
|
74
|
+
* API key, [sign up](https://www.twingly.com/try-for-free) via [twingly.com](https://www.twingly.com/) to get one
|
53
75
|
* Ruby
|
54
|
-
* Ruby
|
76
|
+
* Ruby 2.1, 2.2, 2.3
|
55
77
|
* JRuby 9000
|
56
78
|
|
57
79
|
## Development
|
@@ -70,7 +92,7 @@ Run the tests
|
|
70
92
|
|
71
93
|
1. Make a commit bumping the version in `lib/twingly/search/version.rb`, follow [Semantic Versioning 2.0.0](http://semver.org/). No need to push as this will be taken care of automatically in the next step.
|
72
94
|
1. Build and the release gem with `bundle exec rake release`. This will create a git tag for the version and push the `.gem` file to [RubyGems.org].
|
73
|
-
1. Generate a changelog with `
|
95
|
+
1. Generate a changelog with `github_changelog_generator` (`gem install github_changelog_generator` if you don't have it). Set `CHANGELOG_GITHUB_TOKEN` to a personal access token to increase the API rate limit. (The changelog uses [GitHub Changelog Generator](https://github.com/skywinder/github-changelog-generator/))
|
74
96
|
1. Update release information on the [releases page]. This is done manually.
|
75
97
|
|
76
98
|
[releases page]: https://github.com/twingly/twingly-search-api-ruby/releases
|
@@ -85,26 +107,3 @@ This gem is documented using [YARD]. To start a local YARD server run:
|
|
85
107
|
The YARD server reloads the documentation automatically so there is no need to restart it when making changes.
|
86
108
|
|
87
109
|
[YARD]: http://yardoc.org/
|
88
|
-
|
89
|
-
## License
|
90
|
-
|
91
|
-
The MIT License (MIT)
|
92
|
-
|
93
|
-
Copyright (c) 2013-2016 Twingly AB
|
94
|
-
|
95
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
96
|
-
this software and associated documentation files (the "Software"), to deal in
|
97
|
-
the Software without restriction, including without limitation the rights to
|
98
|
-
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
|
99
|
-
the Software, and to permit persons to whom the Software is furnished to do so,
|
100
|
-
subject to the following conditions:
|
101
|
-
|
102
|
-
The above copyright notice and this permission notice shall be included in all
|
103
|
-
copies or substantial portions of the Software.
|
104
|
-
|
105
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
106
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
107
|
-
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
108
|
-
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
109
|
-
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
110
|
-
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/Rakefile
CHANGED
@@ -10,12 +10,6 @@ task :spec do
|
|
10
10
|
end
|
11
11
|
end
|
12
12
|
|
13
|
-
require "github_changelog_generator/task"
|
14
|
-
GitHubChangelogGenerator::RakeTask.new(:changelog) do |config|
|
15
|
-
config.user = "twingly"
|
16
|
-
config.project = "twingly-search-api-ruby"
|
17
|
-
end
|
18
|
-
|
19
13
|
namespace :yard do
|
20
14
|
require "yard"
|
21
15
|
require "yard/rake/yardoc_task"
|
@@ -8,8 +8,8 @@ class SearchPostStream
|
|
8
8
|
end
|
9
9
|
|
10
10
|
@query = client.query do |query|
|
11
|
-
query.language
|
12
|
-
query.
|
11
|
+
query.language = language
|
12
|
+
query.search_query = "sort-order:asc sort:published #{keyword}"
|
13
13
|
end
|
14
14
|
end
|
15
15
|
|
@@ -25,7 +25,7 @@ class SearchPostStream
|
|
25
25
|
|
26
26
|
break if result.all_results_returned?
|
27
27
|
|
28
|
-
@query.start_time = result.posts.last.
|
28
|
+
@query.start_time = result.posts.last.published_at
|
29
29
|
end
|
30
30
|
end
|
31
31
|
|
data/examples/hello_world.rb
CHANGED
@@ -3,8 +3,8 @@ Bundler.require
|
|
3
3
|
# Set environment variable TWINGLY_SEARCH_KEY
|
4
4
|
client = Twingly::Search::Client.new
|
5
5
|
query = client.query do |query|
|
6
|
-
query.
|
7
|
-
query.start_time
|
6
|
+
query.search_query = '"hello world"'
|
7
|
+
query.start_time = Time.now - (24 * 3600) # search last day
|
8
8
|
end
|
9
9
|
|
10
10
|
result = query.execute
|
@@ -0,0 +1,24 @@
|
|
1
|
+
Bundler.require
|
2
|
+
|
3
|
+
require "twingly/livefeed"
|
4
|
+
|
5
|
+
# Set environment variable TWINGLY_SEARCH_KEY
|
6
|
+
client = Twingly::LiveFeed::Client.new do |client|
|
7
|
+
# get posts published less than an hour ago
|
8
|
+
client.timestamp = (Time.now - 3600)
|
9
|
+
client.max_posts = 4000
|
10
|
+
end
|
11
|
+
|
12
|
+
loop do
|
13
|
+
result = client.next_result
|
14
|
+
|
15
|
+
result.posts.each do |post|
|
16
|
+
puts "#{post.indexed_at} - #{post.url}"
|
17
|
+
end
|
18
|
+
|
19
|
+
# You're faster than Twingly! Take a break and wait for
|
20
|
+
# us to index some new posts before asking again
|
21
|
+
if result.number_of_posts < result.max_number_of_posts
|
22
|
+
sleep 20
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,121 @@
|
|
1
|
+
require "faraday"
|
2
|
+
require "time"
|
3
|
+
|
4
|
+
module Twingly
|
5
|
+
module LiveFeed
|
6
|
+
# Twingly LiveFeed API client
|
7
|
+
#
|
8
|
+
# @attr [String] api_key the API key
|
9
|
+
# @attr [String] user_agent the user agent to be used for all API requests
|
10
|
+
# @attr [Integer] max_posts the maximum number of posts that each request can return
|
11
|
+
# @attr [Time] timestamp the timestamp that will be used in the next request
|
12
|
+
class Client
|
13
|
+
attr_accessor :api_key, :user_agent, :max_posts, :timestamp
|
14
|
+
|
15
|
+
BASE_URL = "https://api.twingly.com"
|
16
|
+
LIVEFEED_API_VERSION = "v5"
|
17
|
+
LIVEFEED_PATH = "/blog/livefeed/api/#{LIVEFEED_API_VERSION}/getdata"
|
18
|
+
|
19
|
+
DEFAULT_USER_AGENT = "Twingly LiveFeed Ruby Client/#{VERSION}"
|
20
|
+
DEFAULT_MAX_POSTS = 1000
|
21
|
+
TIMESTAMP_FORMAT = "%Y-%m-%dT%H:%M:%S.%3N%z"
|
22
|
+
|
23
|
+
# Creates a new Twingly Search API client
|
24
|
+
#
|
25
|
+
# @param api_key [String] the API key provided by Twingly.
|
26
|
+
# If nil, reads key from environment (TWINGLY_SEARCH_KEY).
|
27
|
+
# @param options [Hash]
|
28
|
+
# @option options [String] :user_agent the user agent to be used
|
29
|
+
# for all requests
|
30
|
+
# @option options [String] :max_posts the maximum number of posts that can
|
31
|
+
# be returned for each request
|
32
|
+
# @option options [String] :timestamp the timestamp to start the client at
|
33
|
+
# @raise [AuthError] if an API key is not set
|
34
|
+
def initialize(api_key = nil, options = {})
|
35
|
+
@api_key = api_key
|
36
|
+
@user_agent = options.fetch(:user_agent) { DEFAULT_USER_AGENT }
|
37
|
+
@max_posts = options.fetch(:max_posts) { DEFAULT_MAX_POSTS }
|
38
|
+
@timestamp = options.fetch(:timestamp) { Time.now }
|
39
|
+
|
40
|
+
yield self if block_given?
|
41
|
+
|
42
|
+
@api_key ||= env_api_key || api_key_missing
|
43
|
+
end
|
44
|
+
|
45
|
+
# Get the next result from the API and updates the next timestamp
|
46
|
+
#
|
47
|
+
# Sends a request to the API using the timestamp set with {#timestamp},
|
48
|
+
# updates the timestamp that should be used in the next request and
|
49
|
+
# then returns the result.
|
50
|
+
#
|
51
|
+
# @raise [QueryError] if the timestamp is invalid.
|
52
|
+
# @raise [AuthError] if the API couldn't authenticate you. Make sure your API key is correct.
|
53
|
+
# @raise [ServerError] if the query could not be executed due to a server error.
|
54
|
+
# @return [Result] the result for this request.
|
55
|
+
def next_result
|
56
|
+
assert_valid_time(timestamp)
|
57
|
+
|
58
|
+
get_and_parse_result.tap do |result|
|
59
|
+
update_timestamp(result.next_timestamp)
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
# @return [String] the request url for the next request.
|
64
|
+
def url
|
65
|
+
"#{endpoint_url}?#{url_parameters}"
|
66
|
+
end
|
67
|
+
|
68
|
+
# @return [String] the API endpoint URL.
|
69
|
+
def endpoint_url
|
70
|
+
"#{BASE_URL}#{LIVEFEED_PATH}"
|
71
|
+
end
|
72
|
+
|
73
|
+
# @see #url
|
74
|
+
# @return [String] the query part of the request url.
|
75
|
+
def url_parameters
|
76
|
+
Faraday::Utils.build_query(request_parameters)
|
77
|
+
end
|
78
|
+
|
79
|
+
# @return [Hash] the request parameters.
|
80
|
+
def request_parameters
|
81
|
+
{
|
82
|
+
apikey: api_key,
|
83
|
+
timestamp: timestamp.to_time.utc.strftime(TIMESTAMP_FORMAT),
|
84
|
+
maxPosts: max_posts,
|
85
|
+
}
|
86
|
+
end
|
87
|
+
|
88
|
+
private
|
89
|
+
|
90
|
+
def update_timestamp(timestamp)
|
91
|
+
@timestamp = timestamp
|
92
|
+
end
|
93
|
+
|
94
|
+
def get_and_parse_result
|
95
|
+
response_body = get_response.body
|
96
|
+
Parser.new.parse(response_body)
|
97
|
+
end
|
98
|
+
|
99
|
+
def get_response
|
100
|
+
connection = Faraday.new(url: BASE_URL) do |faraday|
|
101
|
+
faraday.request :url_encoded
|
102
|
+
faraday.adapter Faraday.default_adapter
|
103
|
+
end
|
104
|
+
connection.headers[:user_agent] = user_agent
|
105
|
+
connection.get(LIVEFEED_PATH, request_parameters)
|
106
|
+
end
|
107
|
+
|
108
|
+
def env_api_key
|
109
|
+
ENV['TWINGLY_SEARCH_KEY']
|
110
|
+
end
|
111
|
+
|
112
|
+
def api_key_missing
|
113
|
+
fail AuthError, "No API key has been provided."
|
114
|
+
end
|
115
|
+
|
116
|
+
def assert_valid_time(time)
|
117
|
+
fail QueryError, "Not a Time object" unless time.respond_to?(:to_time)
|
118
|
+
end
|
119
|
+
end
|
120
|
+
end
|
121
|
+
end
|
@@ -0,0 +1,28 @@
|
|
1
|
+
module Twingly
|
2
|
+
module LiveFeed
|
3
|
+
class Error < StandardError
|
4
|
+
def self.from_api_response(code, message)
|
5
|
+
error =
|
6
|
+
case code.to_s
|
7
|
+
when /^400/, /^404/
|
8
|
+
QueryError
|
9
|
+
when /^401/
|
10
|
+
AuthError
|
11
|
+
else
|
12
|
+
ServerError
|
13
|
+
end
|
14
|
+
|
15
|
+
error.new("#{message} (code: #{code})")
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
class AuthError < Error
|
20
|
+
end
|
21
|
+
|
22
|
+
class ServerError < Error
|
23
|
+
end
|
24
|
+
|
25
|
+
class QueryError < Error
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
@@ -0,0 +1,96 @@
|
|
1
|
+
require 'nokogiri'
|
2
|
+
|
3
|
+
module Twingly
|
4
|
+
module LiveFeed
|
5
|
+
class Parser
|
6
|
+
# Parse an API response body.
|
7
|
+
#
|
8
|
+
# @param [String] document containing an API response XML.
|
9
|
+
# @raise [Error] which error depends on the API response (see {Error.from_api_response}).
|
10
|
+
# @return [Result] containing the result.
|
11
|
+
def parse(document)
|
12
|
+
nokogiri = Nokogiri::XML(document)
|
13
|
+
|
14
|
+
failure = nokogiri.at_xpath('/error')
|
15
|
+
handle_failure(failure) if failure
|
16
|
+
|
17
|
+
data_node = nokogiri.at_xpath('/twinglydata')
|
18
|
+
handle_non_xml_document(document) unless data_node
|
19
|
+
|
20
|
+
create_result(data_node)
|
21
|
+
end
|
22
|
+
|
23
|
+
private
|
24
|
+
|
25
|
+
def create_result(data_node)
|
26
|
+
result = Result.new
|
27
|
+
result.ts = parse_time(data_node.attribute('ts').value)
|
28
|
+
result.from = parse_time(data_node.attribute('from').value)
|
29
|
+
result.number_of_posts = data_node.attribute('numberOfPosts').value.to_i
|
30
|
+
result.max_number_of_posts = data_node.attribute('maxNumberOfPosts').value.to_i
|
31
|
+
result.next_timestamp = parse_time(data_node.attribute('nextTimestamp').value)
|
32
|
+
|
33
|
+
unless result.number_of_posts.zero?
|
34
|
+
result.first_post = parse_time(data_node.attribute('firstPost').value)
|
35
|
+
result.last_post = parse_time(data_node.attribute('lastPost').value)
|
36
|
+
end
|
37
|
+
|
38
|
+
data_node.xpath('//post').each do |post|
|
39
|
+
result.posts << parse_post(post)
|
40
|
+
end
|
41
|
+
|
42
|
+
result
|
43
|
+
end
|
44
|
+
|
45
|
+
def parse_post(element)
|
46
|
+
post_params = {}
|
47
|
+
element.element_children.each do |child|
|
48
|
+
post_params[child.name] =
|
49
|
+
case child.name
|
50
|
+
when *%w(tags links images)
|
51
|
+
parse_array(child)
|
52
|
+
when "coordinates"
|
53
|
+
parse_coordinates(child)
|
54
|
+
else
|
55
|
+
child.text
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
post = Post.new
|
60
|
+
post.set_values(post_params)
|
61
|
+
post
|
62
|
+
end
|
63
|
+
|
64
|
+
def parse_array(element)
|
65
|
+
element.element_children.map do |child|
|
66
|
+
child.text
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
# TODO: Decide if a class or hash should be used...
|
71
|
+
def parse_coordinates(element)
|
72
|
+
return {} if element.children.empty?
|
73
|
+
|
74
|
+
{
|
75
|
+
latitude: element.at_xpath("latitude/text()"),
|
76
|
+
longitude: element.at_xpath("longitude/text()"),
|
77
|
+
}
|
78
|
+
end
|
79
|
+
|
80
|
+
def handle_failure(failure)
|
81
|
+
code = failure.attribute('code').value
|
82
|
+
message = failure.at_xpath('message').text
|
83
|
+
|
84
|
+
fail Error.from_api_response(code, message)
|
85
|
+
end
|
86
|
+
|
87
|
+
def handle_non_xml_document(document)
|
88
|
+
fail ServerError, "Failed to parse response: \"#{document}\""
|
89
|
+
end
|
90
|
+
|
91
|
+
def parse_time(time)
|
92
|
+
Time.parse(time)
|
93
|
+
end
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
@@ -0,0 +1,66 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
|
3
|
+
module Twingly
|
4
|
+
module LiveFeed
|
5
|
+
|
6
|
+
# A blog post
|
7
|
+
#
|
8
|
+
# @attr_reader [String] id the post ID (Twingly internal identification)
|
9
|
+
# @attr_reader [String] author the author of the blog post
|
10
|
+
# @attr_reader [String] url the post URL
|
11
|
+
# @attr_reader [String] title the post title
|
12
|
+
# @attr_reader [String] text the blog post text
|
13
|
+
# @attr_reader [String] language_code ISO two letter language code for the
|
14
|
+
# language that the post was written in
|
15
|
+
# @attr_reader [String] location_code ISO two letter country code for the
|
16
|
+
# location of the blog
|
17
|
+
# @attr_reader [Hash] coordinates a hash containing :latitude and :longitude
|
18
|
+
# from the post RSS
|
19
|
+
# @attr_reader [Array] links all links from the blog post to other resources
|
20
|
+
# @attr_reader [Array] tags the post tags/categories
|
21
|
+
# @attr_reader [Array] images image URLs from the post (currently not populated)
|
22
|
+
# @attr_reader [Time] indexed_at the time, in UTC, when the post was indexed by Twingly
|
23
|
+
# @attr_reader [Time] published_at the time, in UTC, when the post was published
|
24
|
+
# @attr_reader [Time] reindexed_at timestamp when the post last was changed in our database/index
|
25
|
+
# @attr_reader [String] inlinks_count number of links to this post that was found in other blog posts
|
26
|
+
# @attr_reader [String] blog_id the blog ID (Twingly internal identification)
|
27
|
+
# @attr_reader [String] blog_name the name of the blog
|
28
|
+
# @attr_reader [String] blog_url the blog URL
|
29
|
+
# @attr_reader [Integer] blog_rank the rank of the blog, based on authority and language.
|
30
|
+
# See https://developer.twingly.com/resources/ranking/#blogrank
|
31
|
+
# @attr_reader [Integer] authority the blog's authority/influence.
|
32
|
+
# See https://developer.twingly.com/resources/ranking/#authority
|
33
|
+
class Post
|
34
|
+
attr_reader :id, :author, :url, :title, :text, :location_code,
|
35
|
+
:language_code, :coordinates, :links, :tags, :images, :indexed_at,
|
36
|
+
:published_at, :reindexed_at, :inlinks_count, :blog_id, :blog_name,
|
37
|
+
:blog_url, :blog_rank, :authority
|
38
|
+
|
39
|
+
# Sets all instance variables for the {Post}, given a Hash.
|
40
|
+
#
|
41
|
+
# @param [Hash] params containing blog post data.
|
42
|
+
def set_values(params)
|
43
|
+
@id = params.fetch('id')
|
44
|
+
@author = params.fetch('author')
|
45
|
+
@url = params.fetch('url')
|
46
|
+
@title = params.fetch('title')
|
47
|
+
@text = params.fetch('text')
|
48
|
+
@language_code = params.fetch('languageCode')
|
49
|
+
@location_code = params.fetch('locationCode')
|
50
|
+
@coordinates = params.fetch('coordinates', {})
|
51
|
+
@links = params.fetch('links', [])
|
52
|
+
@tags = params.fetch('tags', [])
|
53
|
+
@images = params.fetch('images', [])
|
54
|
+
@indexed_at = Time.parse(params.fetch('indexedAt'))
|
55
|
+
@published_at = Time.parse(params.fetch('publishedAt'))
|
56
|
+
@reindexed_at = Time.parse(params.fetch('reindexedAt'))
|
57
|
+
@inlinks_count = params.fetch('inlinksCount').to_i
|
58
|
+
@blog_id = params.fetch('blogId')
|
59
|
+
@blog_name = params.fetch('blogName')
|
60
|
+
@blog_url = params.fetch('blogUrl')
|
61
|
+
@blog_rank = params.fetch('blogRank').to_i
|
62
|
+
@authority = params.fetch('authority').to_i
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
@@ -0,0 +1,39 @@
|
|
1
|
+
module Twingly
|
2
|
+
module LiveFeed
|
3
|
+
# Represents a result from a request to the LiveFeed API
|
4
|
+
#
|
5
|
+
# @see Client#next_result
|
6
|
+
# @attr [Time] ts the exact time when the result was built
|
7
|
+
# @attr [Time] from the timestamp that was sent in the request.
|
8
|
+
# See {Client#timestamp}
|
9
|
+
# @attr [Integer] number_of_posts the number of posts this result contains
|
10
|
+
# @attr [Integer] max_number_of_posts the maximum number of posts this result
|
11
|
+
# could contain. See {Client#max_posts}
|
12
|
+
# @attr [Time] first_post the index time for the first post in the result
|
13
|
+
# @attr [Time] last_post the index time for the last post in the result
|
14
|
+
# @attr [Time] next_timestamp the timestamp that should be used to get the
|
15
|
+
# next batch of posts
|
16
|
+
class Result
|
17
|
+
attr_accessor :ts, :from, :number_of_posts, :max_number_of_posts,
|
18
|
+
:first_post, :last_post, :next_timestamp
|
19
|
+
|
20
|
+
# @return [Array<Post>] all posts that matched the {Query}.
|
21
|
+
def posts
|
22
|
+
@posts ||= []
|
23
|
+
end
|
24
|
+
|
25
|
+
def inspect
|
26
|
+
instance_variables = "@posts, "
|
27
|
+
instance_variables << "@ts=#{self.ts}, "
|
28
|
+
instance_variables << "@from=#{self.from}, "
|
29
|
+
instance_variables << "@number_of_posts=#{self.number_of_posts}, "
|
30
|
+
instance_variables << "@max_number_of_posts=#{self.max_number_of_posts}, "
|
31
|
+
instance_variables << "@first_post=#{self.first_post}, "
|
32
|
+
instance_variables << "@last_post=#{self.last_post}"
|
33
|
+
instance_variables << "@next_timestamp=#{self.next_timestamp}"
|
34
|
+
|
35
|
+
sprintf("#<%s:0x%x %s>", self.class.name, __id__, instance_variables)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
@@ -6,8 +6,9 @@ module Twingly
|
|
6
6
|
class Client
|
7
7
|
attr_accessor :api_key, :user_agent
|
8
8
|
|
9
|
-
BASE_URL
|
10
|
-
|
9
|
+
BASE_URL = "https://api.twingly.com"
|
10
|
+
SEARCH_API_VERSION = "v3"
|
11
|
+
SEARCH_PATH = "/blog/search/api/#{SEARCH_API_VERSION}/search"
|
11
12
|
|
12
13
|
DEFAULT_USER_AGENT = "Twingly Search Ruby Client/#{VERSION}"
|
13
14
|
|
data/lib/twingly/search/error.rb
CHANGED
@@ -1,17 +1,18 @@
|
|
1
1
|
module Twingly
|
2
2
|
module Search
|
3
3
|
class Error < StandardError
|
4
|
-
|
5
|
-
# @return [Error] an instance of {AuthError} or {ServerError}.
|
6
|
-
def self.from_api_response_message(message)
|
4
|
+
def self.from_api_response(code, message)
|
7
5
|
error =
|
8
|
-
|
6
|
+
case code.to_s
|
7
|
+
when /^400/, /^404/
|
8
|
+
QueryError
|
9
|
+
when /^401/, /^402/
|
9
10
|
AuthError
|
10
11
|
else
|
11
12
|
ServerError
|
12
13
|
end
|
13
14
|
|
14
|
-
error.new(message)
|
15
|
+
error.new("#{message} (code: #{code})")
|
15
16
|
end
|
16
17
|
end
|
17
18
|
|