twingly-search 5.0.1 → 5.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.travis.yml +1 -2
- data/CHANGELOG.md +16 -0
- data/README.md +33 -34
- data/Rakefile +0 -6
- data/examples/find_all_posts_mentioning_github.rb +3 -3
- data/examples/hello_world.rb +2 -2
- data/examples/livefeed_loop.rb +24 -0
- data/lib/twingly/livefeed/client.rb +121 -0
- data/lib/twingly/livefeed/error.rb +28 -0
- data/lib/twingly/livefeed/parser.rb +96 -0
- data/lib/twingly/livefeed/post.rb +66 -0
- data/lib/twingly/livefeed/result.rb +39 -0
- data/lib/twingly/livefeed/version.rb +5 -0
- data/lib/twingly/livefeed.rb +6 -0
- data/lib/twingly/search/client.rb +3 -2
- data/lib/twingly/search/error.rb +6 -5
- data/lib/twingly/search/parser.rb +39 -13
- data/lib/twingly/search/post.rb +65 -21
- data/lib/twingly/search/query.rb +46 -16
- data/lib/twingly/search/result.rb +11 -0
- data/lib/twingly/search/version.rb +1 -1
- data/spec/client_spec.rb +2 -2
- data/spec/error_spec.rb +27 -7
- data/spec/fixtures/incomplete_result.xml +2 -0
- data/spec/fixtures/livefeed/empty_api_key_result.xml +3 -0
- data/spec/fixtures/livefeed/non_xml_result.xml +1 -0
- data/spec/fixtures/livefeed/not_found_result.xml +3 -0
- data/spec/fixtures/livefeed/service_unavailable_result.xml +3 -0
- data/spec/fixtures/livefeed/unauthorized_api_key_result.xml +3 -0
- data/spec/fixtures/livefeed/valid_empty_result.xml +2 -0
- data/spec/fixtures/livefeed/valid_result.xml +79 -0
- data/spec/fixtures/minimal_valid_result.xml +81 -52
- data/spec/fixtures/nonexistent_api_key_result.xml +3 -3
- data/spec/fixtures/service_unavailable_result.xml +3 -3
- data/spec/fixtures/unauthorized_api_key_result.xml +3 -3
- data/spec/fixtures/undefined_error_result.xml +3 -3
- data/spec/fixtures/valid_empty_result.xml +2 -2
- data/spec/fixtures/valid_links_result.xml +36 -0
- data/spec/fixtures/vcr_cassettes/livefeed_valid_request.yml +169 -0
- data/spec/fixtures/vcr_cassettes/search_for_spotify_on_sv_blogs.yml +578 -447
- data/spec/fixtures/vcr_cassettes/search_without_valid_api_key.yml +15 -14
- data/spec/livefeed/client_spec.rb +135 -0
- data/spec/livefeed/error_spec.rb +51 -0
- data/spec/livefeed/parser_spec.rb +351 -0
- data/spec/livefeed/post_spec.rb +26 -0
- data/spec/livefeed/result_spec.rb +18 -0
- data/spec/parser_spec.rb +191 -94
- data/spec/post_spec.rb +25 -6
- data/spec/query_spec.rb +41 -34
- data/spec/result_spec.rb +1 -0
- data/spec/spec_helper.rb +10 -0
- data/twingly-search-api-ruby.gemspec +2 -3
- metadata +44 -24
- data/spec/fixtures/valid_non_blog_result.xml +0 -26
- data/spec/fixtures/valid_result.xml +0 -22975
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3c9c84cb7c69272bfd499b2901f475082d4dc251
|
4
|
+
data.tar.gz: d51285306f3bc40452016e81bee6a6b22f783c59
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 407a46a73b5bff7aed5277240f45bf7be611cafc2b64b8b9a22de2b019be3f3cd49d1da236c4ecdd5a6a221e0a82e54482ba60c40ca4babae33f0e6e0f93b389
|
7
|
+
data.tar.gz: 7f64f9eec9386722f72f13701f473d687680016f2bf303be68007a30b116872b61e8790ed6a43c8ff3ba7702edc81baecf31f6b0c7cd4850bf07070da439796b
|
data/.travis.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,21 @@
|
|
1
1
|
# Change Log
|
2
2
|
|
3
|
+
## [v5.0.1](https://github.com/twingly/twingly-search-api-ruby/tree/v5.0.1) (2016-03-03)
|
4
|
+
[Full Changelog](https://github.com/twingly/twingly-search-api-ruby/compare/v5.0.0...v5.0.1)
|
5
|
+
|
6
|
+
**Fixed bugs:**
|
7
|
+
|
8
|
+
- Not possible to remove start\_time from query [\#53](https://github.com/twingly/twingly-search-api-ruby/issues/53)
|
9
|
+
|
10
|
+
**Closed issues:**
|
11
|
+
|
12
|
+
- New release [\#48](https://github.com/twingly/twingly-search-api-ruby/issues/48)
|
13
|
+
|
14
|
+
**Merged pull requests:**
|
15
|
+
|
16
|
+
- Be able to remove start/end time from Query [\#54](https://github.com/twingly/twingly-search-api-ruby/pull/54) ([dentarg](https://github.com/dentarg))
|
17
|
+
- Actually test what we intend to test [\#52](https://github.com/twingly/twingly-search-api-ruby/pull/52) ([dentarg](https://github.com/dentarg))
|
18
|
+
|
3
19
|
## [v5.0.0](https://github.com/twingly/twingly-search-api-ruby/tree/v5.0.0) (2016-02-17)
|
4
20
|
[Full Changelog](https://github.com/twingly/twingly-search-api-ruby/compare/v4.0.1...v5.0.0)
|
5
21
|
|
data/README.md
CHANGED
@@ -3,7 +3,7 @@
|
|
3
3
|
[](https://travis-ci.org/twingly/twingly-search-api-ruby)
|
4
4
|
[](https://codeclimate.com/github/twingly/twingly-search-api-ruby)
|
5
5
|
|
6
|
-
A Ruby gem for Twingly's Search API (previously known as Analytics API). Twingly is a blog search service that provides a searchable API known as [Twingly Search API]
|
6
|
+
A Ruby gem for Twingly's Blog Search API (previously known as Analytics API) and Blog LiveFeed API. Twingly is a blog search service that provides a searchable API known as [Twingly Blog Search API][Blog Search API documentation] and a blog data firehose called [Twingly Blog LiveFeed API][Blog LiveFeed API documentation].
|
7
7
|
|
8
8
|
## Installation
|
9
9
|
|
@@ -21,6 +21,18 @@ gem "twingly-search"
|
|
21
21
|
|
22
22
|
## Usage
|
23
23
|
|
24
|
+
The `twingly-search` gem talks to a commercial API and requires an API key. Best practice is to set the `TWINGLY_SEARCH_KEY` environment variable to the obtained key. `Twingly::Search::Client` and `Twingly::LiveFeed::Client` can be passed a key at initialization if your setup does not allow environment variables.
|
25
|
+
|
26
|
+
To learn more about the features of this gem, read the [gem documentation] or check out the example code that can be found in [examples/](examples/).
|
27
|
+
|
28
|
+
To learn more about the capabilities of Twingly's APIs, please read the [Blog Search API documentation] and [Blog LiveFeed API documentation].
|
29
|
+
|
30
|
+
[gem documentation]: http://www.rubydoc.info/github/twingly/twingly-search-api-ruby
|
31
|
+
[Blog Search API documentation]: https://developer.twingly.com/resources/search/
|
32
|
+
[Blog LiveFeed API documentation]: https://developer.twingly.com/resources/livefeed/
|
33
|
+
|
34
|
+
### Blog Search API
|
35
|
+
|
24
36
|
```ruby
|
25
37
|
require "twingly/search"
|
26
38
|
|
@@ -29,8 +41,8 @@ client = Twingly::Search::Client.new do |client|
|
|
29
41
|
end
|
30
42
|
|
31
43
|
query = client.query do |query|
|
32
|
-
query.
|
33
|
-
query.language
|
44
|
+
query.search_query = "github page-size:10"
|
45
|
+
query.language = "sv"
|
34
46
|
end
|
35
47
|
|
36
48
|
result = query.execute
|
@@ -38,20 +50,30 @@ result = query.execute
|
|
38
50
|
result.posts # will include all returned posts
|
39
51
|
```
|
40
52
|
|
41
|
-
|
53
|
+
### Blog LiveFeed API
|
42
54
|
|
43
|
-
|
55
|
+
```ruby
|
56
|
+
require "twingly/livefeed"
|
44
57
|
|
45
|
-
|
58
|
+
client = Twingly::LiveFeed::Client.new do |client|
|
59
|
+
client.user_agent = "MyCompany/1.0"
|
60
|
+
# Start getting posts indexed by Twingly at this timestamp
|
61
|
+
client.timestamp = Time.now - 3600 # 1 hour ago
|
62
|
+
client.max_posts = 1000 # Maximum number of posts returned per call
|
63
|
+
end
|
46
64
|
|
47
|
-
|
48
|
-
|
65
|
+
# get the next chunk of posts
|
66
|
+
result = client.next_result
|
67
|
+
=> #<Twingly::LiveFeed::Result:0x3fcd98215c14 @posts, @ts=2017-04-18 14:42:18 UTC, @from=2017-04-18 13:42:06 UTC, @number_of_posts=989, @max_number_of_posts=1000, @first_post=2017-04-18 13:42:19 UTC, @last_post=2017-04-18 14:42:13 UTC>
|
68
|
+
|
69
|
+
result.posts # will include all returned posts
|
70
|
+
```
|
49
71
|
|
50
72
|
## Requirements
|
51
73
|
|
52
|
-
* API key,
|
74
|
+
* API key, [sign up](https://www.twingly.com/try-for-free) via [twingly.com](https://www.twingly.com/) to get one
|
53
75
|
* Ruby
|
54
|
-
* Ruby
|
76
|
+
* Ruby 2.1, 2.2, 2.3
|
55
77
|
* JRuby 9000
|
56
78
|
|
57
79
|
## Development
|
@@ -70,7 +92,7 @@ Run the tests
|
|
70
92
|
|
71
93
|
1. Make a commit bumping the version in `lib/twingly/search/version.rb`, follow [Semantic Versioning 2.0.0](http://semver.org/). No need to push as this will be taken care of automatically in the next step.
|
72
94
|
1. Build and the release gem with `bundle exec rake release`. This will create a git tag for the version and push the `.gem` file to [RubyGems.org].
|
73
|
-
1. Generate a changelog with `
|
95
|
+
1. Generate a changelog with `github_changelog_generator` (`gem install github_changelog_generator` if you don't have it). Set `CHANGELOG_GITHUB_TOKEN` to a personal access token to increase the API rate limit. (The changelog uses [GitHub Changelog Generator](https://github.com/skywinder/github-changelog-generator/))
|
74
96
|
1. Update release information on the [releases page]. This is done manually.
|
75
97
|
|
76
98
|
[releases page]: https://github.com/twingly/twingly-search-api-ruby/releases
|
@@ -85,26 +107,3 @@ This gem is documented using [YARD]. To start a local YARD server run:
|
|
85
107
|
The YARD server reloads the documentation automatically so there is no need to restart it when making changes.
|
86
108
|
|
87
109
|
[YARD]: http://yardoc.org/
|
88
|
-
|
89
|
-
## License
|
90
|
-
|
91
|
-
The MIT License (MIT)
|
92
|
-
|
93
|
-
Copyright (c) 2013-2016 Twingly AB
|
94
|
-
|
95
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
96
|
-
this software and associated documentation files (the "Software"), to deal in
|
97
|
-
the Software without restriction, including without limitation the rights to
|
98
|
-
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
|
99
|
-
the Software, and to permit persons to whom the Software is furnished to do so,
|
100
|
-
subject to the following conditions:
|
101
|
-
|
102
|
-
The above copyright notice and this permission notice shall be included in all
|
103
|
-
copies or substantial portions of the Software.
|
104
|
-
|
105
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
106
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
107
|
-
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
108
|
-
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
109
|
-
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
110
|
-
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/Rakefile
CHANGED
@@ -10,12 +10,6 @@ task :spec do
|
|
10
10
|
end
|
11
11
|
end
|
12
12
|
|
13
|
-
require "github_changelog_generator/task"
|
14
|
-
GitHubChangelogGenerator::RakeTask.new(:changelog) do |config|
|
15
|
-
config.user = "twingly"
|
16
|
-
config.project = "twingly-search-api-ruby"
|
17
|
-
end
|
18
|
-
|
19
13
|
namespace :yard do
|
20
14
|
require "yard"
|
21
15
|
require "yard/rake/yardoc_task"
|
@@ -8,8 +8,8 @@ class SearchPostStream
|
|
8
8
|
end
|
9
9
|
|
10
10
|
@query = client.query do |query|
|
11
|
-
query.language
|
12
|
-
query.
|
11
|
+
query.language = language
|
12
|
+
query.search_query = "sort-order:asc sort:published #{keyword}"
|
13
13
|
end
|
14
14
|
end
|
15
15
|
|
@@ -25,7 +25,7 @@ class SearchPostStream
|
|
25
25
|
|
26
26
|
break if result.all_results_returned?
|
27
27
|
|
28
|
-
@query.start_time = result.posts.last.
|
28
|
+
@query.start_time = result.posts.last.published_at
|
29
29
|
end
|
30
30
|
end
|
31
31
|
|
data/examples/hello_world.rb
CHANGED
@@ -3,8 +3,8 @@ Bundler.require
|
|
3
3
|
# Set environment variable TWINGLY_SEARCH_KEY
|
4
4
|
client = Twingly::Search::Client.new
|
5
5
|
query = client.query do |query|
|
6
|
-
query.
|
7
|
-
query.start_time
|
6
|
+
query.search_query = '"hello world"'
|
7
|
+
query.start_time = Time.now - (24 * 3600) # search last day
|
8
8
|
end
|
9
9
|
|
10
10
|
result = query.execute
|
@@ -0,0 +1,24 @@
|
|
1
|
+
Bundler.require
|
2
|
+
|
3
|
+
require "twingly/livefeed"
|
4
|
+
|
5
|
+
# Set environment variable TWINGLY_SEARCH_KEY
|
6
|
+
client = Twingly::LiveFeed::Client.new do |client|
|
7
|
+
# get posts published less than an hour ago
|
8
|
+
client.timestamp = (Time.now - 3600)
|
9
|
+
client.max_posts = 4000
|
10
|
+
end
|
11
|
+
|
12
|
+
loop do
|
13
|
+
result = client.next_result
|
14
|
+
|
15
|
+
result.posts.each do |post|
|
16
|
+
puts "#{post.indexed_at} - #{post.url}"
|
17
|
+
end
|
18
|
+
|
19
|
+
# You're faster than Twingly! Take a break and wait for
|
20
|
+
# us to index some new posts before asking again
|
21
|
+
if result.number_of_posts < result.max_number_of_posts
|
22
|
+
sleep 20
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,121 @@
|
|
1
|
+
require "faraday"
|
2
|
+
require "time"
|
3
|
+
|
4
|
+
module Twingly
|
5
|
+
module LiveFeed
|
6
|
+
# Twingly LiveFeed API client
|
7
|
+
#
|
8
|
+
# @attr [String] api_key the API key
|
9
|
+
# @attr [String] user_agent the user agent to be used for all API requests
|
10
|
+
# @attr [Integer] max_posts the maximum number of posts that each request can return
|
11
|
+
# @attr [Time] timestamp the timestamp that will be used in the next request
|
12
|
+
class Client
|
13
|
+
attr_accessor :api_key, :user_agent, :max_posts, :timestamp
|
14
|
+
|
15
|
+
BASE_URL = "https://api.twingly.com"
|
16
|
+
LIVEFEED_API_VERSION = "v5"
|
17
|
+
LIVEFEED_PATH = "/blog/livefeed/api/#{LIVEFEED_API_VERSION}/getdata"
|
18
|
+
|
19
|
+
DEFAULT_USER_AGENT = "Twingly LiveFeed Ruby Client/#{VERSION}"
|
20
|
+
DEFAULT_MAX_POSTS = 1000
|
21
|
+
TIMESTAMP_FORMAT = "%Y-%m-%dT%H:%M:%S.%3N%z"
|
22
|
+
|
23
|
+
# Creates a new Twingly Search API client
|
24
|
+
#
|
25
|
+
# @param api_key [String] the API key provided by Twingly.
|
26
|
+
# If nil, reads key from environment (TWINGLY_SEARCH_KEY).
|
27
|
+
# @param options [Hash]
|
28
|
+
# @option options [String] :user_agent the user agent to be used
|
29
|
+
# for all requests
|
30
|
+
# @option options [String] :max_posts the maximum number of posts that can
|
31
|
+
# be returned for each request
|
32
|
+
# @option options [String] :timestamp the timestamp to start the client at
|
33
|
+
# @raise [AuthError] if an API key is not set
|
34
|
+
def initialize(api_key = nil, options = {})
|
35
|
+
@api_key = api_key
|
36
|
+
@user_agent = options.fetch(:user_agent) { DEFAULT_USER_AGENT }
|
37
|
+
@max_posts = options.fetch(:max_posts) { DEFAULT_MAX_POSTS }
|
38
|
+
@timestamp = options.fetch(:timestamp) { Time.now }
|
39
|
+
|
40
|
+
yield self if block_given?
|
41
|
+
|
42
|
+
@api_key ||= env_api_key || api_key_missing
|
43
|
+
end
|
44
|
+
|
45
|
+
# Get the next result from the API and updates the next timestamp
|
46
|
+
#
|
47
|
+
# Sends a request to the API using the timestamp set with {#timestamp},
|
48
|
+
# updates the timestamp that should be used in the next request and
|
49
|
+
# then returns the result.
|
50
|
+
#
|
51
|
+
# @raise [QueryError] if the timestamp is invalid.
|
52
|
+
# @raise [AuthError] if the API couldn't authenticate you. Make sure your API key is correct.
|
53
|
+
# @raise [ServerError] if the query could not be executed due to a server error.
|
54
|
+
# @return [Result] the result for this request.
|
55
|
+
def next_result
|
56
|
+
assert_valid_time(timestamp)
|
57
|
+
|
58
|
+
get_and_parse_result.tap do |result|
|
59
|
+
update_timestamp(result.next_timestamp)
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
# @return [String] the request url for the next request.
|
64
|
+
def url
|
65
|
+
"#{endpoint_url}?#{url_parameters}"
|
66
|
+
end
|
67
|
+
|
68
|
+
# @return [String] the API endpoint URL.
|
69
|
+
def endpoint_url
|
70
|
+
"#{BASE_URL}#{LIVEFEED_PATH}"
|
71
|
+
end
|
72
|
+
|
73
|
+
# @see #url
|
74
|
+
# @return [String] the query part of the request url.
|
75
|
+
def url_parameters
|
76
|
+
Faraday::Utils.build_query(request_parameters)
|
77
|
+
end
|
78
|
+
|
79
|
+
# @return [Hash] the request parameters.
|
80
|
+
def request_parameters
|
81
|
+
{
|
82
|
+
apikey: api_key,
|
83
|
+
timestamp: timestamp.to_time.utc.strftime(TIMESTAMP_FORMAT),
|
84
|
+
maxPosts: max_posts,
|
85
|
+
}
|
86
|
+
end
|
87
|
+
|
88
|
+
private
|
89
|
+
|
90
|
+
def update_timestamp(timestamp)
|
91
|
+
@timestamp = timestamp
|
92
|
+
end
|
93
|
+
|
94
|
+
def get_and_parse_result
|
95
|
+
response_body = get_response.body
|
96
|
+
Parser.new.parse(response_body)
|
97
|
+
end
|
98
|
+
|
99
|
+
def get_response
|
100
|
+
connection = Faraday.new(url: BASE_URL) do |faraday|
|
101
|
+
faraday.request :url_encoded
|
102
|
+
faraday.adapter Faraday.default_adapter
|
103
|
+
end
|
104
|
+
connection.headers[:user_agent] = user_agent
|
105
|
+
connection.get(LIVEFEED_PATH, request_parameters)
|
106
|
+
end
|
107
|
+
|
108
|
+
def env_api_key
|
109
|
+
ENV['TWINGLY_SEARCH_KEY']
|
110
|
+
end
|
111
|
+
|
112
|
+
def api_key_missing
|
113
|
+
fail AuthError, "No API key has been provided."
|
114
|
+
end
|
115
|
+
|
116
|
+
def assert_valid_time(time)
|
117
|
+
fail QueryError, "Not a Time object" unless time.respond_to?(:to_time)
|
118
|
+
end
|
119
|
+
end
|
120
|
+
end
|
121
|
+
end
|
@@ -0,0 +1,28 @@
|
|
1
|
+
module Twingly
|
2
|
+
module LiveFeed
|
3
|
+
class Error < StandardError
|
4
|
+
def self.from_api_response(code, message)
|
5
|
+
error =
|
6
|
+
case code.to_s
|
7
|
+
when /^400/, /^404/
|
8
|
+
QueryError
|
9
|
+
when /^401/
|
10
|
+
AuthError
|
11
|
+
else
|
12
|
+
ServerError
|
13
|
+
end
|
14
|
+
|
15
|
+
error.new("#{message} (code: #{code})")
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
class AuthError < Error
|
20
|
+
end
|
21
|
+
|
22
|
+
class ServerError < Error
|
23
|
+
end
|
24
|
+
|
25
|
+
class QueryError < Error
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
@@ -0,0 +1,96 @@
|
|
1
|
+
require 'nokogiri'
|
2
|
+
|
3
|
+
module Twingly
|
4
|
+
module LiveFeed
|
5
|
+
class Parser
|
6
|
+
# Parse an API response body.
|
7
|
+
#
|
8
|
+
# @param [String] document containing an API response XML.
|
9
|
+
# @raise [Error] which error depends on the API response (see {Error.from_api_response}).
|
10
|
+
# @return [Result] containing the result.
|
11
|
+
def parse(document)
|
12
|
+
nokogiri = Nokogiri::XML(document)
|
13
|
+
|
14
|
+
failure = nokogiri.at_xpath('/error')
|
15
|
+
handle_failure(failure) if failure
|
16
|
+
|
17
|
+
data_node = nokogiri.at_xpath('/twinglydata')
|
18
|
+
handle_non_xml_document(document) unless data_node
|
19
|
+
|
20
|
+
create_result(data_node)
|
21
|
+
end
|
22
|
+
|
23
|
+
private
|
24
|
+
|
25
|
+
def create_result(data_node)
|
26
|
+
result = Result.new
|
27
|
+
result.ts = parse_time(data_node.attribute('ts').value)
|
28
|
+
result.from = parse_time(data_node.attribute('from').value)
|
29
|
+
result.number_of_posts = data_node.attribute('numberOfPosts').value.to_i
|
30
|
+
result.max_number_of_posts = data_node.attribute('maxNumberOfPosts').value.to_i
|
31
|
+
result.next_timestamp = parse_time(data_node.attribute('nextTimestamp').value)
|
32
|
+
|
33
|
+
unless result.number_of_posts.zero?
|
34
|
+
result.first_post = parse_time(data_node.attribute('firstPost').value)
|
35
|
+
result.last_post = parse_time(data_node.attribute('lastPost').value)
|
36
|
+
end
|
37
|
+
|
38
|
+
data_node.xpath('//post').each do |post|
|
39
|
+
result.posts << parse_post(post)
|
40
|
+
end
|
41
|
+
|
42
|
+
result
|
43
|
+
end
|
44
|
+
|
45
|
+
def parse_post(element)
|
46
|
+
post_params = {}
|
47
|
+
element.element_children.each do |child|
|
48
|
+
post_params[child.name] =
|
49
|
+
case child.name
|
50
|
+
when *%w(tags links images)
|
51
|
+
parse_array(child)
|
52
|
+
when "coordinates"
|
53
|
+
parse_coordinates(child)
|
54
|
+
else
|
55
|
+
child.text
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
post = Post.new
|
60
|
+
post.set_values(post_params)
|
61
|
+
post
|
62
|
+
end
|
63
|
+
|
64
|
+
def parse_array(element)
|
65
|
+
element.element_children.map do |child|
|
66
|
+
child.text
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
# TODO: Decide if a class or hash should be used...
|
71
|
+
def parse_coordinates(element)
|
72
|
+
return {} if element.children.empty?
|
73
|
+
|
74
|
+
{
|
75
|
+
latitude: element.at_xpath("latitude/text()"),
|
76
|
+
longitude: element.at_xpath("longitude/text()"),
|
77
|
+
}
|
78
|
+
end
|
79
|
+
|
80
|
+
def handle_failure(failure)
|
81
|
+
code = failure.attribute('code').value
|
82
|
+
message = failure.at_xpath('message').text
|
83
|
+
|
84
|
+
fail Error.from_api_response(code, message)
|
85
|
+
end
|
86
|
+
|
87
|
+
def handle_non_xml_document(document)
|
88
|
+
fail ServerError, "Failed to parse response: \"#{document}\""
|
89
|
+
end
|
90
|
+
|
91
|
+
def parse_time(time)
|
92
|
+
Time.parse(time)
|
93
|
+
end
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
@@ -0,0 +1,66 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
|
3
|
+
module Twingly
|
4
|
+
module LiveFeed
|
5
|
+
|
6
|
+
# A blog post
|
7
|
+
#
|
8
|
+
# @attr_reader [String] id the post ID (Twingly internal identification)
|
9
|
+
# @attr_reader [String] author the author of the blog post
|
10
|
+
# @attr_reader [String] url the post URL
|
11
|
+
# @attr_reader [String] title the post title
|
12
|
+
# @attr_reader [String] text the blog post text
|
13
|
+
# @attr_reader [String] language_code ISO two letter language code for the
|
14
|
+
# language that the post was written in
|
15
|
+
# @attr_reader [String] location_code ISO two letter country code for the
|
16
|
+
# location of the blog
|
17
|
+
# @attr_reader [Hash] coordinates a hash containing :latitude and :longitude
|
18
|
+
# from the post RSS
|
19
|
+
# @attr_reader [Array] links all links from the blog post to other resources
|
20
|
+
# @attr_reader [Array] tags the post tags/categories
|
21
|
+
# @attr_reader [Array] images image URLs from the post (currently not populated)
|
22
|
+
# @attr_reader [Time] indexed_at the time, in UTC, when the post was indexed by Twingly
|
23
|
+
# @attr_reader [Time] published_at the time, in UTC, when the post was published
|
24
|
+
# @attr_reader [Time] reindexed_at timestamp when the post last was changed in our database/index
|
25
|
+
# @attr_reader [String] inlinks_count number of links to this post that was found in other blog posts
|
26
|
+
# @attr_reader [String] blog_id the blog ID (Twingly internal identification)
|
27
|
+
# @attr_reader [String] blog_name the name of the blog
|
28
|
+
# @attr_reader [String] blog_url the blog URL
|
29
|
+
# @attr_reader [Integer] blog_rank the rank of the blog, based on authority and language.
|
30
|
+
# See https://developer.twingly.com/resources/ranking/#blogrank
|
31
|
+
# @attr_reader [Integer] authority the blog's authority/influence.
|
32
|
+
# See https://developer.twingly.com/resources/ranking/#authority
|
33
|
+
class Post
|
34
|
+
attr_reader :id, :author, :url, :title, :text, :location_code,
|
35
|
+
:language_code, :coordinates, :links, :tags, :images, :indexed_at,
|
36
|
+
:published_at, :reindexed_at, :inlinks_count, :blog_id, :blog_name,
|
37
|
+
:blog_url, :blog_rank, :authority
|
38
|
+
|
39
|
+
# Sets all instance variables for the {Post}, given a Hash.
|
40
|
+
#
|
41
|
+
# @param [Hash] params containing blog post data.
|
42
|
+
def set_values(params)
|
43
|
+
@id = params.fetch('id')
|
44
|
+
@author = params.fetch('author')
|
45
|
+
@url = params.fetch('url')
|
46
|
+
@title = params.fetch('title')
|
47
|
+
@text = params.fetch('text')
|
48
|
+
@language_code = params.fetch('languageCode')
|
49
|
+
@location_code = params.fetch('locationCode')
|
50
|
+
@coordinates = params.fetch('coordinates', {})
|
51
|
+
@links = params.fetch('links', [])
|
52
|
+
@tags = params.fetch('tags', [])
|
53
|
+
@images = params.fetch('images', [])
|
54
|
+
@indexed_at = Time.parse(params.fetch('indexedAt'))
|
55
|
+
@published_at = Time.parse(params.fetch('publishedAt'))
|
56
|
+
@reindexed_at = Time.parse(params.fetch('reindexedAt'))
|
57
|
+
@inlinks_count = params.fetch('inlinksCount').to_i
|
58
|
+
@blog_id = params.fetch('blogId')
|
59
|
+
@blog_name = params.fetch('blogName')
|
60
|
+
@blog_url = params.fetch('blogUrl')
|
61
|
+
@blog_rank = params.fetch('blogRank').to_i
|
62
|
+
@authority = params.fetch('authority').to_i
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
@@ -0,0 +1,39 @@
|
|
1
|
+
module Twingly
|
2
|
+
module LiveFeed
|
3
|
+
# Represents a result from a request to the LiveFeed API
|
4
|
+
#
|
5
|
+
# @see Client#next_result
|
6
|
+
# @attr [Time] ts the exact time when the result was built
|
7
|
+
# @attr [Time] from the timestamp that was sent in the request.
|
8
|
+
# See {Client#timestamp}
|
9
|
+
# @attr [Integer] number_of_posts the number of posts this result contains
|
10
|
+
# @attr [Integer] max_number_of_posts the maximum number of posts this result
|
11
|
+
# could contain. See {Client#max_posts}
|
12
|
+
# @attr [Time] first_post the index time for the first post in the result
|
13
|
+
# @attr [Time] last_post the index time for the last post in the result
|
14
|
+
# @attr [Time] next_timestamp the timestamp that should be used to get the
|
15
|
+
# next batch of posts
|
16
|
+
class Result
|
17
|
+
attr_accessor :ts, :from, :number_of_posts, :max_number_of_posts,
|
18
|
+
:first_post, :last_post, :next_timestamp
|
19
|
+
|
20
|
+
# @return [Array<Post>] all posts that matched the {Query}.
|
21
|
+
def posts
|
22
|
+
@posts ||= []
|
23
|
+
end
|
24
|
+
|
25
|
+
def inspect
|
26
|
+
instance_variables = "@posts, "
|
27
|
+
instance_variables << "@ts=#{self.ts}, "
|
28
|
+
instance_variables << "@from=#{self.from}, "
|
29
|
+
instance_variables << "@number_of_posts=#{self.number_of_posts}, "
|
30
|
+
instance_variables << "@max_number_of_posts=#{self.max_number_of_posts}, "
|
31
|
+
instance_variables << "@first_post=#{self.first_post}, "
|
32
|
+
instance_variables << "@last_post=#{self.last_post}"
|
33
|
+
instance_variables << "@next_timestamp=#{self.next_timestamp}"
|
34
|
+
|
35
|
+
sprintf("#<%s:0x%x %s>", self.class.name, __id__, instance_variables)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
@@ -6,8 +6,9 @@ module Twingly
|
|
6
6
|
class Client
|
7
7
|
attr_accessor :api_key, :user_agent
|
8
8
|
|
9
|
-
BASE_URL
|
10
|
-
|
9
|
+
BASE_URL = "https://api.twingly.com"
|
10
|
+
SEARCH_API_VERSION = "v3"
|
11
|
+
SEARCH_PATH = "/blog/search/api/#{SEARCH_API_VERSION}/search"
|
11
12
|
|
12
13
|
DEFAULT_USER_AGENT = "Twingly Search Ruby Client/#{VERSION}"
|
13
14
|
|
data/lib/twingly/search/error.rb
CHANGED
@@ -1,17 +1,18 @@
|
|
1
1
|
module Twingly
|
2
2
|
module Search
|
3
3
|
class Error < StandardError
|
4
|
-
|
5
|
-
# @return [Error] an instance of {AuthError} or {ServerError}.
|
6
|
-
def self.from_api_response_message(message)
|
4
|
+
def self.from_api_response(code, message)
|
7
5
|
error =
|
8
|
-
|
6
|
+
case code.to_s
|
7
|
+
when /^400/, /^404/
|
8
|
+
QueryError
|
9
|
+
when /^401/, /^402/
|
9
10
|
AuthError
|
10
11
|
else
|
11
12
|
ServerError
|
12
13
|
end
|
13
14
|
|
14
|
-
error.new(message)
|
15
|
+
error.new("#{message} (code: #{code})")
|
15
16
|
end
|
16
17
|
end
|
17
18
|
|