howlongtobeat 0.2.5 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +29 -0
- data/README.md +28 -2
- data/lib/howlongtobeat/client.rb +178 -0
- data/lib/howlongtobeat/errors.rb +21 -0
- data/lib/howlongtobeat/fields.rb +46 -0
- data/lib/howlongtobeat/game_page_parser.rb +37 -0
- data/lib/howlongtobeat/how_long_to_beat.rb +45 -16
- data/lib/howlongtobeat/html_requests.rb +1 -1
- data/lib/howlongtobeat/http.rb +87 -0
- data/lib/howlongtobeat/records.rb +17 -0
- data/lib/howlongtobeat/search_result_parser.rb +33 -0
- data/lib/howlongtobeat/version.rb +1 -1
- data/lib/howlongtobeat.rb +7 -4
- metadata +23 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 460063aabeca602d857275839c6454dd8c0fb8d4eaa17f8e0a986c08d72cb3f3
|
|
4
|
+
data.tar.gz: a45bd306c0fb36cc2006b1bdde32d5b361cce1eb405e8f71ce31ef4aabd32dfa
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: cf147ef1c998ee37ce46a1d676614fa233617555a9095bc89f12f24f6ef2619a2f9340781fc866a500b45c1500c4efcc8a638cd26f3bc02b0a12a3be49c8fb57
|
|
7
|
+
data.tar.gz: 18ff0a91a30dfa9b5d0380935c5d2053d586dfdb253da52a72f40e571f793ee8d7cb9e8075a290ec6e03b24a27c58fce040c8cde0f27391b5bb21559ad3d1c7b
|
data/CHANGELOG.md
CHANGED
|
@@ -25,6 +25,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
25
25
|
### Security
|
|
26
26
|
- None
|
|
27
27
|
|
|
28
|
+
## [0.3.0] - 2026-10-05
|
|
29
|
+
|
|
30
|
+
HowLongToBeat started answering 429 to bursts of requests in late September 2026, which is why this release paces them.
|
|
31
|
+
|
|
32
|
+
### Added
|
|
33
|
+
- `HowLongToBeat::Client`: a stateful client meant to be kept for the life of a process.
|
|
34
|
+
- `#game(id)` reads a game with one request to its HLTB page. It returns a `GameDetail`: times in hours, release date, aliases and Steam app id.
|
|
35
|
+
- `#search(title)` returns raw `SearchResult` rows; `#search_json(title)` returns the raw body.
|
|
36
|
+
- `HowLongToBeat::Client.default`: a lazily built, process-wide client. The legacy `HowLongToBeat::HowLongToBeat` API uses it unless you pass `client:`, so instances no longer each pay for a full endpoint discovery.
|
|
37
|
+
- Distinct errors: `RateLimitedError` (429), `RequestError` (network, timeout, 5xx, 403, a 404 that is not HLTB's own; `#status` carries the HTTP status) and `ParseError` (unexpected page or response shape, a `/init` response without a token, or a search endpoint that cannot be found).
|
|
38
|
+
|
|
39
|
+
### Changed
|
|
40
|
+
- Requests are paced: at least 2 seconds apart by default.
|
|
41
|
+
- The search endpoint is cached for an hour and the `/init` token for 60 seconds, with a 1.2 second pause before the first search with a new token.
|
|
42
|
+
- `HowLongToBeat#search` and `#search_from_id` now run on `Client`. `search_from_id` takes one request instead of about eight.
|
|
43
|
+
- One current, fixed browser User-Agent.
|
|
44
|
+
- Explicit 5-second connect and 10-second read timeouts.
|
|
45
|
+
- `HowLongToBeat#search_from_id` now builds its entry from the game page: it sets id, name, alias, type, web link, release year, the six times and similarity (1.0); image URL, review score, developer, platforms, JSON content and complexity flags stay unset, and `auto_filter_times` no longer applies to it.
|
|
46
|
+
|
|
47
|
+
### Deprecated
|
|
48
|
+
- The public `HTMLRequests.send_web_request` and `HTMLRequests.get_game_title` are unpaced legacy helpers. Use `Client` instead.
|
|
49
|
+
|
|
50
|
+
### Removed
|
|
51
|
+
- After a 429 the client no longer cycles through retired endpoints.
|
|
52
|
+
- The fallback that disabled SSL certificate verification is gone (applies to the new client).
|
|
53
|
+
|
|
54
|
+
### Fixed
|
|
55
|
+
- `HowLongToBeat#search` and `#search_from_id` return nil for malformed responses and non-numeric ids instead of raising.
|
|
56
|
+
|
|
28
57
|
## [0.2.5] - 2026-09-04
|
|
29
58
|
|
|
30
59
|
### Fixed
|
data/README.md
CHANGED
|
@@ -27,6 +27,30 @@ $ gem install howlongtobeat
|
|
|
27
27
|
|
|
28
28
|
## Usage
|
|
29
29
|
|
|
30
|
+
### Client (recommended)
|
|
31
|
+
|
|
32
|
+
Keep one `Client` per process: use `HowLongToBeat::Client.default`, or build a single instance of your own. It paces requests (2 seconds apart by default) and caches the search endpoint and auth token, so every extra instance pays for its own endpoint discovery. Errors are distinct, so "rate-limited" never looks like "not found". The legacy `HowLongToBeat::HowLongToBeat` API uses `Client.default` unless you pass `client:`.
|
|
33
|
+
|
|
34
|
+
```ruby
|
|
35
|
+
client = HowLongToBeat::Client.default
|
|
36
|
+
|
|
37
|
+
detail = client.game(10270) # one request; nil if HLTB's 404 page says there is no such game
|
|
38
|
+
detail.main_story # => 51.68 (hours)
|
|
39
|
+
detail.steam_app_id # => 292030
|
|
40
|
+
|
|
41
|
+
begin
|
|
42
|
+
client.search("The Witcher 3") # => [HowLongToBeat::SearchResult, ...]; [] when nothing matches
|
|
43
|
+
rescue HowLongToBeat::RateLimitedError
|
|
44
|
+
# HLTB answered 429: back off before trying again
|
|
45
|
+
rescue HowLongToBeat::RequestError
|
|
46
|
+
# network failure, timeout or an HTTP error: retry later
|
|
47
|
+
rescue HowLongToBeat::ParseError
|
|
48
|
+
# HLTB changed its site, so the gem needs an update
|
|
49
|
+
end
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
HowLongToBeat has no official API and rate-limits bursts. Please keep the default pacing.
|
|
53
|
+
|
|
30
54
|
### Basic Search
|
|
31
55
|
|
|
32
56
|
```ruby
|
|
@@ -36,7 +60,7 @@ hltb = HowLongToBeat::HowLongToBeat.new
|
|
|
36
60
|
results = hltb.search("The Witcher 3")
|
|
37
61
|
```
|
|
38
62
|
|
|
39
|
-
The `search` method returns an array of possible games, or `nil` if
|
|
63
|
+
The `search` method returns an array of possible games (`[]` when nothing matches), or `nil` if the game name is empty or the request failed.
|
|
40
64
|
|
|
41
65
|
Each result is an [`HowLongToBeatEntry`](https://github.com/dpashutskii/howlongtobeat/blob/main/lib/howlongtobeat/how_long_to_beat_entry.rb) object containing:
|
|
42
66
|
|
|
@@ -80,6 +104,8 @@ result = hltb.search_from_id(10270) # The Witcher 3: Wild Hunt
|
|
|
80
104
|
|
|
81
105
|
This returns a single [`HowLongToBeatEntry`](https://github.com/dpashutskii/howlongtobeat/blob/main/lib/howlongtobeat/how_long_to_beat_entry.rb) object or `nil` if not found.
|
|
82
106
|
|
|
107
|
+
Since 0.3.0 `search_from_id` reads the game page in one request, so it fills fewer fields than before (no image URL, review score, developer, platforms, JSON content or complexity flags). See the [CHANGELOG](CHANGELOG.md) for the exact list.
|
|
108
|
+
|
|
83
109
|
### Search Modifiers
|
|
84
110
|
|
|
85
111
|
You can filter your search results using modifiers:
|
|
@@ -116,7 +142,7 @@ This gem was originally created for and is being used by [SearchToPlay](https://
|
|
|
116
142
|
|
|
117
143
|
## Development
|
|
118
144
|
|
|
119
|
-
After checking out the repo, run `bundle install` to install dependencies. Then
|
|
145
|
+
After checking out the repo, run `bundle install` to install dependencies. Then run `bundle exec rspec --tag '~live'` to run the offline tests. `rake spec` runs everything, including the `live` specs that call howlongtobeat.com (paced, about 3 minutes); run those with `bundle exec rspec --tag live` before a release.
|
|
120
146
|
|
|
121
147
|
## Contributing
|
|
122
148
|
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
require 'json'
|
|
2
|
+
require 'nokogiri'
|
|
3
|
+
|
|
4
|
+
module HowLongToBeat
|
|
5
|
+
# Polite, stateful HLTB client. Keep one instance per process (Client.default
|
|
6
|
+
# is one): it remembers the discovered search endpoint and the auth token,
|
|
7
|
+
# and its Http enforces a minimum gap between requests. Safe to share
|
|
8
|
+
# between threads.
|
|
9
|
+
class Client
|
|
10
|
+
BASE_URL = Http::BASE_URL
|
|
11
|
+
|
|
12
|
+
# Tried in order only when no script chunk reveals the search endpoint.
|
|
13
|
+
KNOWN_SEARCH_PATHS = %w[/api/search/site /api/bleed /api/finder /api/search /api/s].freeze
|
|
14
|
+
|
|
15
|
+
# HLTB's own Next.js 404 page embeds this in its __NEXT_DATA__.
|
|
16
|
+
NOT_FOUND_MARKER = '"page":"/404"'.freeze
|
|
17
|
+
|
|
18
|
+
Endpoint = Struct.new(:path, :search_info, :found_at)
|
|
19
|
+
Token = Struct.new(:auth, :issued_at)
|
|
20
|
+
EndpointGone = Class.new(StandardError)
|
|
21
|
+
DEFAULT_LOCK = Mutex.new
|
|
22
|
+
private_constant :Endpoint, :Token, :EndpointGone, :DEFAULT_LOCK
|
|
23
|
+
|
|
24
|
+
# The process-wide client, built on first use. Share it so every caller
|
|
25
|
+
# pays for one endpoint discovery and one paced stream of requests.
|
|
26
|
+
def self.default
|
|
27
|
+
DEFAULT_LOCK.synchronize { @default ||= new }
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def initialize(http: Http.new, endpoint_ttl: 3600, token_ttl: 60, token_warmup: 1.2,
|
|
31
|
+
clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) },
|
|
32
|
+
sleeper: ->(seconds) { sleep(seconds) })
|
|
33
|
+
@http = http
|
|
34
|
+
@endpoint_ttl = endpoint_ttl
|
|
35
|
+
@token_ttl = token_ttl
|
|
36
|
+
@token_warmup = token_warmup
|
|
37
|
+
@clock = clock
|
|
38
|
+
@sleeper = sleeper
|
|
39
|
+
@endpoint = nil
|
|
40
|
+
@token = nil
|
|
41
|
+
@lock = Mutex.new # guards @endpoint and @token; never held during the search POST
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
# One GET of the game page. nil only when HLTB's own 404 page says there
|
|
45
|
+
# is no such game; any other 404 (a CDN or proxy page) is a RequestError.
|
|
46
|
+
def game(id)
|
|
47
|
+
response = @http.get("#{BASE_URL}/game/#{Integer(id.to_s, 10)}")
|
|
48
|
+
if response.code == 404
|
|
49
|
+
return nil if response.body.include?(NOT_FOUND_MARKER)
|
|
50
|
+
|
|
51
|
+
raise RequestError.new("HLTB returned an unrecognised 404 for /game/#{id}", status: 404)
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
GamePageParser.parse(response.body)
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# Raw candidates for a title. [] is a genuine "not on HLTB".
|
|
58
|
+
def search(title, modifier: HTMLRequests::SearchModifiers::NONE)
|
|
59
|
+
SearchResultParser.parse(search_json(title, modifier: modifier))
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
# The raw search response body (the legacy API parses it itself).
|
|
63
|
+
def search_json(title, modifier: HTMLRequests::SearchModifiers::NONE)
|
|
64
|
+
post_search(title, modifier)
|
|
65
|
+
rescue EndpointGone
|
|
66
|
+
forget_endpoint!
|
|
67
|
+
begin
|
|
68
|
+
post_search(title, modifier)
|
|
69
|
+
rescue EndpointGone
|
|
70
|
+
raise ParseError, 'HLTB search endpoint returns 404 even after rediscovery'
|
|
71
|
+
end
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
private
|
|
75
|
+
|
|
76
|
+
def post_search(title, modifier)
|
|
77
|
+
endpoint, auth = @lock.synchronize do
|
|
78
|
+
current = current_endpoint
|
|
79
|
+
[current, current_token(current)]
|
|
80
|
+
end
|
|
81
|
+
headers = HTMLRequests.get_search_request_headers(auth)
|
|
82
|
+
body = HTMLRequests.get_search_request_data(title, modifier, 1, endpoint.search_info, auth)
|
|
83
|
+
response = @http.post("#{BASE_URL}#{endpoint.path}", headers, body)
|
|
84
|
+
raise EndpointGone if response.code == 404
|
|
85
|
+
|
|
86
|
+
response.body
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
# Callers hold @lock, so the ivars are read once and updated without a race.
|
|
90
|
+
def current_endpoint
|
|
91
|
+
endpoint = @endpoint
|
|
92
|
+
return endpoint if endpoint && @clock.call - endpoint.found_at < @endpoint_ttl
|
|
93
|
+
|
|
94
|
+
@token = nil
|
|
95
|
+
@endpoint = discover
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
def forget_endpoint!
|
|
99
|
+
@lock.synchronize do
|
|
100
|
+
@endpoint = nil
|
|
101
|
+
@token = nil
|
|
102
|
+
end
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def current_token(endpoint)
|
|
106
|
+
cached = @token
|
|
107
|
+
return cached.auth if cached && @clock.call - cached.issued_at < @token_ttl
|
|
108
|
+
|
|
109
|
+
response = @http.get(init_url(endpoint.path))
|
|
110
|
+
raise EndpointGone if response.code == 404
|
|
111
|
+
|
|
112
|
+
store_token(response.body)
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
# Parses an /init body, caches the token, and gives HLTB a moment to
|
|
116
|
+
# accept it before the first search.
|
|
117
|
+
def store_token(body)
|
|
118
|
+
@token = fresh = Token.new(parse_auth(body), @clock.call)
|
|
119
|
+
@sleeper.call(@token_warmup)
|
|
120
|
+
fresh.auth
|
|
121
|
+
end
|
|
122
|
+
|
|
123
|
+
def init_url(path)
|
|
124
|
+
"#{BASE_URL}#{path}/init?t=#{Time.now.to_i}"
|
|
125
|
+
end
|
|
126
|
+
|
|
127
|
+
def parse_auth(body)
|
|
128
|
+
json = JSON.parse(body.to_s)
|
|
129
|
+
raise ParseError, 'HLTB /init response is not an object' unless json.is_a?(Hash)
|
|
130
|
+
|
|
131
|
+
token = json['token'] || json.dig('data', 'token') || json['auth_token'] || json['authToken']
|
|
132
|
+
raise ParseError, 'HLTB /init response has no token' if token.to_s.empty?
|
|
133
|
+
|
|
134
|
+
key = json.find { |name, _| name.downcase.include?('key') }&.last
|
|
135
|
+
value = json.find { |name, _| name.downcase.include?('val') }&.last
|
|
136
|
+
HTMLRequests::AuthStruct.new(token, key, value)
|
|
137
|
+
rescue JSON::ParserError => e
|
|
138
|
+
raise ParseError, "HLTB /init response is not JSON: #{e.message}"
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
# The search call is the POST fetch in HLTB's JS bundle that sends
|
|
142
|
+
# x-auth-token. Chunk names are opaque, so walk them in page order.
|
|
143
|
+
def discover
|
|
144
|
+
homepage = @http.get(BASE_URL)
|
|
145
|
+
raise RequestError, 'HLTB homepage returned 404' if homepage.code == 404
|
|
146
|
+
|
|
147
|
+
Nokogiri::HTML(homepage.body).css('script[src]').each do |script|
|
|
148
|
+
info = search_info_from(script['src'])
|
|
149
|
+
return Endpoint.new("/#{info.search_url}", info, @clock.call) if info
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
fallback_endpoint
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
def search_info_from(src)
|
|
156
|
+
url = src.start_with?('http') ? src : "#{BASE_URL}#{src}"
|
|
157
|
+
response = @http.get(url)
|
|
158
|
+
return nil unless response.code == 200
|
|
159
|
+
|
|
160
|
+
info = HTMLRequests::SearchInfo.new(response.body)
|
|
161
|
+
info if info.authenticated? && !info.search_url.to_s.empty?
|
|
162
|
+
rescue RequestError
|
|
163
|
+
nil # a redirected or broken chunk (e.g. the consent script) is not fatal
|
|
164
|
+
end
|
|
165
|
+
|
|
166
|
+
def fallback_endpoint
|
|
167
|
+
KNOWN_SEARCH_PATHS.each do |path|
|
|
168
|
+
response = @http.get(init_url(path))
|
|
169
|
+
next unless response.code == 200
|
|
170
|
+
|
|
171
|
+
store_token(response.body) # the probe already fetched /init; don't fetch it again
|
|
172
|
+
return Endpoint.new(path, nil, @clock.call)
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
raise ParseError, 'Could not discover the HLTB search endpoint'
|
|
176
|
+
end
|
|
177
|
+
end
|
|
178
|
+
end
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
module HowLongToBeat
|
|
2
|
+
class Error < StandardError; end
|
|
3
|
+
|
|
4
|
+
# HLTB answered 429. Back off; don't retry right away.
|
|
5
|
+
class RateLimitedError < Error; end
|
|
6
|
+
|
|
7
|
+
# Network failure, timeout, or an HTTP status that means "try later"
|
|
8
|
+
# (5xx, 403, unexpected redirects). Never means "not found".
|
|
9
|
+
# `status` is the HTTP status when there was one, nil for network failures.
|
|
10
|
+
class RequestError < Error
|
|
11
|
+
attr_reader :status
|
|
12
|
+
|
|
13
|
+
def initialize(message = nil, status: nil)
|
|
14
|
+
super(message)
|
|
15
|
+
@status = status
|
|
16
|
+
end
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# HLTB answered, but not in the shape we know. Usually a site change.
|
|
20
|
+
class ParseError < Error; end
|
|
21
|
+
end
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
require 'date'
|
|
2
|
+
|
|
3
|
+
module HowLongToBeat
|
|
4
|
+
# Conversions shared by the game-page and search parsers.
|
|
5
|
+
module Fields
|
|
6
|
+
module_function
|
|
7
|
+
|
|
8
|
+
# HLTB reports times in seconds; 0 means "no data".
|
|
9
|
+
def hours(seconds)
|
|
10
|
+
value = seconds.to_f
|
|
11
|
+
value.positive? ? (value / 3600.0).round(2) : nil
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def aliases(value)
|
|
15
|
+
value.to_s.split(',').map(&:strip).reject(&:empty?)
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def positive_int(value)
|
|
19
|
+
number = value.to_i
|
|
20
|
+
number.positive? ? number : nil
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
# Game pages send "2015-05-19"; search rows send 2015.
|
|
24
|
+
def release_date(value)
|
|
25
|
+
Date.iso8601(value.to_s) if value.to_s.match?(/\A\d{4}-\d{2}-\d{2}\z/)
|
|
26
|
+
rescue Date::Error
|
|
27
|
+
nil
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def release_year(value)
|
|
31
|
+
year = value.to_s[/\A\d{4}/].to_i
|
|
32
|
+
year.positive? ? year : nil
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def times(game)
|
|
36
|
+
{
|
|
37
|
+
main_story: hours(game['comp_main']),
|
|
38
|
+
main_extra: hours(game['comp_plus']),
|
|
39
|
+
completionist: hours(game['comp_100']),
|
|
40
|
+
all_styles: hours(game['comp_all']),
|
|
41
|
+
coop: hours(game['invested_co']),
|
|
42
|
+
multiplayer: hours(game['invested_mp'])
|
|
43
|
+
}
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
require 'json'
|
|
2
|
+
|
|
3
|
+
module HowLongToBeat
|
|
4
|
+
# Reads the record Next.js embeds in every /game/<id> page, so refreshing
|
|
5
|
+
# a known game takes one GET and no search or auth token.
|
|
6
|
+
module GamePageParser
|
|
7
|
+
NEXT_DATA = %r{<script[^>]*\bid="__NEXT_DATA__"[^>]*>(.*?)</script>}m
|
|
8
|
+
|
|
9
|
+
module_function
|
|
10
|
+
|
|
11
|
+
def parse(html)
|
|
12
|
+
encoded_html = html.to_s.dup.force_encoding(Encoding::UTF_8).scrub
|
|
13
|
+
json = encoded_html[NEXT_DATA, 1]
|
|
14
|
+
raise ParseError, 'HLTB game page has no __NEXT_DATA__ script' unless json
|
|
15
|
+
|
|
16
|
+
data = JSON.parse(json)
|
|
17
|
+
game = data.dig('props', 'pageProps', 'game', 'data', 'game')
|
|
18
|
+
game = game.first if game.is_a?(Array)
|
|
19
|
+
raise ParseError, 'HLTB game page has no game record' unless game.is_a?(Hash) && game['game_id']
|
|
20
|
+
|
|
21
|
+
GameDetail.new(
|
|
22
|
+
id: game['game_id'].to_i,
|
|
23
|
+
name: game['game_name'],
|
|
24
|
+
aliases: Fields.aliases(game['game_alias']),
|
|
25
|
+
release_date: Fields.release_date(game['release_world']),
|
|
26
|
+
release_year: Fields.release_year(game['release_world']),
|
|
27
|
+
steam_app_id: Fields.positive_int(game['profile_steam']),
|
|
28
|
+
game_type: game['game_type'],
|
|
29
|
+
**Fields.times(game)
|
|
30
|
+
)
|
|
31
|
+
rescue JSON::ParserError => e
|
|
32
|
+
raise ParseError, "HLTB game page JSON is invalid: #{e.message}"
|
|
33
|
+
rescue TypeError, NoMethodError => e
|
|
34
|
+
raise ParseError, "HLTB game page has unexpected structure: #{e.message}"
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
end
|
|
@@ -1,37 +1,66 @@
|
|
|
1
1
|
module HowLongToBeat
|
|
2
|
+
# The original API, kept for existing callers. It now runs on Client, so
|
|
3
|
+
# it shares the pacing, caching and error handling; any failure still
|
|
4
|
+
# returns nil, as before.
|
|
5
|
+
#
|
|
6
|
+
# Note: search_from_id now builds entries from the game page only. It sets
|
|
7
|
+
# id, name, alias, type, web link, release year, and the six time fields
|
|
8
|
+
# (main_story, main_extra, completionist, all_styles, coop_time, mp_time)
|
|
9
|
+
# plus similarity (1.0). Image URL, review score, developer, platforms,
|
|
10
|
+
# JSON content, and complexity flags stay unset. auto_filter_times does
|
|
11
|
+
# not apply to search_from_id results.
|
|
2
12
|
class HowLongToBeat
|
|
3
|
-
def initialize(input_minimum_similarity = 0.4, input_auto_filter_times = false)
|
|
13
|
+
def initialize(input_minimum_similarity = 0.4, input_auto_filter_times = false, client: Client.default)
|
|
4
14
|
@minimum_similarity = input_minimum_similarity
|
|
5
15
|
@auto_filter_times = input_auto_filter_times
|
|
16
|
+
@client = client
|
|
6
17
|
end
|
|
7
18
|
|
|
8
19
|
def search(game_name, search_modifiers = HTMLRequests::SearchModifiers::NONE,
|
|
9
|
-
|
|
20
|
+
similarity_case_sensitive = true)
|
|
10
21
|
return nil if game_name.nil? || game_name.empty?
|
|
11
22
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
23
|
+
json = @client.search_json(game_name, modifier: search_modifiers)
|
|
24
|
+
# Validate the shape first; SearchResultParser raises ParseError for any
|
|
25
|
+
# malformed response, which rescue Error below will turn into nil.
|
|
26
|
+
SearchResultParser.parse(json)
|
|
27
|
+
parse_web_result(game_name, json, nil, similarity_case_sensitive)
|
|
28
|
+
rescue Error
|
|
29
|
+
nil
|
|
16
30
|
end
|
|
17
31
|
|
|
18
32
|
def search_from_id(game_id)
|
|
19
33
|
return nil if game_id.nil? || game_id == 0
|
|
34
|
+
# Validate the ID is numeric before calling the client; game_id is expected to be
|
|
35
|
+
# an integer, but Integer() will coerce strings like "123" and raise ArgumentError on "abc"
|
|
36
|
+
return nil unless game_id.is_a?(Integer) || (game_id.is_a?(String) && game_id.match?(/\A\d+\z/))
|
|
20
37
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
return nil unless html_result
|
|
26
|
-
|
|
27
|
-
result_list = parse_web_result(game_title, html_result, game_id)
|
|
28
|
-
return nil unless result_list && result_list.size == 1
|
|
29
|
-
|
|
30
|
-
result_list.first
|
|
38
|
+
detail = @client.game(game_id)
|
|
39
|
+
detail && entry_from(detail)
|
|
40
|
+
rescue Error, ArgumentError
|
|
41
|
+
nil
|
|
31
42
|
end
|
|
32
43
|
|
|
33
44
|
private
|
|
34
45
|
|
|
46
|
+
def entry_from(detail)
|
|
47
|
+
HowLongToBeatEntry.new.tap do |entry|
|
|
48
|
+
entry.game_id = detail.id
|
|
49
|
+
entry.game_name = detail.name
|
|
50
|
+
entry.game_alias = detail.aliases.join(', ')
|
|
51
|
+
entry.game_type = detail.game_type
|
|
52
|
+
entry.game_web_link = "#{JSONResultParser::GAME_URL_PREFIX}#{detail.id}"
|
|
53
|
+
entry.release_world = detail.release_year
|
|
54
|
+
entry.main_story = detail.main_story
|
|
55
|
+
entry.main_extra = detail.main_extra
|
|
56
|
+
entry.completionist = detail.completionist
|
|
57
|
+
entry.all_styles = detail.all_styles
|
|
58
|
+
entry.coop_time = detail.coop
|
|
59
|
+
entry.mp_time = detail.multiplayer
|
|
60
|
+
entry.similarity = 1.0
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
|
|
35
64
|
def parse_web_result(game_name, html_result, game_id = nil, similarity_case_sensitive = true)
|
|
36
65
|
parser = JSONResultParser.new(
|
|
37
66
|
game_name,
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
require 'net/http'
|
|
2
|
+
require 'openssl'
|
|
3
|
+
require 'uri'
|
|
4
|
+
require 'zlib'
|
|
5
|
+
|
|
6
|
+
module HowLongToBeat
|
|
7
|
+
# Paced HTTP transport for HLTB. Every request waits until at least
|
|
8
|
+
# `min_interval` seconds have passed since the previous one, however the
|
|
9
|
+
# caller loops. 2xx and 404 come back as a Response; 429 raises
|
|
10
|
+
# RateLimitedError; anything else raises RequestError, so a failure can
|
|
11
|
+
# never look like "not found".
|
|
12
|
+
class Http
|
|
13
|
+
Response = Struct.new(:code, :body)
|
|
14
|
+
|
|
15
|
+
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ' \
|
|
16
|
+
'(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'
|
|
17
|
+
BASE_URL = 'https://howlongtobeat.com'
|
|
18
|
+
|
|
19
|
+
NETWORK_ERRORS = [
|
|
20
|
+
Timeout::Error, IOError, EOFError, SocketError, SystemCallError,
|
|
21
|
+
OpenSSL::SSL::SSLError, Net::HTTPBadResponse, Net::ProtocolError, Zlib::Error
|
|
22
|
+
].freeze
|
|
23
|
+
|
|
24
|
+
def initialize(min_interval: 2.0, open_timeout: 5, read_timeout: 10,
|
|
25
|
+
clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) },
|
|
26
|
+
sleeper: ->(seconds) { sleep(seconds) })
|
|
27
|
+
@min_interval = min_interval
|
|
28
|
+
@open_timeout = open_timeout
|
|
29
|
+
@read_timeout = read_timeout
|
|
30
|
+
@clock = clock
|
|
31
|
+
@sleeper = sleeper
|
|
32
|
+
@mutex = Mutex.new
|
|
33
|
+
@last_request_at = nil
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def get(url, headers = {})
|
|
37
|
+
uri = URI(url)
|
|
38
|
+
perform(uri, Net::HTTP::Get.new(uri.request_uri, default_headers.merge(headers)))
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def post(url, headers, body)
|
|
42
|
+
uri = URI(url)
|
|
43
|
+
request = Net::HTTP::Post.new(uri.request_uri, default_headers.merge(headers))
|
|
44
|
+
request.body = body
|
|
45
|
+
perform(uri, request)
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
private
|
|
49
|
+
|
|
50
|
+
def perform(uri, request)
|
|
51
|
+
pace!
|
|
52
|
+
# Disable Net::HTTP's silent retry (max_retries: 1 by default) so every request,
|
|
53
|
+
# including automatic retries after transient errors, goes through pace!.
|
|
54
|
+
response = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https',
|
|
55
|
+
max_retries: 0,
|
|
56
|
+
open_timeout: @open_timeout,
|
|
57
|
+
read_timeout: @read_timeout) do |connection|
|
|
58
|
+
connection.request(request)
|
|
59
|
+
end
|
|
60
|
+
to_response(uri, response)
|
|
61
|
+
rescue *NETWORK_ERRORS => e
|
|
62
|
+
raise RequestError, "#{request.method} #{uri.path} failed: #{e.class}: #{e.message}"
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
def to_response(uri, response)
|
|
66
|
+
code = response.code.to_i
|
|
67
|
+
raise RateLimitedError, "HLTB rate limited #{uri.path} (429)" if code == 429
|
|
68
|
+
return Response.new(code, response.body.to_s) if (200..299).cover?(code) || code == 404
|
|
69
|
+
|
|
70
|
+
raise RequestError.new("HLTB returned #{code} for #{uri.path}", status: code)
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def pace!
|
|
74
|
+
@mutex.synchronize do
|
|
75
|
+
if @last_request_at
|
|
76
|
+
wait = @min_interval - (@clock.call - @last_request_at)
|
|
77
|
+
@sleeper.call(wait) if wait.positive?
|
|
78
|
+
end
|
|
79
|
+
@last_request_at = @clock.call
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
def default_headers
|
|
84
|
+
{ 'User-Agent' => USER_AGENT, 'Referer' => BASE_URL, 'Origin' => BASE_URL, 'Accept' => '*/*' }
|
|
85
|
+
end
|
|
86
|
+
end
|
|
87
|
+
end
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
module HowLongToBeat
|
|
2
|
+
# A game as its /game/<id> page describes it. Times are hours, nil when
|
|
3
|
+
# HLTB has no data.
|
|
4
|
+
GameDetail = Struct.new(
|
|
5
|
+
:id, :name, :aliases, :release_date, :release_year, :steam_app_id, :game_type,
|
|
6
|
+
:main_story, :main_extra, :completionist, :all_styles, :coop, :multiplayer,
|
|
7
|
+
keyword_init: true
|
|
8
|
+
)
|
|
9
|
+
|
|
10
|
+
# One row of a title search. Search rows carry a release year, not a date,
|
|
11
|
+
# and no Steam id; fetch the game page for those.
|
|
12
|
+
SearchResult = Struct.new(
|
|
13
|
+
:id, :name, :aliases, :release_year, :game_type,
|
|
14
|
+
:main_story, :main_extra, :completionist, :all_styles, :coop, :multiplayer,
|
|
15
|
+
keyword_init: true
|
|
16
|
+
)
|
|
17
|
+
end
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
require 'json'
|
|
2
|
+
|
|
3
|
+
module HowLongToBeat
|
|
4
|
+
# Turns the search API response into SearchResult rows. An empty `data`
|
|
5
|
+
# array is a genuine "not on HLTB"; any other shape is a ParseError.
|
|
6
|
+
module SearchResultParser
|
|
7
|
+
module_function
|
|
8
|
+
|
|
9
|
+
def parse(json)
|
|
10
|
+
payload = JSON.parse(json.to_s.dup.force_encoding(Encoding::UTF_8).scrub)
|
|
11
|
+
games = payload['data'] if payload.is_a?(Hash)
|
|
12
|
+
raise ParseError, 'HLTB search response has no data array' unless games.is_a?(Array)
|
|
13
|
+
|
|
14
|
+
games.map do |game|
|
|
15
|
+
raise ParseError, 'HLTB search response game row is not a dict' unless game.is_a?(Hash)
|
|
16
|
+
raise ParseError, 'HLTB search response game row has no game_id' unless game['game_id']
|
|
17
|
+
|
|
18
|
+
SearchResult.new(
|
|
19
|
+
id: game['game_id'].to_i,
|
|
20
|
+
name: game['game_name'],
|
|
21
|
+
aliases: Fields.aliases(game['game_alias']),
|
|
22
|
+
release_year: Fields.release_year(game['release_world']),
|
|
23
|
+
game_type: game['game_type'],
|
|
24
|
+
**Fields.times(game)
|
|
25
|
+
)
|
|
26
|
+
end
|
|
27
|
+
rescue JSON::ParserError => e
|
|
28
|
+
raise ParseError, "HLTB search response is not JSON: #{e.message}"
|
|
29
|
+
rescue NoMethodError, TypeError => e
|
|
30
|
+
raise ParseError, "HLTB search response has unexpected structure: #{e.message}"
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
data/lib/howlongtobeat.rb
CHANGED
|
@@ -1,9 +1,12 @@
|
|
|
1
1
|
require_relative 'howlongtobeat/version'
|
|
2
|
+
require_relative 'howlongtobeat/errors'
|
|
3
|
+
require_relative 'howlongtobeat/http'
|
|
4
|
+
require_relative 'howlongtobeat/records'
|
|
5
|
+
require_relative 'howlongtobeat/fields'
|
|
6
|
+
require_relative 'howlongtobeat/game_page_parser'
|
|
7
|
+
require_relative 'howlongtobeat/search_result_parser'
|
|
2
8
|
require_relative 'howlongtobeat/how_long_to_beat'
|
|
3
9
|
require_relative 'howlongtobeat/how_long_to_beat_entry'
|
|
4
10
|
require_relative 'howlongtobeat/html_requests'
|
|
5
11
|
require_relative 'howlongtobeat/json_result_parser'
|
|
6
|
-
|
|
7
|
-
module HowLongToBeat
|
|
8
|
-
class Error < StandardError; end
|
|
9
|
-
end
|
|
12
|
+
require_relative 'howlongtobeat/client'
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: howlongtobeat
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Dmitrii Pashutskii
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
11
|
+
date: 2026-10-06 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: nokogiri
|
|
@@ -80,6 +80,20 @@ dependencies:
|
|
|
80
80
|
- - "~>"
|
|
81
81
|
- !ruby/object:Gem::Version
|
|
82
82
|
version: '3.0'
|
|
83
|
+
- !ruby/object:Gem::Dependency
|
|
84
|
+
name: webmock
|
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
|
86
|
+
requirements:
|
|
87
|
+
- - "~>"
|
|
88
|
+
- !ruby/object:Gem::Version
|
|
89
|
+
version: '3.23'
|
|
90
|
+
type: :development
|
|
91
|
+
prerelease: false
|
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
93
|
+
requirements:
|
|
94
|
+
- - "~>"
|
|
95
|
+
- !ruby/object:Gem::Version
|
|
96
|
+
version: '3.23'
|
|
83
97
|
description: A simple Ruby gem to fetch game completion times from HowLongToBeat.com
|
|
84
98
|
email:
|
|
85
99
|
- dpashutskii@gmail.com
|
|
@@ -91,10 +105,17 @@ files:
|
|
|
91
105
|
- CODE_OF_CONDUCT.md
|
|
92
106
|
- README.md
|
|
93
107
|
- lib/howlongtobeat.rb
|
|
108
|
+
- lib/howlongtobeat/client.rb
|
|
109
|
+
- lib/howlongtobeat/errors.rb
|
|
110
|
+
- lib/howlongtobeat/fields.rb
|
|
111
|
+
- lib/howlongtobeat/game_page_parser.rb
|
|
94
112
|
- lib/howlongtobeat/how_long_to_beat.rb
|
|
95
113
|
- lib/howlongtobeat/how_long_to_beat_entry.rb
|
|
96
114
|
- lib/howlongtobeat/html_requests.rb
|
|
115
|
+
- lib/howlongtobeat/http.rb
|
|
97
116
|
- lib/howlongtobeat/json_result_parser.rb
|
|
117
|
+
- lib/howlongtobeat/records.rb
|
|
118
|
+
- lib/howlongtobeat/search_result_parser.rb
|
|
98
119
|
- lib/howlongtobeat/version.rb
|
|
99
120
|
homepage: https://github.com/dpashutskii/howlongtobeat
|
|
100
121
|
licenses:
|