link_thumbnailer 3.3.2 → 3.4.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 281992a6403788c75541c08d69586ab9b85c097e
4
- data.tar.gz: 7cd32584b4c3a96b6a54185576597115fdb84764
2
+ SHA256:
3
+ metadata.gz: 4a61e97176d3dfeee5edd1b272ad4c9bb9e4d967655edd529d128f5ec0bf2a9f
4
+ data.tar.gz: 48430fe14a47d7a76b26df13421c01474b21c4b76df823723f6e13ba70d2a46f
5
5
  SHA512:
6
- metadata.gz: d77d1ae2545a92f79363a1e90d19edd61c610080d5497207cf22bdb9afe0d0acb5569aa95d7c16485dc9aa3ff0beac5c900b7cea24278d87d5a0812900f7aa91
7
- data.tar.gz: 69c55be441d1581c50e54b3f3652130d8c4f9220047714607d978a993c6be581b28b70cb19a62ad3f27dc09c91455d67f4e83bceff753b451a4d3b3a1e532763
6
+ metadata.gz: 206962b937ceb0a38910c71bce981379972c0291e7a71c21b1b2d6ccbd22f82e991be36ec4c2b084a681b6c8e2c7f7713ad420d26a8e617fc5d4733bb9785f7c
7
+ data.tar.gz: 38ee02ac70e404432ce97d276f09e4f2dff02c2b323753098fe7607cb93cbcf549e85b8ec9c0ed11cead8f8c81ba0282bb1c0c22a173208773b10a080e34b9c6
@@ -1 +1 @@
1
- 2.4.0
1
+ 2.6.6
@@ -1,5 +1,4 @@
1
- bundler_args: --without development
2
1
  language: ruby
3
2
  rvm:
4
- - 2.2.2
5
- - 2.4.0
3
+ - 2.5
4
+ - 2.6
@@ -1,12 +1,43 @@
1
- # 3.3.2
1
+ # Changelog
2
+ All notable changes to this project will be documented in this file.
3
+
4
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
5
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
+
7
+ ## [Unreleased]
8
+
9
+ ## [3.4.0]
10
+ ### Adds
11
+
12
+ - Adds `download_size_limit` configuration to raise `LinkThumbnailer::DownloadSizeLimit` when the body of the request is too big. Defaults to `10 * 1024 * 1024` bytes.
13
+ - Adds `favicon_size` configuration to allow to choose which favison
14
+ size the gem should prefer. Defaults to the first favicon found otherwise.
15
+
16
+ ### Fixes
17
+
18
+ - Fixes string encoding in previous versions of Ruby
19
+ - Fixes favicon by providing the full path.
20
+ - When HTML charset cannot be found in the HTML header, we now try
21
+ to find it in the body.
22
+ - Closes the HTTP connection upon completion
23
+
24
+ ### Changes
25
+
26
+ - 401 HTTP errors now raise `LinkThumbnailer::HTTPError`
27
+ - Upgrades [ImageInfo](https://github.com/gottfrois/image_info/blob/master/CHANGELOG.md) gem
28
+
29
+ ## [3.3.2]
30
+ ### Fixes
2
31
 
3
32
  - Frozen strings https://github.com/gottfrois/link_thumbnailer/pull/125
4
33
 
5
- # 3.3.1
34
+ ## [3.3.1]
35
+ ### Changes
6
36
 
7
37
  - Gem upgrade (json)
8
38
 
9
- # 3.3.0
39
+ ## [3.3.0]
40
+ ### Adds
10
41
 
11
42
  - Allows to configure overrided http headers
12
43
 
@@ -16,14 +47,16 @@ LinkThumbnailer.configure do |config|
16
47
  end
17
48
  ```
18
49
 
19
- # 3.2.1
50
+ ## [3.2.1]
51
+ ### Fixes
20
52
 
21
53
  - Fixes #88
22
54
  - Override User-Agent header properly
23
55
  - Match xpath nodes if attribute content is present
24
56
  - Avoid nil urls in image parser
25
57
 
26
- # 3.2.0
58
+ ## [3.2.0]
59
+ ### Adds
27
60
 
28
61
  Makes scrapers configurable by allowing to set the scraping strategy:
29
62
 
@@ -36,7 +69,8 @@ end
36
69
  `opengraph` use the [Open Graph Protocol](http://ogp.me/).
37
70
  `default` use a homemade algorithm
38
71
 
39
- # 3.1.2
72
+ ## [3.1.2]
73
+ ### Adds
40
74
 
41
75
  Allows to customize ideal description length
42
76
 
@@ -58,14 +92,22 @@ end
58
92
  Will default to `120` characters. More information about how the gem manage to find the best description can be found at
59
93
  http://www.codeids.com/2015/06/27/how-to-find-best-description-of-a-website-using-linkthumbnailer/
60
94
 
61
- # 3.1.1
95
+ ## [3.1.1]
96
+ ### Fixes
62
97
 
63
- - Upgrade `video_info` gem
64
98
  - Fixes https://github.com/gottfrois/link_thumbnailer/issues/69
65
99
 
66
- # 3.1.0
100
+ ### Changes
101
+
102
+ - Upgrade `video_info` gem
103
+
104
+ ## [3.1.0]
105
+ ### Fixes
67
106
 
68
107
  - Fix an issue when image sizes could not be retrieved.
108
+
109
+ ### Adds
110
+
69
111
  - Grapers now accepts an optional parameter to customize the weigth of the grader in the probablity computation.
70
112
 
71
113
  ```ruby
@@ -76,64 +118,83 @@ Will give a 3 times more weigth to the `Position` grader compare to other grader
76
118
  By default all graders have a weigth of `1` except the above position grader since position should play a bigger role in
77
119
  order to find good description candidates.
78
120
 
79
- # 3.0.3
121
+ ## [3.0.3]
122
+ ### Fixes
80
123
 
81
124
  - Fix an issue when dealing with absolute urls. https://github.com/gottfrois/link_thumbnailer/issues/68
82
125
  - Fix an issue with http redirection and location header not beeing present. https://github.com/gottfrois/link_thumbnailer/issues/70
83
126
  - Rescue and raise custom LinkThumbnailer exceptions. https://github.com/gottfrois/link_thumbnailer/issues/71
84
127
 
85
- # 3.0.2
128
+ ## [3.0.2]
129
+ ### Fixes
86
130
 
87
131
  - Replace FastImage gem dependency by [ImageInfo](https://github.com/gottfrois/image_info) to improve performances when
88
132
  fetching multiple images size information. Benchmark shows an order of magnitude improvement response time.
89
133
  - Fixes [#57](https://github.com/gottfrois/link_thumbnailer/issues/57)
90
134
 
91
- # 3.0.1
135
+ ## [3.0.1]
136
+ ### Fixes
92
137
 
93
138
  - Remove useless dependencies
94
139
 
95
- # 3.0.0
140
+ ## [3.0.0]
141
+ ### Changes
96
142
 
97
143
  - Improved description sorting.
98
144
  - Refactored how graders work. More information [here](https://github.com/gottfrois/link_thumbnailer/wiki/How-to-build-your-own-Grader%3F)
99
145
 
100
- # 2.6.1
146
+ ## [2.6.1]
147
+ ### Fixes
101
148
 
102
149
  - Fix remove useless dependency
103
150
 
104
- # 2.6.0
151
+ ## [2.6.0]
152
+ ### Adds
105
153
 
106
154
  - Introduce new `raise_on_invalid_format` option (false by default) to raise `LinkThumbnailer::FormatNotSupported` if http `Content-Type` is invalid. Fixes #61 and #64.
107
155
 
108
- # 2.5.2
156
+ ## [2.5.2]
157
+ ### Fixes
109
158
 
110
159
  - Fix OpenURI::HTTPError exception raised when video_info gem is not able to parse video metadata. Fixes #60.
111
160
 
112
- # 2.5.1
161
+ ## [2.5.1]
162
+ ### Adds
113
163
 
114
164
  - Implement `Set-Cookie` header between http redirections to set cookies when site requires it. Fixes #55.
115
165
 
116
- # 2.5.0
166
+ ## [2.5.0]
167
+ ### Adds
117
168
 
118
169
  - Handles seamlessly `og:image` and `og:image:url`
119
170
  - Handles seamlessly `og:video` and `og:video:url`
120
171
  - Handles `og:video:width` and `og:video:height` for one video only (please create a ticket if you want support for multiple videos/images width & height)
172
+
173
+ ### Fixes
174
+
121
175
  - Fix calling `as_json` on `website` to return `as_json` representation of videos and images, not just their urls
122
- - Gem updates and fix rspec deprecation warnings
123
176
 
124
- # 2.4.0
177
+ ### Changes
178
+
179
+ - Gem updates
180
+
181
+ ## [2.4.0]
182
+ ### Adds
125
183
 
126
184
  - Handle connection through proxy automatically using the `ENV['HTTP_PROXY']` variable thanks to [taganaka](https://github.com/taganaka).
127
185
 
128
- # 2.3.2
186
+ ## [2.3.2]
187
+ ### Fixes
129
188
 
130
189
  - Fix an issue with vimeo opengraph urls. Fixes [#46](https://github.com/gottfrois/link_thumbnailer/pull/46)
131
190
 
132
- # 2.3.1
191
+ ## [2.3.1]
192
+ ### Fixes
133
193
 
134
194
  - Fix an issue with the link density grader caused by links with image instead of text. Fixes [#45](https://github.com/gottfrois/link_thumbnailer/issues/45)
135
195
 
136
- # 2.3.0
196
+ ## [2.3.0]
197
+ ### Adds
137
198
 
138
199
  - Add requested favicon scraper [#40](https://github.com/gottfrois/link_thumbnailer/issues/40)
139
200
 
@@ -151,19 +212,23 @@ o.favicon
151
212
  => "https://github.com/fluidicon.png"
152
213
  ```
153
214
 
154
- # 2.2.3
215
+ ## [2.2.3]
216
+ ### Fixes
155
217
 
156
218
  - Fixes [#41](https://github.com/gottfrois/link_thumbnailer/issues/41)
157
219
 
158
- # 2.2.2
220
+ ## [2.2.2]
221
+ ### Fixes
159
222
 
160
223
  - Fixes [#41](https://github.com/gottfrois/link_thumbnailer/issues/41)
161
224
 
162
- # 2.2.1
225
+ ## [2.2.1]
226
+ ### Fixes
163
227
 
164
- - Fix issue when computing link density ratio
228
+ - Fixes issue when computing link density ratio
165
229
 
166
- # 2.2.0
230
+ ## [2.2.0]
231
+ ### Adds
167
232
 
168
233
  - Add support for `og:video`
169
234
  - Add support for multiple `og:video` as well
@@ -189,30 +254,35 @@ Ex:
189
254
  config.attributes = [:title, :images, :description, :videos]
190
255
  ```
191
256
 
192
- # 2.1.0
257
+ ## [2.1.0]
258
+ ### Adds
193
259
 
194
260
  - Increased `og:image` scraping performance by parsing `og:image:width` and `og:image:height` attribute if specified
195
261
  - Introduced `image_stats` option to allow disabling image size and type parsing causing performance issues.
196
262
 
197
263
  When disabled, size will be `[0, 0]` and type will be `nil`
198
264
 
199
- # 2.0.4
265
+ ## [2.0.4]
266
+ ### Fixes
200
267
 
201
268
  - Fixes [#39](https://github.com/gottfrois/link_thumbnailer/issues/39)
202
269
 
203
- # 2.0.3
270
+ ## [2.0.3]
271
+ ### Fixes
204
272
 
205
273
  - Fixes [#37](https://github.com/gottfrois/link_thumbnailer/issues/37)
206
274
 
207
- # 2.0.2
275
+ ## [2.0.2]
276
+ ### Fixes
208
277
 
209
- - Fix couple of issues with `URI` class namespace
278
+ - Fixes couple of issues with `URI` class namespace
210
279
 
211
- # 2.0.1
280
+ ## [2.0.1]
212
281
 
213
- - Fix issue with image parser (fastimage) when given an URI instance instead of a string
282
+ - Fixes issue with image parser (fastimage) when given an URI instance instead of a string
214
283
 
215
- # 2.0.0
284
+ ## [2.0.0]
285
+ ### Changes
216
286
 
217
287
  - Fully refactored LinkThumbnailer
218
288
  - Introduced [Graders](https://github.com/gottfrois/link_thumbnailer/wiki/How-to-build-your-own-Grader%3F)
@@ -228,60 +298,85 @@ When disabled, size will be `[0, 0]` and type will be `nil`
228
298
  To update from `1.x.x` to `2.x.x` you need to run `rails g link_thumbnailer:install` to get the new configuration file.
229
299
  If you used the `PreviewsController` feature, you need to build it yourself since it is not supported anymore.
230
300
 
231
- # 1.1.2
301
+ ## [1.1.2]
302
+ ### Fixes
232
303
 
233
- - Fix issue with FastImage URLs [https://github.com/gottfrois/link_thumbnailer/pull/31](https://github.com/gottfrois/link_thumbnailer/pull/31)
304
+ - Fixes issue with FastImage URLs [https://github.com/gottfrois/link_thumbnailer/pull/31](https://github.com/gottfrois/link_thumbnailer/pull/31)
234
305
 
235
- # 1.1.1
306
+ ## [1.1.1]
307
+ ### Fixes
236
308
 
237
- - Fix route helper not working under rails 4.
309
+ - Fixes route helper not working under rails 4.
238
310
 
239
- # 1.1.0
311
+ ## [1.1.0]
312
+ ### Changes
240
313
 
241
314
  - Replace RMagick by [FastImage](https://github.com/sdsykes/fastimage)
242
315
  - Rename `rmagick_attributes` config into `image_attributes`
243
316
 
244
- # 1.0.9
317
+ ## [1.0.9]
318
+ ### Fixes
319
+
320
+ - Fixes issue when Location header used a relative path instead of an absolute path
321
+
322
+ ### Changes
245
323
 
246
- - Fix issue when Location header used a relative path instead of an absolute path
247
324
  - Update gemfile to be more flexible when using Hashie gem
248
325
 
249
- # 1.0.8
326
+ ## [1.0.8]
327
+ # Adds
250
328
 
251
329
  - Thanks to [juriglx](https://github.com/juriglx), support for canonical urls
330
+
331
+ ### Fixes
332
+
252
333
  - Bug fixes
253
334
 
254
- # 1.0.7
335
+ ## [1.0.7]
336
+ ### Fixes
255
337
 
256
- - Fix: Issue with preview controller
338
+ - Fixes an issue with the preview controller
257
339
 
258
- # 1.0.6
340
+ ## [1.0.6]
341
+ ### Fixes
259
342
 
260
- - Fix: Issue when setting `strict` option. Always returning OG representation.
343
+ - Fixes an issue when setting `strict` option. Always returning OG representation.
261
344
 
262
- # 1.0.5
345
+ ## [1.0.5]
346
+ ### Adds
263
347
 
264
348
  - Thanks to [phlegx](https://github.com/phlegx), support for timeout http connection through configurations.
265
349
 
266
- # 1.0.4
350
+ ## [1.0.4]
351
+ ### Fixes
352
+
353
+ - Fixes issue #7: nil img was returned when exception is raised. Now skiping nil images in results.
354
+
355
+ ### Adds
267
356
 
268
- - Fix issue #7: nil img was returned when exception is raised. Now skiping nil images in results.
269
357
  - Thanks to [phlegx](https://github.com/phlegx), support for SSL and User Agent customization through configurations.
270
358
 
271
- # 1.0.3
359
+ ## [1.0.3]
360
+ ### Fixes
272
361
 
273
- - Fix issue #5: Url was incorect in case of HTTP Redirections.
362
+ - Fixes issue #5: Url was incorect in case of HTTP Redirections.
274
363
 
275
- # 1.0.2
364
+ ## [1.0.2]
365
+ ### Fixes
276
366
 
277
- - Feature: User can now set options at runtime by passing valid options to ```generate``` method
278
367
  - Bug fix when doing ```rails g link_thumbnailer:install``` by explicitly specifying the scope of Rails
279
368
 
280
- # 1.0.1
369
+ ### Adds
370
+
371
+ - User can now set options at runtime by passing valid options to ```generate``` method
372
+
373
+ ## [1.0.1]
374
+ ### Fixes
281
375
 
282
376
  - Refactor LinkThumbnailer#generate method to have a cleaner code
283
377
 
284
- # 1.0.0
378
+ ## [1.0.0]
379
+ ### Changes
285
380
 
286
381
  - Update readme
287
382
  - Add PreviewController for easy integration with user's app
@@ -289,46 +384,103 @@ If you used the `PreviewsController` feature, you need to build it yourself sinc
289
384
  - Refactor some code
290
385
  - Change 'to_a' method to 'to_hash' in object model
291
386
 
292
- # 0.0.6
387
+ ## [0.0.6]
388
+ ### Adds
293
389
 
294
390
  - Update readme
295
391
  - Add `to_a` to WebImage class
296
- - Refactor `to_json` for WebImage class
297
392
  - Add specs corresponding
298
393
 
299
- # 0.0.5
394
+ ### Fixes
395
+
396
+ - Refactor `to_json` for WebImage class
397
+
398
+ ## [0.0.5]
399
+ ### Fixes
300
400
 
301
401
  - Bug fix
302
402
  - Remove `require 'rails'` from spec_helper.rb
303
403
  - Remove rails dependences (blank? method) in code
304
404
  - Spec fix
305
405
 
306
- # 0.0.4
406
+ ## [0.0.4]
407
+ ### Adds
307
408
 
308
409
  - Add specs for almost all classes
309
410
  - Add a method `to_json` for WebImage class to be able to get a usable array of images' attributes
310
411
 
311
- # 0.0.3
412
+ ## [0.0.3]
413
+ ### Adds
312
414
 
313
415
  - Add specs for LinkThumbnailer class
416
+
417
+ ### Fixes
418
+
314
419
  - Refactor config system, now using dedicated configuration class
315
420
 
316
- # 0.0.2
421
+ ## [0.0.2]
422
+ ### Adds
317
423
 
318
424
  - Added Rspec
319
- - Bug fixes:
320
- - Now checking if attribute is blank for LinkThumbnailer::Object.valid? method
321
-
322
- # 0.0.1
323
-
324
- - LinkThumbnailer::Object
325
- - LinkThumbnailer::Doc
326
- - LinkThumbnailer::DocParser
327
- - LinkThumbnailer::Fetcher
328
- - LinkThumbnailer::ImgComparator
329
- - LinkThumbnailer::ImgParser
330
- - LinkThumbnailer::ImgUrlFilter
331
- - LinkThumbnailer::Opengraph
332
- - LinkThumbnailer::WebImage
333
- - LinkThumbnailer.configure
334
- - LinkThumbnailer.generate
425
+
426
+ ### Fixes
427
+
428
+ - Now checking if attribute is blank for LinkThumbnailer::Object.valid? method
429
+
430
+ ## [0.0.1]
431
+ ### Adds
432
+
433
+ - First release 🎆
434
+
435
+ [Unreleased]: https://github.com/gottfrois/link_thumbnailer/compare/v3.4.0...HEAD
436
+ [3.4.0]: https://github.com/gottfrois/link_thumbnailer/compare/v3.3.2...v3.4.0
437
+ [3.3.2]: https://github.com/gottfrois/link_thumbnailer/compare/v3.3.1...v3.3.2
438
+ [3.3.1]: https://github.com/gottfrois/link_thumbnailer/compare/v3.3.0...v3.3.1
439
+ [3.3.0]: https://github.com/gottfrois/link_thumbnailer/compare/v3.2.1...v3.3.0
440
+ [3.2.1]: https://github.com/gottfrois/link_thumbnailer/compare/v3.2.0...v3.2.1
441
+ [3.2.0]: https://github.com/gottfrois/link_thumbnailer/compare/v3.1.2...v3.2.0
442
+ [3.1.2]: https://github.com/gottfrois/link_thumbnailer/compare/v3.1.1...v3.1.2
443
+ [3.1.1]: https://github.com/gottfrois/link_thumbnailer/compare/v3.1.0...v3.1.1
444
+ [3.1.0]: https://github.com/gottfrois/link_thumbnailer/compare/v3.0.3...v3.1.0
445
+ [3.0.3]: https://github.com/gottfrois/link_thumbnailer/compare/v3.0.2...v3.0.3
446
+ [3.0.2]: https://github.com/gottfrois/link_thumbnailer/compare/v3.0.1...v3.0.2
447
+ [3.0.1]: https://github.com/gottfrois/link_thumbnailer/compare/v3.0.0...v3.0.1
448
+ [3.0.0]: https://github.com/gottfrois/link_thumbnailer/compare/v2.6.1...v3.0.0
449
+ [2.6.1]: https://github.com/gottfrois/link_thumbnailer/compare/v2.6.0...v2.6.1
450
+ [2.6.0]: https://github.com/gottfrois/link_thumbnailer/compare/v2.5.2...v2.6.0
451
+ [2.5.2]: https://github.com/gottfrois/link_thumbnailer/compare/v2.5.1...v2.5.2
452
+ [2.5.1]: https://github.com/gottfrois/link_thumbnailer/compare/v2.5.0...v2.5.1
453
+ [2.5.0]: https://github.com/gottfrois/link_thumbnailer/compare/v2.4.0...v2.5.0
454
+ [2.4.0]: https://github.com/gottfrois/link_thumbnailer/compare/v2.3.2...v2.4.0
455
+ [2.3.2]: https://github.com/gottfrois/link_thumbnailer/compare/v2.3.1...v2.3.2
456
+ [2.3.1]: https://github.com/gottfrois/link_thumbnailer/compare/v2.3.0...v2.3.1
457
+ [2.3.0]: https://github.com/gottfrois/link_thumbnailer/compare/v2.2.3...v2.3.0
458
+ [2.2.3]: https://github.com/gottfrois/link_thumbnailer/compare/v2.2.2...v2.2.3
459
+ [2.2.2]: https://github.com/gottfrois/link_thumbnailer/compare/v2.2.1...v2.2.2
460
+ [2.2.1]: https://github.com/gottfrois/link_thumbnailer/compare/v2.2.0...v2.2.1
461
+ [2.2.0]: https://github.com/gottfrois/link_thumbnailer/compare/v2.1.0...v2.2.0
462
+ [2.1.0]: https://github.com/gottfrois/link_thumbnailer/compare/v2.0.4...v2.1.0
463
+ [2.0.4]: https://github.com/gottfrois/link_thumbnailer/compare/v2.0.3...v2.0.4
464
+ [2.0.3]: https://github.com/gottfrois/link_thumbnailer/compare/v2.0.2...v2.0.3
465
+ [2.0.2]: https://github.com/gottfrois/link_thumbnailer/compare/v2.0.1...v2.0.2
466
+ [2.0.1]: https://github.com/gottfrois/link_thumbnailer/compare/v2.0.0...v2.0.1
467
+ [2.0.0]: https://github.com/gottfrois/link_thumbnailer/compare/v1.1.2...v2.0.0
468
+ [1.1.2]: https://github.com/gottfrois/link_thumbnailer/compare/v1.1.1...v1.1.2
469
+ [1.1.1]: https://github.com/gottfrois/link_thumbnailer/compare/v1.1.0...v1.1.1
470
+ [1.1.0]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.9...v1.1.0
471
+ [1.0.9]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.8...v1.0.9
472
+ [1.0.8]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.7...v1.0.8
473
+ [1.0.7]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.6...v1.0.7
474
+ [1.0.6]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.5...v1.0.6
475
+ [1.0.5]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.4...v1.0.5
476
+ [1.0.4]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.3...v1.0.4
477
+ [1.0.3]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.2...v1.0.3
478
+ [1.0.2]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.1...v1.0.2
479
+ [1.0.1]: https://github.com/gottfrois/link_thumbnailer/compare/v1.0.0...v1.0.1
480
+ [1.0.0]: https://github.com/gottfrois/link_thumbnailer/compare/v0.0.6...v1.0.0
481
+ [0.0.6]: https://github.com/gottfrois/link_thumbnailer/compare/v0.0.5...v0.0.6
482
+ [0.0.5]: https://github.com/gottfrois/link_thumbnailer/compare/v0.0.4...v0.0.5
483
+ [0.0.4]: https://github.com/gottfrois/link_thumbnailer/compare/v0.0.3...v0.0.4
484
+ [0.0.3]: https://github.com/gottfrois/link_thumbnailer/compare/v0.0.2...v0.0.3
485
+ [0.0.2]: https://github.com/gottfrois/link_thumbnailer/compare/v0.0.1...v0.0.2
486
+ [0.0.1]: https://github.com/gottfrois/link_thumbnailer/releases/tag/v0.0.1
@@ -35,6 +35,11 @@ LinkThumbnailer.configure do |config|
35
35
  #
36
36
  # config.attributes = [:title, :images, :description, :videos, :favicon]
37
37
 
38
+ # Prior favicon size. If the website doesn't have such size - returns the first favicon.
39
+ # Value should be like '32x32' or '16x16'. Default value is nil.
40
+ #
41
+ # config.favicon_size = nil
42
+
38
43
  # List of procedures used to rate the website description. Add you custom class
39
44
  # here. See wiki for more details on how to build your own graders.
40
45
  #
@@ -83,6 +88,10 @@ LinkThumbnailer.configure do |config|
83
88
  #
84
89
  # config.scrapers = [:opengraph, :default]
85
90
 
91
+ # Limit for download size in bytes. When using ActiveSupport, you can also use values like 10.megabytes
92
+ #
93
+ # config.download_size_limit = 10 * 1024 * 1024
94
+
86
95
  # Sets the default encoding.
87
96
  #
88
97
  # config.encoding = 'utf-8'
@@ -28,7 +28,8 @@ module LinkThumbnailer
28
28
  :verify_ssl, :http_open_timeout, :http_read_timeout, :attributes,
29
29
  :graders, :description_min_length, :positive_regex, :negative_regex,
30
30
  :image_limit, :image_stats, :raise_on_invalid_format, :max_concurrency,
31
- :scrapers, :http_override_headers, :encoding
31
+ :scrapers, :http_override_headers, :download_size_limit, :encoding,
32
+ :favicon_size
32
33
 
33
34
  alias_method :http_timeout, :http_open_timeout
34
35
  alias_method :http_timeout=, :http_open_timeout=
@@ -65,6 +66,7 @@ module LinkThumbnailer
65
66
  @max_concurrency = 20
66
67
  @scrapers = [:opengraph, :default]
67
68
  @http_override_headers = { 'Accept-Encoding' => 'none' }
69
+ @download_size_limit = 10 * 1024 * 1024
68
70
  @encoding = 'utf-8'
69
71
  end
70
72
 
@@ -8,4 +8,5 @@ module LinkThumbnailer
8
8
  ScraperInvalid = Class.new(Exceptions)
9
9
  HTTPError = Class.new(Exceptions)
10
10
  SyntaxError = Class.new(Exceptions)
11
+ DownloadSizeLimit = Class.new(Exceptions)
11
12
  end
@@ -12,9 +12,9 @@ module LinkThumbnailer
12
12
  def sanitize(str)
13
13
  return unless str
14
14
 
15
- str = str.dup
16
- str.encode!("UTF-16", "UTF-8", invalid: :replace, undef: :replace, replace: "")
17
- str.encode!("UTF-8", "UTF-16").strip.gsub(/[\r\n\f]+/, "\n")
15
+ str = str.encode("UTF-16", "UTF-8", invalid: :replace, undef: :replace, replace: "")
16
+ str = str.encode("UTF-8", "UTF-16").strip.gsub(/[\r\n\f]+/, "\n")
17
+ str
18
18
  end
19
19
  end
20
20
  end
@@ -17,7 +17,7 @@ module LinkThumbnailer
17
17
  end
18
18
 
19
19
  def generate
20
- @source = processor.call(url)
20
+ @source = processor.start(url)
21
21
  scraper.call
22
22
  end
23
23
 
@@ -17,6 +17,12 @@ module LinkThumbnailer
17
17
  super(config)
18
18
  end
19
19
 
20
+ def start(url)
21
+ result = call(url)
22
+ shutdown
23
+ result
24
+ end
25
+
20
26
  def call(url = '', redirect_count = 0, headers = {})
21
27
  self.url = url
22
28
  @redirect_count = redirect_count
@@ -28,12 +34,16 @@ module LinkThumbnailer
28
34
  set_http_options
29
35
  perform_request
30
36
  end
31
- rescue ::Net::HTTPExceptions, ::SocketError, ::Timeout::Error => e
37
+ rescue ::Net::HTTPExceptions, ::SocketError, ::Timeout::Error, ::Net::HTTP::Persistent::Error => e
32
38
  raise ::LinkThumbnailer::HTTPError.new(e.message)
33
39
  end
34
40
 
35
41
  private
36
42
 
43
+ def shutdown
44
+ http.shutdown
45
+ end
46
+
37
47
  def with_valid_url
38
48
  raise ::LinkThumbnailer::BadUriFormat unless valid_url_format?
39
49
  yield if block_given?
@@ -53,7 +63,7 @@ module LinkThumbnailer
53
63
  end
54
64
 
55
65
  def perform_request
56
- response = http.request(url)
66
+ response = request_in_chunks
57
67
  headers = {}
58
68
  headers['Cookie'] = response['Set-Cookie'] if response['Set-Cookie'].present?
59
69
 
@@ -73,6 +83,19 @@ module LinkThumbnailer
73
83
  end
74
84
  end
75
85
 
86
+ def request_in_chunks
87
+ body = String.new
88
+ response = http.request(url) do |resp|
89
+ raise ::LinkThumbnailer::DownloadSizeLimit if too_big_download_size?(resp.content_length)
90
+ resp.read_body do |chunk|
91
+ body.concat(chunk)
92
+ raise ::LinkThumbnailer::DownloadSizeLimit if too_big_download_size?(body.length)
93
+ end
94
+ end
95
+ response.body = body
96
+ response
97
+ end
98
+
76
99
  def resolve_relative_url(location)
77
100
  location.start_with?('http') ? location : build_absolute_url_for(location)
78
101
  end
@@ -101,6 +124,10 @@ module LinkThumbnailer
101
124
  config.verify_ssl
102
125
  end
103
126
 
127
+ def download_size_limit
128
+ config.download_size_limit
129
+ end
130
+
104
131
  def too_many_redirections?
105
132
  redirect_count > redirect_limit
106
133
  end
@@ -120,6 +147,10 @@ module LinkThumbnailer
120
147
  false
121
148
  end
122
149
 
150
+ def too_big_download_size?(size)
151
+ size.to_i > download_size_limit.to_i
152
+ end
153
+
123
154
  def url=(url)
124
155
  @url = ::URI.parse(url.to_s)
125
156
  end
@@ -18,8 +18,8 @@ module LinkThumbnailer
18
18
 
19
19
  def extract_charset
20
20
  content_type = @response['Content-Type'] || ''
21
- m = content_type.match(/charset=(\w+)/)
22
- (m && m[1]) || ''
21
+ m = content_type.match(/charset=([\w-]+)/)
22
+ (m && m[1]) || @response.body.scrub =~ /<meta[^>]*charset\s*=\s*["']?(.+?)["' >]/i && $1 || ''
23
23
  end
24
24
 
25
25
  def extract_body
@@ -15,7 +15,11 @@ module LinkThumbnailer
15
15
  private
16
16
 
17
17
  def to_uri(href)
18
- ::URI.parse(href)
18
+ uri = ::URI.parse(href)
19
+ uri.scheme ||= website.url.scheme
20
+ uri.host ||= website.url.host
21
+ uri.path = uri.path&.sub(%r{^(?=[^\/])}, '/')
22
+ uri
19
23
  rescue ::URI::InvalidURIError
20
24
  nil
21
25
  end
@@ -25,13 +29,21 @@ module LinkThumbnailer
25
29
  end
26
30
 
27
31
  def node
28
- document.xpath("//link[contains(@rel, 'icon')]").first
32
+ icons = document.xpath("//link[contains(@rel, 'icon')]")
33
+ retrieve_by_size(icons) || icons.first
29
34
  end
30
35
 
31
36
  def modelize(uri)
32
37
  model_class.new(uri)
33
38
  end
34
39
 
40
+ def retrieve_by_size(icons)
41
+ return if config.favicon_size.nil?
42
+
43
+ icons.find do |icon|
44
+ icon.attributes['sizes']&.value == config.favicon_size
45
+ end
46
+ end
35
47
  end
36
48
  end
37
49
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module LinkThumbnailer
4
- VERSION = '3.3.2'
4
+ VERSION = '3.4.0'
5
5
  end
@@ -25,5 +25,5 @@ Gem::Specification.new do |spec|
25
25
  spec.add_dependency 'nokogiri', '>= 1.6'
26
26
  spec.add_dependency 'net-http-persistent', '>= 2.9'
27
27
  spec.add_dependency 'video_info', '>= 2.6'
28
- spec.add_dependency 'image_info', '>= 1.0'
28
+ spec.add_dependency 'image_info', ['~> 1.0', '>= 1.2.0']
29
29
  end
@@ -109,6 +109,25 @@ describe 'Fixture' do
109
109
 
110
110
  end
111
111
 
112
+ context 'with 32 favicon size' do
113
+ let(:action) { LinkThumbnailer.generate(url, favicon_size: '32x32') }
114
+ let(:favicon) { 'http://foo.com/foo32x32.ico' }
115
+ let(:html) { File.open(File.dirname(__FILE__) + '/fixtures/default_with_few_favicons.html').read }
116
+
117
+ it { expect(action.favicon).to eq(favicon) }
118
+ end
119
+
120
+ context 'when favicon with root path in the href' do
121
+ let(:html) { File.open(File.dirname(__FILE__) + '/fixtures/with_root_path_in_href.html').read }
122
+
123
+ it { expect(action.favicon).to eq(favicon) }
124
+ end
125
+
126
+ context 'when favicon with related path in the href' do
127
+ let(:html) { File.open(File.dirname(__FILE__) + '/fixtures/with_related_path_in_href.html').read }
128
+
129
+ it { expect(action.favicon).to eq(favicon) }
130
+ end
112
131
  end
113
132
 
114
133
  end
@@ -0,0 +1,15 @@
1
+ <html>
2
+ <head>
3
+ <title>Title from meta</title>
4
+ <link rel="shortcut icon" href="http://foo.com/foo.ico">
5
+ <link rel="shortcut icon" sizes='32x32' href="http://foo.com/foo32x32.ico">
6
+ <link rel="shortcut icon" sizes='16x16' href="http://foo.com/foo16x16.ico">
7
+ </head>
8
+ <body>
9
+
10
+ <p>Description from body</p>
11
+
12
+ <img src="http://foo.com/foo.png">
13
+
14
+ </body>
15
+ </html>
@@ -0,0 +1,6 @@
1
+ <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="ja"><head><meta content="世界中のあらゆる情報を検索するためのツールを提供しています。さまざまな検索機能を活用して、お探しの情報を見つけてください。" name="description"><meta content="noodp" name="robots"><meta content="text/html" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:'716VWJL3JYq18QWo_qzoAg',kEXPI:'750722,1351903,1352241,1352381,3700253,3700347,4028875,4029815,4032677,4038012,4043492,4045839,4048347,4062666,4065787,4068816,4069773,4069838,4069840,4070138,4072773,4073405,4073726,4073959,4076096,4076931,4076999,4077777,4078438,4079081,4079105,4079894,4081038,4081485,4082194,4082201,4082298,4082619,4083044,4083476,4084343,4084673,4085336,4085412,4085683,4086011,4089003,4089144,4089183,4089427,4089538,4089913,4090414,4090547,4090549,4090598,4090657,4090806,4090893,4091966,4092028,4092182,4092218,4092474,4092478,4092598,4092864,4092867,4092875,4092897,4092934,4093073,4093948,4094169,4094250,4094769,4094987,4094997,4095554,4095771,4095907,4095998,8300096,8300272,8507380,8507419,8507861,8507940,8508624,8510023,10200083,13500022,13500024',authuser:0,kscs:'c9c918f0_24'};google.kHL='ja';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};google.wl=function(a,b){try{google.ml(Error(a),!1,b)}catch(c){}};google.time=function(){return(new Date).getTime()};google.log=function(a,b,c,d,g){a=google.logUrl(a,b,c,d,g);if(""!=a){b=new Image;var e=google.lc,f=google.li;e[f]=b;b.onerror=b.onload=b.onabort=function(){delete e[f]};window.google&&window.google.vel&&window.google.vel.lu&&window.google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,c,d,g){var e="",f=google.ls||"";c||-1!=b.search("&ei=")||(e="&ei="+google.getEI(d),-1==b.search("&lei=")&&(d=google.getLEI(d))&&(e+="&lei="+d));a=c||"/"+(g||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+e+f+"&zx="+google.time();/^http:/i.test(a)&&google.https()&&(google.ml(Error("a"),!1,{src:a,glmm:1}),a="");return a};google.y={};google.x=function(a,b){google.y[a.id]=[a,b];return!1};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};}).call(this);var a=window.location,b=a.href.indexOf("#");if(0<=b){var c=a.href.substring(b+1);/(^|&)q=/.test(c)&&-1==c.indexOf("#")&&a.replace("/search?"+c.replace(/(^|&)fp=[^&]*/g,"")+"&cad=h")};</script><style>#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}
2
+ </style><style>body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}.ts td{padding:0}.ts{border-collapse:collapse}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-left:4px}input{font-family:inherit}a.gb1,a.gb2,a.gb3,a.gb4{color:#11c !important}body{background:#fff;color:black}a{color:#11c;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}a.gb1,a.gb4{text-decoration:underline}a.gb3:hover{text-decoration:none}#ghead a.gb2:hover{color:#fff !important}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;border-color:#ccc #999 #999 #ccc;height:30px}.lsbb{display:block}.ftl,#fll a{display:inline-block;margin:0 12px}.lsb{background:url(/images/nav_logo229.png) 0 -261px repeat-x;border:none;color:#000;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#ccc}.lst:focus{outline:none}</style><script></script><link href="/images/branding/product/ico/googleg_lodp.ico" rel="shortcut icon"></head><body bgcolor="#fff"><script>(function(){var src='/images/nav_logo229.png';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}
3
+ if (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}
4
+ }
5
+ })();</script><div id="mngb"> <div id=gbar><nobr><b class=gb1>検索</b> <a class=gb1 href="https://www.google.co.jp/imghp?hl=ja&tab=wi">画像</a> <a class=gb1 href="https://maps.google.co.jp/maps?hl=ja&tab=wl">マップ</a> <a class=gb1 href="https://play.google.com/?hl=ja&tab=w8">Play</a> <a class=gb1 href="https://www.youtube.com/?gl=JP&tab=w1">YouTube</a> <a class=gb1 href="https://news.google.co.jp/nwshp?hl=ja&tab=wn">ニュース</a> <a class=gb1 href="https://mail.google.com/mail/?tab=wm">Gmail</a> <a class=gb1 href="https://drive.google.com/?tab=wo">ドライブ</a> <a class=gb1 style="text-decoration:none" href="https://www.google.co.jp/intl/ja/options/"><u>もっと見る</u> &raquo;</a></nobr></div><div id=guser width=100%><nobr><span id=gbn class=gbi></span><span id=gbf class=gbf></span><span id=gbe></span><a href="http://www.google.co.jp/history/optout?hl=ja" class=gb4>ウェブ履歴</a> | <a href="/preferences?hl=ja" class=gb4>設定</a> | <a target=_top id=gb_70 href="https://accounts.google.com/ServiceLogin?hl=ja&passive=true&continue=https://www.google.co.jp/" class=gb4>ログイン</a></nobr></div><div class=gbh style=left:0></div><div class=gbh style=right:0></div> </div><center><br clear="all" id="lgpd"><div id="lga"><div style="padding:28px 0 3px"><div style="height:110px;width:276px;background:url(/images/branding/googlelogo/1x/googlelogo_white_background_color_272x92dp.png) no-repeat" title="Google" align="left" id="hplogo" onload="window.lol&&lol()"><div style="color:#777;font-size:16px;font-weight:bold;position:relative;top:70px;left:218px" nowrap="">日本</div></div></div><br></div><form action="/search" name="f"><table cellpadding="0" cellspacing="0"><tr valign="top"><td width="25%">&nbsp;</td><td align="center" nowrap=""><input name="ie" value="Shift_JIS" type="hidden"><input value="ja" name="hl" type="hidden"><input name="source" type="hidden" value="hp"><input name="biw" type="hidden"><input name="bih" type="hidden"><div class="ds" style="height:32px;margin:4px 0"><input style="color:#000;margin:0;padding:5px 8px 0 6px;vertical-align:top" autocomplete="off" class="lst" value="" title="Google 検索" maxlength="2048" name="q" size="57"></div><br style="line-height:0"><span class="ds"><span class="lsbb"><input class="lsb" value="Google 検索" name="btnG" type="submit"></span></span><span class="ds"><span class="lsbb"><input class="lsb" value="I'm Feeling Lucky" name="btnI" onclick="if(this.form.q.value)this.checked=1; else top.location='/doodles/'" type="submit"></span></span></td><td class="fl sblc" align="left" nowrap="" width="25%"><a href="/advanced_search?hl=ja&amp;authuser=0">検索オプション</a><a href="/language_tools?hl=ja&amp;authuser=0">言語ツール</a></td></tr></table><input id="gbv" name="gbv" type="hidden" value="1"></form><div id="gac_scont"></div><div style="font-size:83%;min-height:3.5em"><br></div><span id="footer"><div style="font-size:10pt"><div style="margin:19px auto;text-align:center" id="fll"><a href="/intl/ja/ads/">広告掲載</a><a href="http://www.google.co.jp/intl/ja/services/">ビジネス ソリューション</a><a href="https://plus.google.com/115899767381375908215" rel="publisher">+Google</a><a href="/intl/ja/about.html">Google について</a><a href="https://www.google.co.jp/setprefdomain?prefdom=US&amp;sig=__NSwOcr0nRFmauaxJWrHJmztdkTc%3D" id="fehl">Google.com</a></div></div><p style="color:#767676;font-size:8pt">&copy; 2017 - <a href="/intl/ja/policies/privacy/">プライバシー</a> - <a href="/intl/ja/policies/terms/">規約</a></p></span></center><script>(function(){window.google.cdo={height:0,width:0};(function(){var a=window.innerWidth,b=window.innerHeight;if(!a||!b)var c=window.document,d="CSS1Compat"==c.compatMode?c.documentElement:c.body,a=d.clientWidth,b=d.clientHeight;a&&b&&(a!=google.cdo.width||b!=google.cdo.height)&&google.log("","","/client_204?&atyp=i&biw="+a+"&bih="+b+"&ei="+google.kEI);}).call(this);})();</script><div id="xjsd"></div><div id="xjsi"><script>(function(){function c(b){window.setTimeout(function(){var a=document.createElement("script");a.src=b;document.getElementById("xjsd").appendChild(a)},0)}google.dljp=function(b,a){google.xjsu=b;c(a)};google.dlj=c;}).call(this);(function(){window.google.xjsrm=[];})();if(google.y)google.y.first=[];if(!google.xjs){window._=window._||{};window._._DumpException=function(e){throw e};if(google.timers&&google.timers.load.t){google.timers.load.t.xjsls=new Date().getTime();}google.dljp('/xjs/_/js/k\x3dxjs.hp.en_US.usfcb5N0rIw.O/m\x3dsb_he,d/rt\x3dj/d\x3d1/t\x3dzcms/rs\x3dACT90oH_6o1ZdJ1OdBerX9LppN-xEd11Eg','/xjs/_/js/k\x3dxjs.hp.en_US.usfcb5N0rIw.O/m\x3dsb_he,d/rt\x3dj/d\x3d1/t\x3dzcms/rs\x3dACT90oH_6o1ZdJ1OdBerX9LppN-xEd11Eg');google.xjs=1;}google.pmc={"sb_he":{"agen":true,"cgen":true,"client":"heirloom-hp","dh":true,"dhqt":true,"ds":"","fl":true,"host":"google.co.jp","isbh":28,"jam":0,"jsonp":true,"msgs":{"cibl":"検索をクリア","dym":"もしかして:","lcky":"I\u0026#39;m Feeling Lucky","lml":"詳細","oskt":"入力ツール","psrc":"この検索キーワードは\u003Ca href=\"/history\"\u003Eウェブ履歴\u003C/a\u003Eから削除されました","psrl":"削除","sbit":"画像で検索","srch":"Google 検索"},"nds":true,"ovr":{},"pq":"","refpd":true,"refspre":true,"rfs":[],"scd":10,"sce":5,"stok":"WXDdTX_0YGr0J9JRNcHRvk23BlI"},"d":{}};google.y.first.push(function(){if(google.med){google.med('init');google.initHistory();google.med('history');}});if(google.j&&google.j.en&&google.j.xi){window.setTimeout(google.j.xi,0);}
6
+ </script></div></body></html>
@@ -0,0 +1,13 @@
1
+ <html>
2
+ <head>
3
+ <title>Title from meta</title>
4
+ <link rel="shortcut icon" href="foo.ico">
5
+ </head>
6
+ <body>
7
+
8
+ <p>Description from body</p>
9
+
10
+ <img src="http://foo.com/foo.png">
11
+
12
+ </body>
13
+ </html>
@@ -0,0 +1,13 @@
1
+ <html>
2
+ <head>
3
+ <title>Title from meta</title>
4
+ <link rel="shortcut icon" href="/foo.ico">
5
+ </head>
6
+ <body>
7
+
8
+ <p>Description from body</p>
9
+
10
+ <img src="http://foo.com/foo.png">
11
+
12
+ </body>
13
+ </html>
@@ -7,7 +7,6 @@ describe LinkThumbnailer::Processor do
7
7
  let(:page) { ::LinkThumbnailer::Page.new(url, {}) }
8
8
  let(:instance) { described_class.new }
9
9
  let(:url) { 'http://foo.com' }
10
-
11
10
  before do
12
11
  allow(LinkThumbnailer).to receive(:page).and_return(page)
13
12
  end
@@ -15,6 +14,7 @@ describe LinkThumbnailer::Processor do
15
14
  describe '#call' do
16
15
 
17
16
  let(:action) { instance.call(url) }
17
+ let(:https_action) { instance.call(https_url) }
18
18
 
19
19
  context 'when redirect_count is greater than config' do
20
20
 
@@ -44,6 +44,16 @@ describe LinkThumbnailer::Processor do
44
44
 
45
45
  end
46
46
 
47
+ context 'on http unauthorized error' do
48
+
49
+ before do
50
+ stub_request(:get, url).to_return(status: 401, body: '', headers: {})
51
+ end
52
+
53
+ it { expect { action }.to raise_error(LinkThumbnailer::HTTPError) }
54
+
55
+ end
56
+
47
57
  context 'on http redirection' do
48
58
 
49
59
  let(:body) { 'foo' }
@@ -202,31 +212,41 @@ describe LinkThumbnailer::Processor do
202
212
 
203
213
  end
204
214
 
205
- context 'when access non-utf8-encoding website' do
215
+ context 'when access non-utf8 encoded website' do
216
+
206
217
  let(:code) { 200 }
207
- let(:body_shift_jis) do
208
- File.read(File.expand_path('fixtures/google_shift_jis.html', File.dirname(__FILE__)))
209
- end
210
- let(:body_utf8) do
211
- File.read(File.expand_path('fixtures/google_utf8.html', File.dirname(__FILE__)))
212
- end
213
- let(:response) do
214
- r = ::Net::HTTPSuccess.new('', code, body_shift_jis)
215
- r['Content-Type'] = 'text/html'
216
- r.body = body_shift_jis
217
- r.instance_variable_set(:@read, true)
218
- r
218
+ let(:utf8_encoded_body) { File.read(File.expand_path('fixtures/google_utf8.html', File.dirname(__FILE__))) }
219
+ let(:shift_jis_encoded_body) { File.read(File.expand_path('fixtures/google_shift_jis.html', File.dirname(__FILE__))) }
220
+ let(:response) { ::Net::HTTPSuccess.new('', code, body) }
221
+
222
+ before do
223
+ allow(response).to receive(:body).and_return(body)
219
224
  end
220
225
 
221
- context 'when http success with valid charset provided in Content-Type' do
222
- before { response['Content-Type'] = 'text/html; charset=Shift_JIS' }
223
- it { expect(action).to eq body_utf8 }
226
+ context 'when http success with valid charset provided in Content-Type header' do
227
+
228
+ let(:body) { shift_jis_encoded_body }
229
+
230
+ before do
231
+ response['Content-Type'] = 'text/html; charset=Shift-JIS'
232
+ end
233
+
234
+ it { expect(action).to eq(shift_jis_encoded_body) }
235
+
224
236
  end
225
237
 
226
- context 'when http success with invalid charset provided in Content-Type' do
227
- before { response['Content-Type'] = 'text/html; charset=Shift-JIS' }
228
- it { expect(action).to eq body_shift_jis }
238
+ context 'when http success with valid charset provided in Content-Type header' do
239
+
240
+ let(:body) { shift_jis_encoded_body }
241
+
242
+ before do
243
+ response['Content-Type'] = 'text/html; charset=Shift_JIS'
244
+ end
245
+
246
+ it { expect(action).to eq(utf8_encoded_body) }
247
+
229
248
  end
249
+
230
250
  end
231
251
 
232
252
  context 'when http redirection' do
@@ -24,12 +24,16 @@ describe LinkThumbnailer::Response do
24
24
  File.read(File.expand_path('fixtures/google_utf8.html', File.dirname(__FILE__)))
25
25
  end
26
26
 
27
+ let(:body_utf8_no_meta_charset) do
28
+ File.read(File.expand_path('fixtures/google_utf8_no_meta_charset.html', File.dirname(__FILE__)))
29
+ end
30
+
27
31
  before do
28
32
  allow(LinkThumbnailer).to receive(:page).and_return(page)
29
33
  end
30
34
 
31
35
  describe '#charset' do
32
- context 'when charset provided in content-type' do
36
+ context 'when charset provided in content-type "Shift_JIS"' do
33
37
  before do
34
38
  response['Content-Type'] = 'text/html; charset=Shift_JIS'
35
39
  end
@@ -37,7 +41,25 @@ describe LinkThumbnailer::Response do
37
41
  it { expect(instance.charset).to eq 'Shift_JIS' }
38
42
  end
39
43
 
40
- context 'when no charset available in content-type' do
44
+ context 'when charset provided in content-type "utf-8"' do
45
+ before do
46
+ response['Content-Type'] = 'text/html; charset=utf-8'
47
+ end
48
+
49
+ it { expect(instance.charset).to eq 'utf-8' }
50
+ end
51
+
52
+ context 'when no charset available in content-type and charset provided in meta tag "UTF-8"' do
53
+ before do
54
+ response.body = body_utf8
55
+ end
56
+ it { expect(instance.charset).to eq 'UTF-8' }
57
+ end
58
+
59
+ context 'when no charset available in content-type and body' do
60
+ before do
61
+ response.body = body_utf8_no_meta_charset
62
+ end
41
63
  it { expect(instance.charset).to eq '' }
42
64
  end
43
65
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: link_thumbnailer
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.3.2
4
+ version: 3.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Pierre-Louis Gottfrois
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2018-03-22 00:00:00.000000000 Z
11
+ date: 2020-07-24 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -98,16 +98,22 @@ dependencies:
98
98
  name: image_info
99
99
  requirement: !ruby/object:Gem::Requirement
100
100
  requirements:
101
- - - ">="
101
+ - - "~>"
102
102
  - !ruby/object:Gem::Version
103
103
  version: '1.0'
104
+ - - ">="
105
+ - !ruby/object:Gem::Version
106
+ version: 1.2.0
104
107
  type: :runtime
105
108
  prerelease: false
106
109
  version_requirements: !ruby/object:Gem::Requirement
107
110
  requirements:
108
- - - ">="
111
+ - - "~>"
109
112
  - !ruby/object:Gem::Version
110
113
  version: '1.0'
114
+ - - ">="
115
+ - !ruby/object:Gem::Version
116
+ version: 1.2.0
111
117
  description: Ruby gem generating thumbnail images from a given URL.
112
118
  email:
113
119
  - pierrelouis.gottfrois@gmail.com
@@ -177,13 +183,17 @@ files:
177
183
  - spec/fixtures/bar.png
178
184
  - spec/fixtures/default_from_body.html
179
185
  - spec/fixtures/default_from_meta.html
186
+ - spec/fixtures/default_with_few_favicons.html
180
187
  - spec/fixtures/foo.png
181
188
  - spec/fixtures/google_shift_jis.html
182
189
  - spec/fixtures/google_utf8.html
190
+ - spec/fixtures/google_utf8_no_meta_charset.html
183
191
  - spec/fixtures/og_not_valid_example.html
184
192
  - spec/fixtures/og_valid_example.html
185
193
  - spec/fixtures/og_valid_multi_image_example.html
186
194
  - spec/fixtures/og_valid_multi_video_example.html
195
+ - spec/fixtures/with_related_path_in_href.html
196
+ - spec/fixtures/with_root_path_in_href.html
187
197
  - spec/grader_spec.rb
188
198
  - spec/graders/base_spec.rb
189
199
  - spec/graders/html_attribute_spec.rb
@@ -211,7 +221,7 @@ files:
211
221
  homepage: https://github.com/gottfrois/link_thumbnailer
212
222
  licenses: []
213
223
  metadata: {}
214
- post_install_message:
224
+ post_install_message:
215
225
  rdoc_options: []
216
226
  require_paths:
217
227
  - lib
@@ -226,9 +236,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
226
236
  - !ruby/object:Gem::Version
227
237
  version: '0'
228
238
  requirements: []
229
- rubyforge_project:
230
- rubygems_version: 2.6.14
231
- signing_key:
239
+ rubygems_version: 3.0.8
240
+ signing_key:
232
241
  specification_version: 4
233
242
  summary: Ruby gem ranking images from a given URL returning an object containing images
234
243
  and website informations.
@@ -238,13 +247,17 @@ test_files:
238
247
  - spec/fixtures/bar.png
239
248
  - spec/fixtures/default_from_body.html
240
249
  - spec/fixtures/default_from_meta.html
250
+ - spec/fixtures/default_with_few_favicons.html
241
251
  - spec/fixtures/foo.png
242
252
  - spec/fixtures/google_shift_jis.html
243
253
  - spec/fixtures/google_utf8.html
254
+ - spec/fixtures/google_utf8_no_meta_charset.html
244
255
  - spec/fixtures/og_not_valid_example.html
245
256
  - spec/fixtures/og_valid_example.html
246
257
  - spec/fixtures/og_valid_multi_image_example.html
247
258
  - spec/fixtures/og_valid_multi_video_example.html
259
+ - spec/fixtures/with_related_path_in_href.html
260
+ - spec/fixtures/with_root_path_in_href.html
248
261
  - spec/grader_spec.rb
249
262
  - spec/graders/base_spec.rb
250
263
  - spec/graders/html_attribute_spec.rb