gs_img_fetcher 0.1.0 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: fbf77d3d6f77afcb8fa99b171c7fa0ea5aa3bcb729c1fc34c281af653b48f0a6
4
- data.tar.gz: 1ccc20a72beded5b5699a6a965fa5ce0fd2b0e44ac0bf84bb96237e6b752bda2
3
+ metadata.gz: a9349da7d358fea7bb7ae7bb75472a396cd6810e3348f1ce09ad199a5e47ad7e
4
+ data.tar.gz: eeb7ef868ad8f4f7c1dbc5a260880a83c8fd1baca676b1994d05546af5751fe7
5
5
  SHA512:
6
- metadata.gz: cbeef36cae41f32123c024070984905931920c2cdf0cba6fb29da0ce15f81a13f135861e49607a52cb7a57fd8c3f111531c12c5f78b8a895cb9f0298cb8e3a1e
7
- data.tar.gz: 9632ebe62a05cc0c1a931381f7c139d0957d57f4a76b75dcdb992921f52ac23e24e1a093ec2b94b1745615a131949caa4c9aa7efdb6b6efd8e44fd8d7f2625ed
6
+ metadata.gz: 74dd2fe6d6d39ed3fb9f485c163d28121fcef11baa708d8ed2e0f713ed39d310efdd2db725e468ea7440e8615cad175bc52a20d31cdd60e17d954d06489f4cc1
7
+ data.tar.gz: 1376fb3f38b62c1cf2e336efa6596af1d29b861368a6ae185a18b706e52cc7e1606b3649d9f04c75d8cb650f5e8bcb08c46c5d1dfd9ffed63967f06e3b7f3838
data/.travis.yml CHANGED
@@ -1,12 +1,11 @@
1
- ---
2
1
  language: ruby
3
2
  cache: bundler
4
3
  env:
5
- - NOLOG=true
4
+ global:
5
+ - NOLOG=true
6
6
  rvm:
7
7
  - 2.5.7
8
8
  - 2.6.5
9
9
  before_install: gem install bundler -v 2.1.4
10
10
  script:
11
11
  - bundle exec rspec
12
- - gem build gs_img_fetcher.gemspec
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- gs_img_fetcher (0.1.0)
4
+ gs_img_fetcher (0.1.1)
5
5
  activesupport (~> 6.0)
6
6
  down (~> 5.1)
7
7
  dry-cli (~> 0.6)
data/README.md CHANGED
@@ -1,4 +1,8 @@
1
- # GsImgFetcher [![Build Status](https://travis-ci.org/AkihikoITOH/gs_img_fetcher.svg?branch=master)](https://travis-ci.org/AkihikoITOH/gs_img_fetcher)
1
+ # 🖼 GsImgFetcher
2
+
3
+ [![Build Status](https://travis-ci.org/AkihikoITOH/gs_img_fetcher.svg?branch=master)](https://travis-ci.org/AkihikoITOH/gs_img_fetcher)
4
+ [![Gem Version](https://badge.fury.io/rb/gs_img_fetcher.svg)](https://badge.fury.io/rb/gs_img_fetcher)
5
+ [![Maintainability](https://api.codeclimate.com/v1/badges/277255cf51b425a22551/maintainability)](https://codeclimate.com/github/AkihikoITOH/gs_img_fetcher/maintainability)
2
6
 
3
7
  `gs_img_fetcher` is a tool to download images from remote hosts and save them on your local storage.
4
8
 
@@ -20,20 +24,54 @@ Or install it yourself as:
20
24
 
21
25
  ## Usage
22
26
 
27
+ ### CLI
23
28
  Let's say you have in your current directory a text file named `urls.txt` containing list of image URLs, each line containing one URL.
24
29
 
25
- Running `gs_img_fetcher run urls.txt output` would take URLs from `urls.txt`, downloads the images and save them in the directory `output`.
30
+ ```sh
31
+ $ cat urls.txt
32
+ http://example.com/image1.jpg
33
+ http://example.com/image1.png
34
+ http://example.com/image1.svg
35
+
36
+ $ gs_img_fetcher run urls.txt output
37
+ I, [2020-05-17T13:09:01.420214 #87392] INFO -- : Processing 3 URLs (3 valid, 0 invalid)
38
+ ...
39
+ I, [2020-05-17T13:09:02.709097 #87392] INFO -- : Fetch complete (3 successful, 0 failed)
40
+
41
+ $ ls output
42
+ 1e8256aa-5cb7-4545-9109-65aaa550deac.jpg 49d4f436-110f-4206-a2d6-07cc6156fc56.png a5b4ce07-1fc3-49e3-b558-44f8c4afaaab.svg
43
+ ```
44
+
45
+ Running `gs_img_fetcher run urls.txt output` would take URLs from `urls.txt`, downloads the images and saves them in the directory `output`.
26
46
 
27
47
  Run `gs_img_fetcher --help` to show usage guide.
28
48
 
29
49
  Set the environment variable `NOLOG` to a truthy value to suppress logs.
30
50
 
31
- ## Components
51
+ ### Hooking GsImgFetcher into your own application
32
52
 
33
- `gs_img_fetcher` is designed with concurrency in mind. It can be configured to fetch images either asynchronously or synchronously.
34
- By default, it runs asynchronously and the maximum number of threads depends on what your machine allows.
35
- For a relatively small input, it would be better to specify `--no-async` option.
36
- Check out the options `async` and `max_threads`.
53
+ #### Fetching a single image
54
+ ```ruby
55
+ fetcher = GsImgFetcher::Fetcher.new(
56
+ GsImgFetcher::InputEntry.new('http://example.com/image.png'),
57
+ 'output'
58
+ )
59
+ fetcher.fetch
60
+ fetcher.save
61
+ fetcher.successful?
62
+ ```
63
+
64
+ #### Fetching multiple images
65
+ ```ruby
66
+ input = GsImgFetcher::Input.from_file('urls.txt')
67
+ # or
68
+ urls = ['http://example.com/image.png', 'http://example.com/image2.png']
69
+ entries = urls.map { |url| GsImgFetcher::InputEntry.new(url) }
70
+ input = GsImgFetcher::Input.new (entries)
71
+
72
+ manager = GsImgFetcher::Manager.new(input, output_dir: 'output', async: false)
73
+ manager.setup.fetch
74
+ ```
37
75
 
38
76
  - `Manager` is what controls the entire process of handling the input and fetching and saving the images.
39
77
  - `Input` is responsible for finding the input file and parsing, sanitizing and validating the list of URLs.
@@ -4,15 +4,20 @@ require 'active_support/core_ext/object/blank'
4
4
 
5
5
  module GsImgFetcher
6
6
  class Input
7
- def initialize(path)
8
- @path = path
7
+ class << self
8
+ def from_file(path)
9
+ entries = File.open(path, 'r') { |f| f.each_line.map(&:strip).map(&:presence) }
10
+ .compact
11
+ .uniq
12
+ .map { |url| InputEntry.new(url) }
13
+ new(entries)
14
+ end
9
15
  end
10
16
 
11
- def entries
12
- @entries ||= File.open(@path, 'r') { |f| f.each_line.map(&:strip).map(&:presence) }
13
- .compact
14
- .uniq
15
- .map { |url| InputEntry.new(url) }
17
+ attr_reader :entries
18
+
19
+ def initialize(entries)
20
+ @entries = entries
16
21
  end
17
22
 
18
23
  def valid_entries
@@ -5,17 +5,18 @@ require 'concurrent'
5
5
  module GsImgFetcher
6
6
  class Manager
7
7
  class << self
8
- def fetch(*args, **opts)
9
- new(*args, **opts).setup.fetch
8
+ def fetch(input_path, **opts)
9
+ input = Input.from_file(input_path)
10
+ new(input, **opts).setup.fetch
10
11
  end
11
12
  end
12
13
 
13
- attr_reader :entries, :successful_fetches, :failed_fetches
14
+ attr_reader :input, :entries, :successful_fetches, :failed_fetches
14
15
 
15
16
  DEFAULT_MAX_THREADS = Concurrent.processor_count
16
17
 
17
- def initialize(input_path, output_dir: nil, async: true, max_threads: nil)
18
- @input_path = input_path
18
+ def initialize(input, output_dir: nil, async: true, max_threads: nil)
19
+ @input = input
19
20
  @output_dir = output_dir || Dir.pwd
20
21
  @async = async
21
22
  @max_threads = max_threads || DEFAULT_MAX_THREADS
@@ -24,10 +25,6 @@ module GsImgFetcher
24
25
  @failed_fetches = Queue.new
25
26
  end
26
27
 
27
- def input
28
- @input ||= Input.new(@input_path)
29
- end
30
-
31
28
  def setup
32
29
  tap do
33
30
  log_entries
@@ -81,7 +78,7 @@ module GsImgFetcher
81
78
  end
82
79
 
83
80
  def log_entries
84
- LOGGER.info("Processing #{input.entries.count} URLs (#{input.valid_entries.count} valid, #{input.invalid_entries.count} invalid) from #{@input_path}")
81
+ LOGGER.info("Processing #{input.entries.count} URLs (#{input.valid_entries.count} valid, #{input.invalid_entries.count} invalid)")
85
82
  end
86
83
 
87
84
  def log_result
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module GsImgFetcher
4
- VERSION = '0.1.0'
4
+ VERSION = '0.1.1'
5
5
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gs_img_fetcher
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.1.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Akihiko Ito